JP2002135785A - Conversion method of motion vector and conversion apparatus thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To attain a conversion method of motion vector capable of enhancing coding efficiency of image coding of MPEG4 in a conversion method of image coding format. SOLUTION: From a bit stream of input image coding of MPEG2, I frame and P frame of the bit stream are extracted, after that, with abandoning the I frame or the P frame at every other piece, a bit stream of output image coding of MPEG4 having one-fourth of vertical/horizontal resolution of respective vertical/horizontal resolution of the bit stream of input coding is acquired, and with performing a time correcting processing to expand in two times in the direction of time of the motion vector of the P frame of the bit stream of output image coding which is encoded immediately following the I frame or P frame to be abandoned, a motion vector of 8×8 of the bit stream of output image coding is acquired.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for receiving a bit stream indicating image information compressed by orthogonal transform such as discrete cosine transform and motion compensation via network media such as satellite broadcasting, cable TV and the Internet. Used when processing on a storage medium such as optical, magnetic disk, or flash memory,
The present invention relates to a motion vector conversion method (vector conversion device) in an image information conversion method (image information conversion device) using the MPEG system.

[0002]

2. Description of the Related Art In recent years, image information has been treated as digital data. At this time, orthogonal transformation such as discrete cosine transformation and the like are performed by utilizing the redundancy inherent in the image information for the purpose of efficiently transmitting and storing information. Devices conforming to the MPEG system for performing compression by compensation are becoming widespread in both information distribution at broadcast stations and the like and information reception in ordinary households.

In particular, MPEG2 (ISO / IEC 13
818-2) is defined as a general-purpose image coding method, and is a standard method covering both interlaced scan images and progressive scan images, as well as standard resolution images and high definition images, and is widely used for professional and consumer applications. It is expected to be used in applications in the future. By using the compression method of MPEG2,
For example, a standard resolution interlaced scan image having 720 × 480 pixels or a high resolution interlaced scan image having 4 to 8 Mbps and 1920 × 1088 pixels is 18 to 2
By assigning a code amount (bit rate) of 2 Mbps, it is possible to realize a high compression rate and good image quality.

[0004] MPEG2 is mainly intended for high-quality coding suitable for broadcasting, but does not support a coding amount (bit rate) lower than that of MPEG1, that is, a coding system with a higher compression rate. With the spread of mobile devices,
It is expected that the need for such an encoding scheme will increase in the future. In response, the MPEG4 encoding system has been standardized. Regarding the image coding method of MPEG4, 1
In December 998, the standard was approved as an international standard as ISO / IEC 14496-2.

[0005] By the way, a bit stream indicating MPEG2 image compression information once encoded for digital broadcasting is more suitable for processing on a portable terminal or the like. There is a need to convert to a bitstream that represents information.

[0006] Fig. 1 shows a conventional example of an image information conversion apparatus which achieves the above object, and this will be described below. The image information conversion device in FIG. 1 is a device that converts an input bit stream indicating MPEG2 image compression information into a bit stream indicating MPEG4 image compression information. That is, the input bit stream indicating the MPEG2 image compression information is restored by the MPEG2 image decoding device 1. The restored image signal is transmitted to the resolution frame rate conversion device 2 and is converted into an image signal having any different resolution and frame rate. The converted image signal is input to an MPEG4 image encoding device 3, and the MPEG4 image information encoding device 3
It is encoded into a bit stream indicating PEG4 image compression information and output.

As shown in FIG. 1, in a conventional image information conversion apparatus, an image signal restored by an MPEG2 decoding method is encoded by an MPEG4 image encoding apparatus.
A bit stream indicating MPEG4 image compression information is output. In the image information decoding apparatus 1 of MPEG2, both the horizontal and vertical components
8th-order DC of a bit stream indicating image compression information
It is conceivable to perform a decoding process using all the T (discrete cosine transform) coefficients. However, in the vertical direction, all the eighth-order coefficients are used, and in the horizontal direction, decoding processing using only low-frequency four components of the eighth-order coefficients (hereinafter referred to as 4 × 8 down-decoding) or horizontal decoding is performed. By performing a decoding process using only the four low-frequency components of the eighth-order coefficients in the horizontal direction in the vertical direction (hereinafter referred to as 4 × 4 down-decoding), the calculation is performed while minimizing image quality deterioration. A configuration in which the amount and the video memory capacity are reduced and the downsampling process in the subsequent stage is further simplified can be considered.

In such a conventional method, in the MPEG4 image encoding apparatus, when encoding an input image signal, the amount of arithmetic processing for detecting a motion vector is about 60 to 70% of the total arithmetic processing amount. Occupy. As a result, it becomes difficult to process images in real time, causing a time delay, causing problems such as an increase in the size of the apparatus.

As means for solving such a problem, the present inventors have previously proposed an image information conversion apparatus (unknown at the time of filing the present application) shown in FIG.

FIG. 3 shows a motion vector in a bit stream indicating MPEG2 image compression information,
It is a figure which shows the correlation of the motion vector in the bit stream which shows the image compression information of EG4. 3A and 3B show images of the current frame before and after resolution conversion, respectively. When the resolution of the image is converted, the horizontal component of the motion vector from the position in the previous frame after the conversion to the position in the current frame is obtained by converting the horizontal component of the motion vector before the resolution conversion into the horizontal resolution of the image. It can be determined by the rate. The vertical component after resolution conversion is obtained from the vertical component of the motion vector before resolution conversion and the vertical resolution conversion rate of the image. That is, there is a large correlation between the motion vector before resolution conversion and the motion vector after conversion.
Using the correlation, a motion vector after conversion can be obtained from a motion vector before resolution conversion.

That is, in the image information conversion device shown in FIG.
The input bit stream indicating the MPEG2 image compression information is converted into M by using parameters such as the motion vector of the MPEG2 macroblock and the macroblock type.
Briefly convert to PEG4 motion vector. MPEG4
In the image information encoding device 7, the image encoding using the converted motion vector is performed without detecting the motion vector. As a result, since no motion detection is performed in the MPEG4 image encoding apparatus, the processing amount is greatly reduced.

As described above, the conversion from the MPEG2 motion vector to the MPEG4 motion vector is performed. In addition to the motion vector, the parameters used for decoding in MPEG2 or the parameters after the conversion are adopted. , MPEG4, the processing amount of the image information encoding device 7 is reduced, so that the time delay of signals by the device 7 can be reduced.

In FIG. 3, the input bit stream indicating the image compression information of MPEG2 is the MPEG2 of FIG.
The second image information decoding device 4 performs a decoding process, and outputs a bit stream indicating MPEG4 image compression information. In the image information decoding device 4 of MPEG2, the input M
It is conceivable to perform decoding processing using all the 8th-order DCT coefficients of the bit stream indicating the PEG2 image compression information. However, by performing 4 × 8 down decoding or 4 × 4 down decoding, image quality degradation is minimized. It is also conceivable to reduce the calculation amount and the video memory capacity while minimizing the downsampling and further simplify the subsequent downsampling processing.
The image signal output from the decoding device 4 is sent to the resolution frame rate conversion device 5, and after performing the resolution frame rate conversion, the image signal is suitable for MPEG4 image encoding by the image size adjustment flag input from the outside. An image signal having a resolution is output.

Here, the resolution frame rate converter 5
First, the resolution of the image signal input from the MPEG2 image decoding device 4 is converted by a resolution frame converter. Here, a case where the resolution is reduced to 1/4 in both the vertical and horizontal directions will be described as an example. As shown in FIG. 4, in the vertical downsampling, the first field or the second field of the input interlaced scan image is extracted and converted into a progressive scan image. The resolution in the horizontal direction is converted to 1/4 resolution using a downsampling filter.
In the vertical direction, 1 /
4 resolution. Also, in order to realize a low bit rate, not only compression by resolution conversion but also I / O
Only the first picture or the second field is extracted from the P picture, and the frame rate is reduced in the time direction. For example, IBBPBB of MPEG2 shown in FIG.
After the resolution frame conversion, the P image has the configuration of the first field (top field) of the IPPP. An image that has undergone resolution frame conversion has both a vertical and horizontal pixel count of 1 so that it can be encoded by the MPEG4 image encoding method.
Make it a multiple of 6. Therefore, the pixel is supplemented or the pixel is removed by a pixel supplement or pixel removing circuit based on an image size adjustment flag input from the outside.

The image size adjustment flag is input from the outside of the resolution frame rate converter 5 and is used to determine whether pixels are to be added to or removed from the image when the number of vertical and horizontal pixels of the image is not a multiple of 16. Flag.

Referring to FIG. 5, the processing of an image by the image size adjustment flag will be described. Assuming that the resolution of an image output from the MPEG2 image decoding device 4 in FIG. 2 is m pixels × n pixels, both m and n are multiples of 16;
The vertical and horizontal resolutions m / 4 and n / 4 obtained by down-sampling the vertical and horizontal resolutions into quarters, respectively, are integer multiples of 16 or divided by 16 to leave a remainder of 4, 8, or 12 pixels. (Surplus pixels). If both m / 4 and n / 4 are multiples of 16, the image is not processed because the image is suitable for the MPEG4 encoding method. Other than that, it is not suitable for the MPEG4 encoding method, so it is necessary to process the image with the image size adjustment flag. The image size adjustment flag has two options of pixel supplement and removal. 16 vertical and horizontal resolutions m / 4 and n / 4
If there is a remainder of 4, 8, or 12 pixels, the removal of the image is selected to remove the extra pixels. That is, the output image is (m / 4 surplus pixels) or (n / 4 supplementary pixels).

On the other hand, if the pixel interpolation is selected, the newly created 4, 8 or 12 pixels, or the supplementary pixels duplicated from the original image or the rows of the compensation pixels suitable for the image, or the beginning or the end of the column. Add pixels. That is, the output image is (m / 4 + supplemented pixel) or (n /
4 + supplemented pixels). As a result, the width and height of the converted image resolution are multiples of 16, and an image having a size suitable for the MPEG4 encoding method is output.

On the other hand, the input MPEG2-encoded image bit stream is subjected to variable-length code decoding by the MPEG-2 image decoding device 4 and then to a macroblock motion vector or macroblock type of P-picture only. Other parameters are extracted and sent to the motion vector converter 6.

Next, the operation principle of the motion vector conversion in the motion vector conversion device 6 will be described with reference to FIG. FIG. 6 shows an example of the operation principle of the motion vector converter,
Each one of the square lattices separated by a solid line in FIGS. 6A and 6B indicates a macroblock. FIG.
The image output from the MPEG2 decoding device 4, that is, the image before the resolution conversion. The image of FIG.
The image A is an image obtained by converting the image in both the vertical and horizontal resolutions to quarter by the resolution frame rate converter 5. For example, the four 16 × 16 macroblocks hatched in the upper left of FIG. 6A showing the image before conversion become 8 × 8 blocks in the upper left of FIG. 6B showing the image after conversion. One 8 × 8 motion vector (MV) is obtained from the four normalized motion vectors after resolution conversion. Further, the 16 shaded 16 × 16 macroblocks on the right side of FIG. 6A showing the image before the conversion are the shaded 16 × 16 macroblocks on the right side of FIG. 6B showing the image after the conversion. Become a block. From the normalized 16 motion vectors after the resolution conversion, one 16 × 16 motion vector (M
V). Thereby, four 8 of motion vectors used for encoding by the image encoding method of MPEG4 are obtained.
A × 8 motion vector (MV) and one 16 × 16 motion vector (MV) are created.

FIG. 7 shows an example of a specific configuration of the motion vector converter 6 in FIG. 2, which will be described below. The input parameters such as the MPEG2 motion vector and the image size are the MPEG2 16 × 16 motion vector (macroblock motion vector) → MPEG.
The normalized motion vector (MV) before correction is created by the conversion device 8 for converting the normalized motion vector into a normalized motion vector (motion vector according to resolution).

16 × 16 MV of MPEG2 in FIG.
→ The principle of operation of the MPEG4 normalized MV converter 8
This will be described with reference to the flowchart of FIG. Since a frame structure is generally used in a bit stream indicating MPEG2 image compression information of interlaced scanning, a conversion method only for processing in the case of a frame structure will be described this time.

At step ST-1, the inputted MPEG
It is determined whether the motion vector and the macroblock type of the macroblock 2 are an intra macroblock, a skip macroblock, or an inter macroblock. If it is determined in step ST-1 that the block is an intra macro block, the normalized motion vector is set to 0 in step ST-2, and an intra mode flag is provided for performing processing by the compensator 12.
In MPEG2, if the block is an intra macro block,
The intra mode flag is set.

If it is determined in step ST-1 that the block is a skipped macroblock, the normalized motion vector is set to 0 in step ST-3.

If it is determined in step ST-1 that the block is an intra macro block, the process proceeds to step ST-4 to determine whether it is frame prediction or field prediction. If the image has a frame structure and is frame prediction, in step ST-5, conversion to a motion vector suitable for frame prediction is performed. This conversion to a motion vector will be described with reference to FIG. As explained in FIG.
The horizontal component of the motion vector after conversion is obtained from the horizontal component of the motion vector before conversion and the horizontal resolution conversion rate of the image. The vertical component is obtained from the vertical component of the motion vector before conversion and the vertical resolution conversion rate of the image. That is, when the resolution in the horizontal direction is converted to a quarter, the horizontal component of the converted motion vector also becomes a quarter before the conversion. When the resolution in the vertical direction is converted to a quarter, the vertical component of the converted motion vector also becomes a quarter before the conversion.

For example, the motion vector shown in FIG.
(8, 12) before the conversion in FIG. 9A is changed to (2, 3) after the conversion in FIG. 9B. In this case, the interval between the intermediate values (half pixels) of the integer pixels is set to 1. Before the resolution conversion in FIG. 9A, the black circles indicate the positions of integer pixels, and the diamonds indicate the positions of half pixels. After resolution conversion in FIG. 9B, half pixels are indicated by white circles.

As can be seen from FIG. 9, the motion vector shown at the position of the integer pixel before the conversion and located at the quarter pixel value after the conversion is not an integer pixel but a half pixel position. Convert as shown. This is because a decoded image signal originally contains distortion due to quantization, so that if it is used as a predicted image as it is, the prediction efficiency is reduced and image quality may be degraded. In order to reduce this, half-pixel precision may be selected by linearly interpolating 1: 1 between each pixel on the reference screen corresponding to a low-pass filter, thereby avoiding image quality degradation. . Therefore, even in the encoding by the MPEG4 image encoding method, in order to improve the prediction efficiency and prevent the image quality from deteriorating, even when the encoding is preferentially converted to the MPEG4 format, the conversion is performed as indicated by the half pixel position. The motion vectors before and after the conversion are shown in the table of FIG. 10A. The table of FIG. 10A shows the remainder obtained by dividing the motion vector before conversion by 8 and the motion vector after conversion. [MV / 4] represents an integer part of a quotient obtained by dividing MV by 4.

If it is determined in step ST-4 in FIG. 8 that the image has a frame structure and the field is predicted, the process proceeds to step S-4.
The process proceeds to T-6, and it is determined whether it is the first field prediction (top field prediction) or the second field prediction (bottom field prediction).

In the case of top field prediction, in step ST-7, conversion of a motion vector suitable for the first field prediction of field prediction is performed. FIG. 11 shows the concept of motion vector conversion in the case of top field prediction, which will be described below. The horizontal component of the motion vector is subjected to the same processing as in the table of FIG. 10A. In the vertical direction, the resolution is reduced to half by extracting the top field. Since the top field prediction is performed, the motion vector before the conversion is further reduced by half according to the table of FIG. In the table of FIG. 10B, [MV / 2] is MV
Divided by 2 represents the integer part of the quotient.

If it is determined in step ST-6 in FIG. 8 that the current field is the second field prediction (bottom field prediction), a motion vector conversion suitable for the second field prediction of the field prediction is performed in step ST-8. FIG. 12 shows the concept of motion vector conversion in the case of bottom field prediction, which will be described below. At the time of resolution conversion, only the top field is extracted, so after the conversion, the top field is used as a reference image. Therefore, MPE
Spatio-temporal correction of the motion vector is performed so that the bottom field used as the prediction image in G2 is converted into the top field prediction after resolution conversion.

In FIG. 11, spatial correction is performed to approximately convert from bottom field prediction to top field prediction. That is, 1 is added to the vertical component of the motion vector. As can be seen from the figure, when 1 is added to the vertical component of the motion vector obtained by the bottom field prediction, one row is moved up, so that the bottom field reaches the same spatial position as the top field, and It looks like a motion vector obtained by prediction. The following equation 1 represents the vertical component of the motion vector MVtop when the bottom field at the same spatial position as the top field, that is, the approximate top field is predicted by the spatial correction.

[0031]

## EQU1 ## Vertical component: approximate MVtop = MVbottom + 1

In MPEG2, there is a time lag between the top field and the bottom field of the interlace. For this reason, temporal correction is performed to eliminate a time lag between the top field approximated to the bottom field and the actual top field. FIG. 12 shows the temporal positional relationship of each field. Here, the interval between the top field and the bottom field is 1, and
If a is the interval between the bottom field of the I picture and the top field of the P picture, a
It becomes an odd number such as 1, 3, 5, 7,.
If a is 1, the image configuration is IPPP @
This is the case of ‥. The time-corrected motion vector MV 'is shown by equation (2).

[0033]

## EQU2 ## Vertical component: MV ′ = {(a + 1) / a} approximation MVtop

When the equation (2) is substituted into the equation (1), the vertical component of the converted motion vector becomes the following equation (3).

[0035]

## EQU3 ## Vertical component: MV '= {(a + 1) / a} (MV
bottom + 1)

Note that the horizontal component of the converted motion vector is obtained by multiplying the motion vector before conversion by (a + 1) / a, performing temporal correction, and then calculating according to the calculation in the table of FIG. 10A.

If necessary, the vertical component of the motion vector may be temporally corrected and then spatially corrected. The vertical component of the motion vector MV 'in this case is shown by the following equation (4). Note that the horizontal component has the same value for the space / time correction (time correction is performed after the space correction is performed) and the time / space correction (space correction is performed after the time correction is performed).

[0038]

## EQU4 ## Vertical component: MV ′ = ｛(a + 1) / a｝ MVbottom + 1

The difference between the equations (3) and (4), that is, the difference between the vertical component of the motion vector when the space / time correction is performed and the time / space correction is 1 / a. Therefore,
The effect of the difference differs depending on the value of a. Therefore, when a is 1, it is larger than 1, that is, 3, 5, 7,.
The correction method in the two cases ‥‥‥ will be described.

When 1 is substituted for a in the equation (3), the vertical component of the motion vector is as shown in the following equation (5).

[0041]

## EQU5 ## Vertical component: MV '= 2.times. (MVbottom + 1)

When 1 is substituted for a in equation (4), the vertical component of the motion vector is as shown in equation (6).

[0043]

## EQU00006 ## Vertical component: MV '= 2.times. (MVbottom + 1) -1

As a result, the motion vector MVbott before conversion
When 0, 1, 2, and om are substituted for om, the value according to the equation of Equation 5 becomes an even number such as 2, 4, 6, and ‥‥‥‥. That is, regardless of whether the motion vector before the conversion is shown at the position of an integer pixel or at the position of a half pixel, if the space / time correction is performed, the motion vector after conversion is shown at the position of the integer pixel.

Further, the values obtained by the equation (6) are 1, 3, 5,
It becomes an odd number like ‥‥‥‥. That is, regardless of whether the motion vector before the conversion is shown at the position of an integer pixel or at the position of a half pixel, if the time / space correction is performed, the motion vector after conversion is shown at the position of a half pixel. Therefore, when the motion vector indicated at the position of the integer pixel before the conversion is indicated at the position of the integer pixel even after the conversion, the space / time correction is performed. In addition, when the motion vector indicated at the position of the half-pixel before the conversion is indicated at the position of the half-pixel after the conversion, time / space correction is performed. That is, the motion vector before conversion is alternately subjected to space correction and time correction to convert to a motion vector after resolution conversion, or to the motion vector before conversion, all time and space correction, Alternatively, space and time corrections are all performed.

When a is not 1, ie, 3,
In the cases of 5, 7, and ‥‥‥, 1 / a, which is the difference between the time / space correction and the space / time correction, can be approximated to 0. In this case, space / time correction may be performed, or time / space correction may be performed.

After the motion vector conversion processing with reference to the bottom field is completed, the motion vector is further reduced to one half according to the table of FIG. 10B to obtain a converted motion vector.

As described above, the MPEG4 normalized motion vector before the correction is output. As shown in FIG. 7, the output normalized motion vector is sent to a motion vector adjuster 9 based on an image size adjustment flag, and a motion vector suitable for the image size is output according to an image size adjustment flag input from the outside. I do.

The operation of the motion vector adjuster 9 based on the image size adjustment flag in FIG. 7 is shown in the flowchart of FIG. In step ST-11, it is determined whether or not m / 4 and n / 4 are both multiples of 16 with respect to the input image size of m pixels × n pixels. The MPEG4 normalized motion vector is output as it is without processing. Step ST-11
If any of m / 4 and n / 4 is not a multiple of 16, the image size adjustment flag input from outside is activated, and it is determined in step ST-12 whether or not pixel removal is to be performed. In the case of removal, the normalized motion vector of the removed surplus pixel is not output, and other normalized motion vectors are output. If pixel removal is not determined in step ST-12, whether or not pixel replacement is performed is determined in step ST-13. In the case of pixel replacement, the normalized motion vector of the filled pixel is set to 0, and other pixels are replaced. It is output together with the input normalized motion vector.

Returning to FIG. 7, the normalized motion vector suitable for the image size output from the motion vector adjuster 9 based on the image size adjustment flag is calculated by the arithmetic circuit 10 into the following equation (7). By performing the operation, M
Four MPEG2s that make up a PEG4 macroblock
Among the macroblocks, a sum of normalized motion vectors converted from non-intra macroblocks divided by the number of non-intra macroblocks is output as 8 × 8 MV.

[0051]

(Equation 7)

In this case, when a normalized motion vector converted from a macroblock in which the DCT coefficient of an MPEG2 macroblock before conversion is zero is included, the normalized motion vector is used as a motion vector having high coding efficiency. Select 8x8MV. When calculating the average 8 × 8 MV, the average may be determined by performing a weighted average process on the DCT coefficients of macroblocks of MPEG2.

On the other hand, the arithmetic circuit 11 performs the operation shown in the following equation (8), whereby the normalization converted from the non-intra macroblock among the 16 MPEG2 macroblocks constituting the MPEG4 macroblock is performed. An average obtained by dividing the sum of the motion vectors by the number of non-intra macroblocks is output as 16 × 16 MV.

[0054]

(Equation 8)

In this case, when a normalized motion vector converted from a macroblock in which the DCT coefficient of an MPEG2 macroblock before conversion is zero is included, the normalized motion vector is regarded as a motion vector having high encoding efficiency. Select 8x8MV. When an average of 16 × 16 MV is obtained, the average may be determined by performing a weighted average process on DCT coefficients of macroblocks of MPEG2. The 8 × 8 MV obtained by the operation of Expression 7 and the 16 × 16 MV obtained by Expression 8 are combined and output as a motion vector of MPEG4.

FIG. 16 shows the MPEG4 8 × format shown in FIG.
8 shows a conversion device (an earlier example according to the invention of the present author, etc., which is unknown at the time of filing of the present application) in which the conversion processing in the motion vector conversion device of FIG. 8 is improved. In FIG. 16, the prediction residual of the MPEG2 macroblock in the macroblock information buffer 13 is referred to, and among the four MPEG2 macroblocks constituting the MPEG4 block, the one having the smallest prediction residual is set as the representative macroblock. A normalized motion vector obtained by performing scaling by space / time compensation from the motion vector of the macroblock is defined as 8 × 8 of MPEG4.
Is output as a motion vector. Other configurations are the same as those in FIG.

FIG. 17 shows a configuration diagram of a motion vector re-search apparatus. The entirety of the MPEG4 8 × 8 motion vector conversion apparatus shown in FIG. 16 is collectively shown as a motion vector space / time correction unit 171. MPEG4 output from the motion vector space / time compensator 171
The motion vector of the motion vector
The search device 172 specifies a search range for re-search based on the motion vector correction direction flag, and searches only integer pixel values. Next, a motion vector having an integer pixel value with the smallest prediction residual is output and input to the motion vector half-pixel search device 173. Search device 17
In step 3, nine half-pixel values around the integer pixel value motion vector are searched, and a motion vector with the smallest prediction residual is output. This is similarly calculated for both 8 × 8 and 16 × 16 motion vectors. The motion vector compensation direction flag has an arbitrary value, for example, two horizontal integer pixels, one vertical integer pixel, or the like. A corresponding asymmetric search search window may be specified according to the motion vector information of MPEG2.

The MPEG4 image encoding device 3 shown in FIG. 1 receives the output image from the resolution frame rate conversion device 2 and uses the MPEG4 motion vector output from the motion vector conversion device 2 to encode the MPEG4 image. Encoding is performed according to the method, and an MPEG4 image encoded bit stream is output.

[0059]

SUMMARY OF THE INVENTION The present invention relates to MPEG2
Of the input image encoded bit stream of the MPEG2
A motion vector conversion method in an image coding format conversion method for converting an output image coding bit stream of MPEG4 having a different image coding format from the input image coding bit stream of It is an object of the present invention to propose a technique that can increase the coding efficiency of the coding.

The present invention also relates to an image encoding method for converting an MPEG2 input image encoded bit stream into an MPEG4 output image encoded bit stream having an image encoding format different from that of the MPEG2 input image encoded bit stream. An object of the present invention is to propose a motion vector conversion device in a format conversion device that can increase the coding efficiency of MPEG4 image coding in an image coding format conversion method.

[0061]

SUMMARY OF THE INVENTION A first aspect of the present invention is an MPEG system.
2 of the input image encoded bit stream
In the motion vector conversion method in the image encoding format conversion method for converting the input image encoding bit stream into an MPEG4 output image encoding bit stream having a different image encoding format from the input image encoding bit stream from the MPEG-2 input image encoding bit stream, By extracting the I frame and the P frame, and thereafter discarding the I frame or the P frame every other frame, the vertical and horizontal resolution of each of the vertical and horizontal resolutions of the input image encoded bit stream is 1/4. A motion vector conversion method in an image coding format conversion method for obtaining an MPEG4 output image coded bit stream having a resolution of: Use and destroy I Encoded immediately after the frame or P-frame, against the motion vector of the P-frame of the output image encoded bit stream of MPEG4, by performing a time correction processing for expanding twice in the time direction, MPEG4
Is a motion vector conversion method for obtaining an 8 × 8 motion vector of an output image encoded bit stream.

According to the first aspect of the invention, an I frame and a P frame are extracted from an input image coded bit stream of MPEG2, and thereafter, the I frame or the P frame is discarded every other frame. 1 for each of the vertical and horizontal resolution of the input image encoded bit stream
An output image encoded bit stream of MPEG4 having a vertical and horizontal resolution of / 4 is obtained. And MP
Using the motion vector information and the macroblock information in the EG2 input image encoded bit stream, the MP encoded immediately after the I frame or the P frame to be discarded.
An 8 × 8 motion vector of an MPEG4 output image encoded bit stream is obtained by performing a time compensation process of extending the motion vector of the P frame of the output image encoded bit stream of EG4 by a factor of 2 in the time direction. To

According to a second aspect, in the motion vector conversion method according to the first aspect, an MPE obtained by performing time correction processing
G4 8 × 8 motion vector is input, and motion vector compensation is performed by re-searching the motion vector centered on the input motion vector value to perform four 8 × 8 motion vectors forming one macro block. Among the motion vectors, an MPEG4 8 × 8 motion vector having the least prediction error is assigned to a 16 × 16 motion vector, and the MPEG4
This is a motion vector conversion method for obtaining a 16 × 16 motion vector.

A third invention, an image encoding format for converting an MPEG2 input image encoded bit stream into an MPEG4 output image encoded bit stream having a different image encoding format from the MPEG2 input image encoded bit stream. In the motion vector converter in the converter, an I frame and a P frame are extracted from the MPEG2 input image coded bit stream, and thereafter, a discarding unit is provided for discarding the I frame or the P frame every other frame. Image encoding format conversion by means of discarding means to obtain an MPEG4 output image encoded bit stream having a vertical and horizontal resolution of １ ／ each of the vertical and horizontal resolution of the MPEG2 input image encoded bit stream. Motion vector conversion device And using the motion vector information and macroblock information in the MPEG2 input image encoded bit stream, the MPEG-4 output image encoded bit stream encoded immediately after the I frame or P frame to be discarded. The motion vector of the P frame is subjected to a time compensation process of expanding the motion vector twice in the time direction, and the 8
This is a motion vector conversion device provided with time compensation processing means for obtaining a × 8 motion vector.

In the motion vector conversion apparatus according to the fourth and third aspects of the present invention, an 8 × 8 MPEG4 motion vector obtained by time correction processing by the time correction processing means is input, and the input motion vector value , The motion vector is re-searched and the motion vector is compensated to obtain four 8 × 8 pixels constituting one macro block.
MPEG with the least prediction error among the motion vectors
This is a motion vector conversion device provided with a motion vector compensating means for allocating an 8 × 8 motion vector of 4 to a 16 × 16 motion vector and obtaining a 16 × 16 motion vector of MPEG4.

[0066]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an example of a motion vector conversion method and a conversion apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 18 shows an MP2 stream from an MPEG2 stream.
Picture type and VOP (Video
FIG. 3 shows a conceptual diagram of correspondence of the object plane type. When converting an MPEG2 stream of 30 frames per second having a GOP (screen groove) structure of N15M3 into an MPEG4 stream of 5 frames per second having a GOV structure of N5M1, I and P frames (I-frame, P-frame) are converted into one. Convert to MPEG4 every other sheet. MPE as shown in FIG.
Since 1 GOV of N5M1 of MPEG4 stream corresponds to 2GOP of N15M1 of G2 stream, MPE
When converting from G2 to MPEG4, there are a case where the P frame is discarded and a case where the I frame is discarded.

FIG. 19 shows a conceptual diagram of the motion vector conversion. The macro block is a shaded macro block of the frame. In the macroblock, one MPE is used based on the method described in the related art.
Of the four MPEG2 macroblocks constituting the G4 block, the motion vector of the macroblock with the smallest prediction residual is set as the representative motion vector, and the motion vector of the macroblock is subjected to space / time correction by scaling. Next, the length of the motion vector in the time direction is doubled so as to refer to the MPEG4 VOP converted from the I or P frame preceding the discarded frame. Thereby, the VOP converted from the I or P frame immediately before the frame in which the motion vector of the P-VOP is discarded is referred to. In this way, after performing scaling by space / time compensation, a motion vector expanded twice in the time direction is generated and output as an 8 × 8 motion vector.

FIG. 20 is a block diagram showing the configuration of the motion vector converter. The motion vector conversion device is the same as the motion vector conversion device 6 in FIG. First, MPEG
2 → MPEG4 8 × 8 motion vector converter 201
Then, the same motion vector space / time compensation as described above is performed to generate an 8 × 8 motion vector (MV). Next, in the motion vector integer pixel search device 202, 8 × 8
As described in the related art, an 8 × 8 motion vector re-search process is performed on the motion vector. For example, a search is performed using an integer pixel in the search search window with two pixels each vertically and horizontally, centering on the reference destination of the generated 8 × 8 motion vector. This makes it possible to improve the prediction accuracy of an 8 × 8 motion vector. Next, the prediction error re-searched for each of the four 8 × 8 motion vectors constituting one macroblock and the 8 × 8 motion vector subjected to the re-search processing are converted to 8 × 8 MV → 16 × 16 MV. Conversion device 20
Enter 3

8 × 8 MV → 16 × 16 MV converter 20
3, among the prediction residuals obtained at the time of re-searching the 8 × 8 motion vector among the four 8 × 8 motion vectors,
A motion vector having the least prediction error is determined, and a 16 × 1
6 motion vectors. Next 16x generated
In the same manner as described in the prior art, 16 motion vectors are subjected to re-search processing in the motion vector integer pixel search device 204 to improve the prediction accuracy of 16 × 16 motion vectors. As a result, an 8 × 8 motion vector and a 16 × 16 motion vector of MPEG4 are generated and output.

The output 8 × 8 motion vector and 16
Each of the × 16 motion vectors is input to the motion vector half-pixel search device 205, and the half-pixel accuracy of 8 × 8 and 16
A × 16 motion vector is output.

As described above, in the procedure of inputting the image information compression information of MPEG2 and obtaining the motion vector of 8 × 8 and 16 × 16 of MPEG4, the motion vector converter expands or adds the motion vector, and further scales the motion vector. By performing the motion vector correction by re-searching the obtained motion vector information around the motion vector, it is possible to minimize the reduction in the coding efficiency in the MPEG4 image coding apparatus.

As described above, the image compression information (bit stream) of MPEG2 has been targeted for input and the image compression information (bit stream) of MPEG4 has been targeted for output.
Both input and output are not limited to this, and for example, MPEG-1 or H.264. H.263 or other image compression information (bit stream).

[0073]

According to the first aspect of the present invention, there is provided an image for converting an MPEG2 input image encoded bit stream into an MPEG4 output image encoded bit stream having an image encoding format different from the MPEG2 input image encoded bit stream. In the motion vector conversion method in the encoding format conversion method, an I frame and a P frame are extracted from an input image coded bit stream of MPEG2, and then the I frame or the P frame is set to 1
Image coding in which an MPEG4 output image coded bit stream having a vertical and horizontal resolution of 1/4 of the vertical and horizontal resolution of the input image coded bit stream is obtained by discarding every other image. A motion vector conversion method in a format conversion method, wherein MP
Using the motion vector information and the macroblock information in the EG2 input image encoded bit stream, the MP encoded immediately after the I frame or the P frame to be discarded.
An 8 × 8 motion vector of an MPEG4 output image encoded bit stream is obtained by performing a time compensation process of extending the motion vector of the P frame of the output image encoded bit stream of EG4 by a factor of 2 in the time direction. This is a motion vector conversion method.

According to a second aspect, in the motion vector conversion method according to the first aspect, an MPE obtained by performing time compensation processing is provided.
G4 8 × 8 motion vector is input, and motion vector compensation is performed by re-searching the motion vector centered on the input motion vector value to perform four 8 × 8 motion vectors forming one macro block. Among the motion vectors, an MPEG4 8 × 8 motion vector having the least prediction error is assigned to a 16 × 16 motion vector, and the MPEG4
This is a motion vector conversion method for obtaining a 16 × 16 motion vector.

According to the first and second aspects of the present invention, it is possible to obtain a motion vector conversion method that can increase the coding efficiency of MPEG4 image coding in the image coding format conversion method.

A third invention, an image encoding format for converting an MPEG2 input image encoded bitstream into an MPEG4 output image encoded bitstream having a different image encoding format from the MPEG2 input image encoded bitstream. In the motion vector converter in the converter, an I frame and a P frame are extracted from the MPEG2 input image coded bit stream, and thereafter, a discarding unit is provided for discarding the I frame or the P frame every other frame. Image encoding format conversion by means of discarding means to obtain an MPEG4 output image encoded bit stream having a vertical and horizontal resolution of １ ／ each of the vertical and horizontal resolution of the MPEG2 input image encoded bit stream. Motion vector conversion device And using the motion vector information and macroblock information in the MPEG2 input image encoded bit stream, the MPEG-4 output image encoded bit stream encoded immediately after the I frame or P frame to be discarded. The motion vector of the P frame is subjected to a time compensation process of expanding the motion vector twice in the time direction, and the 8
This is a motion vector conversion device provided with time compensation processing means for obtaining a × 8 motion vector.

In the motion vector conversion apparatus according to the fourth and third aspects of the present invention, an 8 × 8 motion vector of MPEG4 obtained by time correction processing by the time correction processing means is input, and the input motion vector value , The motion vector is re-searched and the motion vector is compensated to obtain four 8 × 8 pixels constituting one macro block.
MPEG with the least prediction error among the motion vectors
This is a motion vector conversion device provided with a motion vector compensating means for allocating an 8 × 8 motion vector of 4 to a 16 × 16 motion vector and obtaining a 16 × 16 motion vector of MPEG4.

According to the third and fourth aspects of the present invention, it is possible to obtain a motion vector conversion device capable of increasing the coding efficiency of MPEG4 image coding in the image coding format conversion method.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a conventional example of an image information conversion apparatus that realizes conversion from a bit stream indicating MPEG2 image compression information to a bit stream indicating MPEG4 image compression information.

FIG. 2 is a block diagram showing a configuration of an image information conversion apparatus (preceding example) for realizing conversion from a bit stream indicating MPEG2 image compression information to a bit stream indicating MPEG4 image information compression information according to the present invention; is there.

FIG. 3 is an explanatory diagram showing a correlation between a motion vector in a bit stream indicating MPEG2 image compression information and a motion vector in a bit stream indicating MPEG4 image compression information.

FIG. 4 is an explanatory diagram showing an operation principle of a resolution frame rate conversion device 5 in the image information conversion device of FIG.

FIG. 5 is an explanatory diagram showing an operation principle of pixel supplementation or removal in the resolution frame rate conversion device 5 according to an image frame size adjustment flag in the image information conversion device of FIG.

FIG. 6 is an explanatory diagram showing a motion vector conversion method of the motion vector conversion device 6 in the image information conversion device of FIG. 2;

7 is a block diagram showing a specific configuration of a motion vector conversion device 6 in the image information conversion device of FIG.

8 shows an MPE in the motion vector converter of FIG.
16 is a flowchart showing the operation of the 16 × 16 motion vector G2 → MPEG4 normalized motion vector converter 8.

9 shows an MPE in the motion vector converter of FIG.
FIG. 9 is an explanatory diagram showing a frame structure in a 16 × 16 motion vector of G2 → Normalized motion vector converter 8 of MPEG4 and a concept of motion vector conversion in the case of frame prediction.

FIG. 10 is a table showing, in contrast, a remainder obtained by dividing a motion vector before A conversion by 8 and a motion vector after conversion. B is a table showing the remainder obtained by dividing the motion vector before conversion by 4 and the motion vector after conversion.

FIG. 11 is an explanatory diagram showing a concept of motion vector conversion when an image has a frame structure and is a first field prediction in FIG. 9;

FIG. 12 is an explanatory diagram showing a concept of motion vector conversion when an image has a frame structure and is a second field prediction in FIG. 9;

FIG. 13 is an explanatory diagram of time adjustment of a motion vector when an image has a frame structure and is a second field prediction in FIG. 9;

FIG. 14 is a flowchart for explaining the operation of the motion vector adjuster 9 based on the image size adjustment flag of the motion vector conversion device in FIG. 7;

FIG. 15 shows a motion vector corrector 1 for an MPEG2 intra macroblock in the motion vector converter of FIG.
FIG. 3 is a block diagram showing the operation principle of No. 2;

FIG. 16 is a block diagram showing a motion vector conversion apparatus (preceding example) for realizing conversion from MPEG2 motion vector information to MPEG4 motion vectors according to the present invention.

FIG. 17 is a block diagram showing a motion vector re-search apparatus.

FIG. 18 is an explanatory diagram of frame discarding according to the embodiment of this invention.

FIG. 19 is an explanatory diagram of discarding an I frame according to the embodiment of this invention.

FIG. 20 is a block diagram illustrating an example of a motion vector conversion device according to an embodiment of the present invention.

[Explanation of symbols]

201 MPEG2 → MPEG4 8 × 8 motion vector converter, 202 motion vector integer pixel searcher, 2
03 8 × 8 motion vector → 16 × 16 motion vector converter, 204 motion vector integer pixel searcher,
205 Motion vector half-pixel search device.

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Teruhiko Suzuki 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation F-term (reference) 5C059 KK41 MA00 MA05 NN03 NN21 PP06 SS10 TA65 TB04 TC12 TD05 UA02 UA23 UA32

Claims (4)

[Claims] 1. An image encoding format conversion method for converting an MPEG2 input image encoded bitstream into an MPEG4 output image encoded bitstream having a different image encoding format from the MPEG2 input image encoded bitstream. In the motion vector conversion method, an I frame and a P frame are extracted from the MPEG2 input image encoded bit stream, and then the I frame and the P frame are extracted.
The MPEG4 output image encoded bit stream having a vertical and horizontal resolution of 1/4 of the vertical and horizontal resolution of the input image encoded bit stream by discarding every other frame or the P frame. A motion vector conversion method in an image coding format conversion method that obtains a motion vector information and a macro block information using a motion vector information and a macro block information in the MPEG2 input image coding bit stream.
By performing a time correction process of extending the motion vector of the P frame of the output image encoded bit stream of MPEG4 immediately after the I frame or the P frame to be discarded in the time direction by twice in the time direction, MP
A motion vector conversion method characterized by obtaining an 8 × 8 motion vector of an output image encoded bit stream of EG4. 2. The motion vector conversion method according to claim 1, wherein an 8 × 8 MPEG4 motion vector obtained by performing the time compensation processing is input, and a motion vector centered on the input motion vector value is calculated. A single macro block is formed by performing a search again and performing motion vector correction.
Among the 8 × 8 motion vectors, an MPEG4 8 × 8 motion vector with the least prediction error is allocated to a 16 × 16 motion vector to obtain an MPEG4 16 × 16 motion vector. Motion vector conversion method. 3. An image encoding format conversion apparatus for converting an MPEG2 input image encoded bit stream into an MPEG4 output image encoded bit stream having an image encoding format different from that of the MPEG2 input image encoded bit stream. In the motion vector converter, an I frame and a P frame are extracted from the MPEG2 input image encoded bit stream, and then the I frame and the P frame are extracted.
A discarding means for discarding every other frame or the P frame is provided, and the discarding means has a vertical and horizontal resolution of １ ／ each of the vertical and horizontal resolution of the MPEG2 input image encoded bit stream. What is claimed is: 1. A motion vector conversion device in an image coding format conversion device for obtaining said MPEG4 output image coding bit stream, wherein the motion vector information and the macroblock information in said MPEG2 input image coding bit stream are used. hand,
The motion vector of the P frame of the output image coded bit stream of the MPEG4, which is coded immediately after the I frame or the P frame to be discarded, is subjected to time compensation processing for expanding the motion vector twice in the time direction, and the MPEG4 A motion vector conversion device comprising a time compensation processing means for obtaining an 8 × 8 motion vector of an output image coded bit stream. 4. The motion vector conversion device according to claim 3, wherein an 8 × 8 MPEG4 motion vector obtained by performing time correction processing by the time correction processing means is input,
By re-searching for a motion vector centered on the input motion vector value and performing motion vector compensation, of the four 8 × 8 motion vectors that constitute one macroblock, the MPEG4 of the least prediction error is used. Assigning 8 × 8 motion vectors to 16 × 16 motion vectors,
A motion vector conversion device comprising a motion vector compensating means for obtaining a 16 × 16 MPEG-4 motion vector.