JP2003348595A - Image processor and image processing method, recording medium and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a calculation amount for acquiring a predictive image. <P>SOLUTION: An image generating part 101 uses an FIR filter and linear interpolation to generate, for example, an interpolated image of 1/2 pixel accuracy, and a frame interpolated image and a field interpolated image are stored in a frame memory 102 and a field memory 103, respectively. A selector 104 reads the frame interpolated image of 1/2 pixel accuracy stored in the frame memory 102 when a motion predicting and compensating part 24 needs a frame interpolated image. The read frame interpolated image of 1/2 pixel accuracy is used, and for example, a predictive image of 1/4 pixel accuracy is generated by using the FIR filter and linear interpolation. The present invention is applicable to an encoding device for encoding image information and a decoding device for decoding the image information. <P>COPYRIGHT: (C)2004,JPO

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly, to image information (bit stream) compressed by orthogonal transform such as discrete cosine transform or Karhunen-Loeve transform and motion compensation. A, satellite broadcasting, cable television broadcasting,
The present invention relates to an image processing apparatus and method, a recording medium, and a program suitable for use when transmitting and receiving via a network medium such as the Internet or when processing on a storage medium such as an optical disk, a magnetic disk, and a flash memory.

[0002]

2. Description of the Related Art In recent years, image information has been handled as digital data.
MPEG (Moving Pic) that compresses by orthogonal transform such as discrete cosine transform and motion compensation using the redundancy inherent in image information
devices based on a system such as a network expert group are becoming widespread in both information distribution of broadcast stations and the like and information reception in ordinary households.

[0003] In particular, MPEG2 (ISO / IEC 13818-2) is a standard defined as a general-purpose image compression method, and is a standard covering both interlaced scan images and progressive scan images, as well as standard resolution images and high definition images. For example, DVD (Digit
al Versatile Disk), and is widely used in a wide range of applications for professional use and consumer use.

[0004] By using this MPEG2 compression method,
For example, for a standard resolution interlaced scan image having 720 × 480 pixels, 4 to 8 Mbps and 1920 × 1
By assigning a code amount (bit rate) of 18 to 22 Mbps to a high-resolution interlaced scan image having 088 pixels, a high compression rate and good image quality can be realized.

[0005] MPEG2 was mainly intended for high-quality encoding suitable for broadcasting, but did not support an encoding system with a higher compression rate. Therefore, the MPEG4 encoding system was standardized. Regarding the image coding method, 1998
In December 2000, the standard was approved as an international standard as ISO / IEC 14496-2.

Further, in recent years, with the initial purpose of image coding for video conferences, ITU-T (International Telecommunication Uni
The standardization of H.26L (ITU-T Q6 / 16VCEG) by the on-telecommunication standardization sector is in progress. H. It is known that 26L requires a greater amount of computation for encoding and decoding than encoding schemes such as MPEG2 and MPEG4, but realizes higher encoding efficiency.

[0007] Currently, as part of MPEG4 activities,
This H. H. 26L. Standardization of coding technology that achieves higher coding efficiency by incorporating functions that are not supported by 26L has been jointly implemented with ITU-T by JVT (Joint
Video Team).

Here, image compression by orthogonal transform such as discrete cosine transform or Karhunen-Loeve transform and motion compensation will be described. FIG. 1 is a diagram showing a configuration of an example of a conventional image information encoding device.

In the image information encoding device 10 shown in FIG. 1, image information composed of an analog signal input from an input terminal 11 is converted into a digital signal by an A / D converter 12. Then, the screen rearrangement buffer 13
The GOP (Gr) of the image information supplied from the A / D converter 12
oup of Pictures) The frames are rearranged according to the structure.

[0010] Here, the screen rearrangement buffer 13 supplies image information of the entire frame to the orthogonal transformation unit 15 for an image on which intra (intra-image) encoding is performed. The orthogonal transform unit 15 performs orthogonal transform such as discrete cosine transform or Karhunen-Loeve transform on the image information, and supplies transform coefficients to the quantization unit 16. The quantization unit 16 performs a quantization process on the transform coefficient supplied from the orthogonal transform unit 15.

The lossless encoding unit 17 determines an encoding mode from the quantized transform coefficients and the quantization scale supplied from the quantization unit 16, and performs variable length encoding or Lossless coding such as arithmetic coding is performed to form information to be inserted into a header portion of an image coding unit. Then, the lossless encoding unit 17 supplies the encoded encoding mode to the accumulation buffer 18 and accumulates the encoded encoding mode. This encoded encoding mode is output to the output terminal 1 as image compression information.
9 is output.

The lossless encoding unit 17 performs lossless encoding such as variable-length encoding or arithmetic encoding on the quantized transform coefficients, and supplies the encoded transform coefficients to the storage buffer 18. And accumulate. The encoded transform coefficient is output from the output terminal 19 as image compression information.

The behavior of the quantization unit 16 is determined by the accumulation buffer 18
Is controlled by the rate control unit 20 based on the data amount of the conversion coefficient accumulated in the. Also, the quantization unit 20
The quantized transform coefficient is supplied to the inverse quantization unit 21, and the inverse quantization unit 21 inversely quantizes the quantized transform coefficient.
The inverse orthogonal transform unit 22 performs an inverse orthogonal transform process on the inversely quantized transform coefficient to generate decoded image information, and supplies the information to the frame memory 23 for storage.

The screen rearrangement buffer 13 supplies image information to the motion prediction / compensation unit 24 for an image to be subjected to inter (inter-image) encoding. The motion prediction / compensation unit 24 extracts image information to be referred to simultaneously from the frame memory 23 and performs motion prediction / compensation processing to generate reference image information. The motion prediction / compensation unit 24 supplies the generated reference image information to the adder 14, and the adder 14
The reference image information is converted into a difference signal from the corresponding image information. The motion prediction / compensation unit 24 supplies the motion vector information to the lossless encoding unit 17 at the same time.

The lossless encoding unit 17 determines an encoding mode based on the quantized transform coefficient and quantization scale supplied from the quantization unit 16, the motion vector information supplied from the motion prediction / compensation unit 24, and the like. Then, lossless coding such as variable length coding or arithmetic coding is performed on the determined coding mode to generate information to be inserted into the header part of the image coding unit. Then, the lossless encoding unit 17 supplies the encoded encoding mode to the accumulation buffer 18 and accumulates the encoded encoding mode. This encoded encoding mode is output as image compression information.

The lossless encoding unit 17 performs a lossless encoding process such as variable-length encoding or arithmetic encoding on the motion vector information to generate information to be inserted into a header portion of an image encoding unit. I do.

Also, unlike the intra coding, in the case of the inter coding, the image information input to the orthogonal transform unit 15 is a difference signal obtained from the adder 14. The other processing is the same as that of the image compression information to be subjected to the intra coding, and the description thereof will be omitted.

Next, FIG. 2 shows an example of the configuration of an image information decoding device corresponding to the above-described image information encoding device 10.
In the image information decoding device 40 shown in FIG. 2, the image compression information input from the input terminal 41 is stored in a storage buffer 42.
, And then transferred to the lossless decoding unit 43.

The lossless decoding unit 43 performs processing such as variable-length decoding or arithmetic decoding on the image compression information based on the determined format of the image compression information, and obtains the encoding mode information stored in the header part. Then, it is supplied to the inverse quantization unit 44 and the like. Similarly, the lossless decoding unit 43 obtains the quantized transform coefficient and supplies it to the inverse quantization unit 44. Furthermore, if the frame to be decoded is inter-coded, the lossless decoding unit 43 also decodes the motion vector information stored in the header part of the image compression information, and decodes the information into the motion prediction / compensation unit. 51.

The inverse quantization section 44 inversely quantizes the quantized transform coefficient supplied from the lossless decoding section 43 and supplies the transform coefficient to the inverse orthogonal transform section 45. The inverse orthogonal transform unit 45 performs an inverse discrete cosine transform or an inverse Karhunen transform on the transform coefficients based on the format of the determined image compression information.
Performs an inverse orthogonal transform such as a Loeve transform.

Here, if the target frame has been intra-coded, the image information subjected to the inverse orthogonal transform processing is stored in the screen rearrangement buffer 47,
The signal is output from the output terminal 49 after the D / A conversion processing in the / A conversion section 48.

When the target frame is inter-coded, the motion prediction / compensation unit 51 converts the motion vector information subjected to the lossless decoding process into the image information stored in the frame memory 50. A reference image is generated based on the reference image and supplied to the adder. The adder 46 combines the reference image and the output from the inverse orthogonal transform unit 45. Note that the other processing is the same as that of the intra-coded frame, and a description thereof will not be repeated.

FIG. 3 is a diagram for explaining the frame motion prediction compensation mode. The frame motion prediction compensation performs motion prediction compensation on a frame in which two fields in the interlaced scanning are combined.
The prediction is performed for every 16 pixels × 16 line blocks of the interlaced scanning. FIG. 3 shows an example of performing forward motion prediction compensation from a reference frame one frame away. The frame motion prediction compensation is an effective prediction method when the motion is relatively slow and moves at a constant speed with a high correlation in the frame.

FIG. 4 is a diagram for explaining the field motion prediction compensation mode. In the field motion prediction compensation, motion compensation is performed for each field, and as shown in FIG.
For the v1 and the second field, prediction is performed using the motion vector mv2. The reference field is the first
Field may be used, and Motion in macro block data
Set by the Vertical Field Select flag. In FIG. 4, the second field is used as a reference field in both the first field and the second field. In field motion compensation, prediction is performed for each field in a macroblock.
It is predicted in units of field blocks of pixels × 8 lines.

With respect to the number of motion vector information, in the unidirectional prediction of a P picture or a B picture, two pieces of motion vector information are required for one macroblock. Also, bidirectional prediction of a B picture requires four pieces of motion vector information per macroblock. For this reason, field motion prediction compensation makes it possible to perform prediction on a field-by-field basis and increase the prediction efficiency for local motion or acceleration-based motion, but on the other hand, the number of motion vector information is smaller than that of frame motion prediction compensation. Because it is required twice,
The overall coding efficiency can be reduced.

In MPEG2, a frame motion prediction compensation mode and a field motion prediction compensation mode can be selected for each macroblock. The selection of these modes is made such that the one with higher encoding efficiency is selected.

On the other hand, H. In 26L, frame motion prediction compensation or field motion prediction compensation cannot be selected on a macroblock basis in the current regulations.
However, VCEG-O37 (L. Wang et. Al; Interlace Co.
ding Tools for H.26L VideoCoding; ITU-T Q6 / SG16, 26t
h Q6 / SG16, 26th November `01), it is proposed that frame motion prediction compensation or field motion prediction compensation can be selected in macroblock units.

In MPEG4, as shown in FIG. 5, a motion vector is a VOP (Video Object Plan).
e) It is specified that it may point outside the boundary. When the region specified by the motion vector is located outside the VOP boundary, information on the pixels located on the VOP boundary is used as the predicted value. H. At 26L,
It is defined that the motion vector may point outside the VOP boundary.

By the way, H. In 26L, 1 /
A high-precision motion prediction compensation process such as 4, 1/8 pixel accuracy is defined. In order to generate this decimal precision prediction image, it is prescribed that a filter with several taps and linear interpolation are combined. However, at present, only processing for progressively scanned images is specified, and the prediction mode is a state in which only frame prediction is specified.

In the following, H. 26L
The motion prediction compensation processing with 4, 1/8 pixel accuracy will be described. FIG. It is a figure for explaining motion prediction compensation processing of 1/4 pixel precision defined in 26L. First, based on the pixels stored in the frame memory, FIR (Finite) of 6 taps in each of the horizontal direction and the vertical direction is used.
An impulse response (Impulse Response) filter is used to generate a pixel value with 1/2 pixel accuracy. The following is defined as an example of the FIR filter coefficient. (1-5 20 20-5 1) // 32 In this FIR filter coefficient, // indicates that the division is a division with rounding (rounding). In this specification, // indicates a rounded division.

A pixel value of 1/4 pixel precision is obtained by linear interpolation from two adjacent pixel values of 1/2 pixel precision obtained above.

FIG. 1 defined in 26L
It is a figure for explaining motion prediction compensation processing of / 8 pixel accuracy. First, based on the pixels stored in the frame memory, FI taps of 8 taps each in the horizontal and vertical directions are used.
Using the R filter, pixel values with １ ／, ／, and ／ pixel accuracy are generated. The following are defined as FIR filter coefficients. (-3 12 -37 229 71 -216-
1) // 256 (-3 12 -39 158 158 -39 12
−3) // 256 (−1−6 −21 71 229 −37 12 −)
3) // 256

The 1/8 pixel precision pixel value is obtained by dividing the two 1/4 pixel precision pixel values generated as described above into two pixel values as shown in FIG. Obtained by linear interpolation.

[0034]

SUMMARY OF THE INVENTION In a coding apparatus capable of selecting frame motion prediction compensation and field motion prediction compensation in units of macroblocks, and in a decoding apparatus for decoding image compression information from the coding apparatus, the motion prediction compensation is used. When obtaining a predicted image, the amount of calculation for obtaining a predicted image with decimal precision becomes a problem. That is, the calculation of the interpolation pixel with decimal precision has been performed by the several tap filter and the linear interpolation as described above. However, there is a problem that performing these calculations for each pixel is a heavy process, which may affect other processes.

In particular, in the motion prediction processing, many pixels near a predetermined area are repeatedly referred to many times. Therefore, when encoding or decoding an image signal, a predicted image can be processed at high speed. Acquisition is important but difficult.

The present invention has been made in view of such circumstances, and it is an object of the present invention to reduce the processing involved in calculating an interpolation pixel and to obtain an interpolation pixel at high speed.

[0037]

An image processing apparatus according to the present invention performs a motion prediction compensation by using a FIR filter or a linear interpolation to obtain image data having a predetermined resolution. Generating means for generating image data necessary for performing the image processing, and a frame memory for storing the image data when the image data generated by the generating means is image data required for performing the frame prediction compensation mode When the image data generated by the generation means is image data required for performing the field prediction compensation mode, the image data is stored in a field memory for storing the image data and a frame memory or a field memory. Prediction compensation means for performing motion prediction compensation using image data.

The motion prediction compensation performed by the prediction compensation means is as follows:
In the frame prediction compensation mode, image data may be read from the frame memory, and in the field prediction compensation mode, a reading unit for reading image data from the field memory may be further included.

[0039] The prediction compensation means can determine a motion vector having the smallest prediction error in both the frame prediction compensation mode and the field prediction compensation mode.

[0040] The prediction compensation means may further determine a motion vector having the smallest prediction error as a motion vector of a macroblock to be processed.

The prediction compensation means can determine a motion vector having the smallest value of the cost evaluation function in both the frame prediction compensation mode and the field prediction compensation mode.

[0042] The prediction compensation means can further determine a motion vector having the smallest value of the cost evaluation function as a motion vector of a macroblock to be processed.

The frame memory and the field memory can separately store image data of a plurality of resolutions.

The resolution may be １ ／, ４, or 画素 pixel accuracy of the input image information.

[0045] The generating means may generate image data of at least one field image of 1/2 or 1/4 pixel accuracy of the input image information by linear interpolation between fields. .

The generation means generates image data of at least one of the field images having 1/2 and 1/4 pixel precision of the input image information using an FIR filter having a predetermined number of taps between fields. You can make it.

When // represents a rounding operation, the generation means converts a field image having half-pixel precision of the input image information into {1, -5, 20, 20,
It can be generated using an FIR filter that performs an operation represented by the equation of −5, 1｝ // 32.

[0048] The generation means may generate a 1/4 pixel precision field image of the input image information from the 1/2 pixel precision field image by using linear interpolation between fields. it can.

The generation means converts the image data of at least one field image of 精度, ／ and ／ pixel accuracy of the input image information into a predetermined number of taps between fields. It can be generated using an FIR filter.

The generation means, when // represents a rounding operation, 生成 of the input image information,
Field images with 2/4 and 3/4 pixel precision are calculated as follows: ３ ， -3,12, -37,229,71, -21,6,-
1｝ // 256 ｛-3,12, -37,229,71, -21,6,-
1｝ // 256 ｛-1,6, -21,71,229, -37,12,-
It can be generated using an FIR filter that performs the operation represented by the formula of 3｝ / 256.

The generation means converts a field image of 1/8 pixel precision of the input image information into 1/4, 2 /
It can be generated from a field image of either 4 or 3/4 pixel accuracy using linear interpolation between fields.

[0052] The generating means may be adapted to output 1/8, 2/8, 3/8, 4/8, 5/8, 6/8,
And image data of at least one field image of 7/8 pixel precision using an FIR filter with a predetermined number of taps between fields.

The generation means, when // represents a rounding operation, 生成 of the input image information,
2/8, 3/8, 4/8, 5/8, 6/8, and 7 /
The field image with 8-pixel accuracy is expressed as follows: ｛−3, 12, −37, 485, 71, −21, 6, −
1｝ // 512１２-3,12, -37,229,71, -21,6,-
1｝ // 256 ｛−6, 24, −76, 387, 229, −60, 1
8, -4｝ // 512１２-3,12, -39,158,158, -39,1
2, -3｝ // 256２-4,18, -60,229,387, -76,2
4, -6｝ // 512 ｛-1,6, -21,71,229, -37,12,-
3｝ // 256 ｛-1,6, -21,71,485, -37,12,-
It can be generated using an FIR filter that performs an operation represented by the formula of 3｝ / 512.

The frame memory and the field memory store V
A padding area for accommodating motion compensation from outside the OP boundary may be provided.

The image processing method of the present invention is an image processing method using FIR filters or linear interpolation, which is image data of a predetermined resolution, and which is necessary for performing motion prediction compensation. When the image data generated in the generation step of generating data and the image data generated in the processing of the generation step is image data required for performing the frame prediction compensation mode, the image data is stored in a frame memory for storing the image data. When the image data generated in the first storage control step to be controlled and the processing in the generation step is image data required for performing the field prediction compensation mode, the image data is stored in the field memory for storing the image data. Using a storage control step to control storage, and image data stored in a frame memory or a field memory, Characterized in that it comprises a prediction compensation step of performing prediction compensation can.

The program of the recording medium of the present invention is
By processing using a filter or processing using linear interpolation, image data of a preset resolution,
A generating step of generating image data necessary for performing motion prediction compensation, and when the image data generated in the processing of the generation step is image data required for performing the frame prediction compensation mode, A first storage control step for controlling storage in a frame memory for storing data, and when the image data generated in the processing of the generation step is image data required for performing the field prediction compensation mode, A storage control step of controlling storage in a field memory for storing the image data, and a prediction compensation step of performing motion prediction compensation using image data stored in a frame memory or a field memory. I do.

According to the program of the present invention, image data having a predetermined resolution, which is necessary for performing motion prediction compensation, is obtained by processing using an FIR filter or processing using linear interpolation. The generation step to generate, and the image data generated in the processing of the generation step,
A first storage control step of controlling storage of the image data in the frame memory for storing the image data when the image data is required for performing the frame prediction compensation mode;
When the image data generated in the processing of the generation step is image data required for performing the field prediction compensation mode, a storage control step of controlling storage of the image data in a field memory, and a frame. And a prediction compensation step of performing motion prediction compensation using image data stored in a memory or a field memory.

The image processing apparatus and method and the program according to the present invention include a frame memory and a field memory for storing a frame image and a field image necessary for performing motion prediction compensation, respectively.

[0059]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 9 is a diagram showing a configuration of an embodiment of an image information encoding device to which the image processing device of the present invention is applied. In the image information encoding device 100 shown in FIG. 9, the image information encoding device 10 shown in FIG.
Blocks having the same functions as those described above are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

The image information encoding device 100 shown in FIG.
Indicates that, when the data output from the inverse orthogonal transform unit 22 is input to the image generation unit 101 and the image generated by the image generation unit 101 is a frame interpolation image, the data stored in the frame memory 102 is a field interpolation image. In such a case, the data is input to the field memory 103. Further, a selector 104 for selecting an image stored in the frame memory 102 or the field memory 103 and outputting the selected image to the motion prediction / compensation unit 24 is provided.

The configuration of the other parts is the same as that of the image information coding apparatus 10 shown in FIG. 1, and the description thereof will be omitted.

FIG. 10 shows the configuration of an embodiment of an image information decoding apparatus to which the image processing apparatus of the present invention is applied, corresponding to the image information encoding apparatus 100 shown in FIG. In the image information decoding device 120 shown in FIG. 10, blocks having the same functions as those of the image information decoding device 40 shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

The image information decoding device 120 shown in FIG.
Indicates that the data output from the adder 46 is
21 and the image generated by the image generation unit 121 is a frame interpolation image, the frame memory 1
In the case where the input image is a field-interpolated image, the image data is input to the field memory 123. Further, a selector 124 for selecting an image stored in the frame memory 122 or the field memory 123 and outputting the selected image to the motion prediction / compensation unit 51 is provided.

The structure of the other parts is the same as that of the image information decoding apparatus 40 shown in FIG.

In the present embodiment, image generation performed by the image generation unit 101 of the image information encoding device 100 shown in FIG. 9 and image generation performed by the image generation unit 121 of the image information decoding device 120 shown in FIG. Is generated in a similar manner. The generated image is stored in the frame memory 102.
The processing stored in (122) or the field memory 103 (123) is similarly performed by the image information encoding device 100 and the image information decoding device 120.

Here, in consideration of the above, FIG.
Image generation unit 101 of the image information encoding apparatus 100 shown in FIG.
The processing up to the generation of the image and the processing up to the storage in the frame memory 102 or the field memory 103 will be described as an example, and the generation of the image by the image generation unit 121 of the image information decoding apparatus 120 shown in FIG. Frame memory 1
The description of the processing up to storage in the storage 22 or the field memory 123 is omitted. Note that the processes of the selector 104 and the selector 124 are different, and will be described later.

First, the case where the image generation unit 101 generates a frame interpolation image or a field interpolation image in the 1/4 pixel accuracy mode will be described as an example. The generation of a frame interpolation image is described in H. This is performed according to the method specified in 26L. As shown in FIG. 11, first, a 6-tap FIR filter coefficient is used from a reference
A frame interpolation image with pixel accuracy is generated, and a similar FIR filter coefficient is used to generate a frame interpolation image with １ ／ pixel accuracy from the frame interpolation image with 画素 pixel accuracy. The generated 1/4 pixel-accurate frame interpolation image is stored in the frame memory 102. An example of a 6-tap FIR filter coefficient is shown below. (1-5 20 20-5 1) // 32

As described above, a 6-tap FIR filter may be used to generate an interpolated image with 1/4 pixel accuracy, or a linear interpolated image of adjacent pixels may be used to generate a frame interpolated image with 1/4 pixel accuracy. It may be generated.
In this way, even when an interpolated image with quarter-pixel accuracy is generated by linear interpolation (interpolation) between adjacent pixels, first, 1 / linear interpolation is performed from the reference image by linear interpolation.
An interpolated image with two-pixel accuracy is generated, and the generated 1 /
From the interpolated image with 2 pixel precision, further 1/4 by linear interpolation
An interpolated image with pixel accuracy is generated.

The interpolated image of 1/2 pixel precision and the interpolated image of 1/4 pixel precision generated in this way are stored in the frame memory 102. Only one of them may be stored.

Further, the generation of an interpolated image using an FIR filter and the generation of an interpolated image using linear interpolation may be used together to generate an interpolated image with 1/4 pixel precision. For example, first, from the reference image, a 6-tap FIR
A filter is used to generate a half-pixel accurate frame interpolation image. Then, a 補 間 pixel accuracy frame interpolation image may be generated from the generated 画素 pixel accuracy frame interpolation image by linear interpolation.

Next, a method of generating a field interpolation image performed by the image generation unit 101 will be described. Referring to FIG. 12, first, a reference image is divided into a first field image and a second field image. Next, for the first field image and the second field image, 6
Using a tap FIR filter, a field-interpolated image with half-pixel accuracy in which the number of pixels in both the vertical and horizontal directions is doubled is generated. In addition, the generated 1/2
From the pixel-interpolated field interpolation image, the same 6-tap FIR filter is used to generate a 1 / 4-pixel accuracy field interpolation image. The generated field interpolation image is stored in the field memory 103. An example of a 6-tap FIR filter is shown below. (1-5 20 20-5 1) // 32

In this way, an interpolated image with 1/4 pixel accuracy may be generated using the 6-tap FIR filter, or a field interpolated image with 1/4 pixel accuracy may be generated by linear interpolation between adjacent pixels. It may be generated.
As described above, even when an interpolated image with quarter-pixel accuracy is generated by linear interpolation (interpolation) between adjacent pixels, first, a half of the reference image is linearly interpolated by linear interpolation.
A pixel-accurate interpolated image is generated, and the generated １ ／
From the pixel-accurate interpolated image, a 1 / 4-pixel-accurate interpolated image is generated by linear interpolation.

The interpolated image of 1/2 pixel precision and the interpolated image of 1/4 pixel precision generated in this way are stored in the field memory 103. Only one of them may be stored.

In the case of generating a 1/4 pixel-accurate field interpolated image, similarly to the case of generating a 1/4 pixel-accurate frame interpolated image, an interpolated image is used by combining an FIR filter and linear interpolation. May be generated.

Next, the generation of a frame-interpolated image or a field-interpolated image in the 1 / 8-pixel accuracy mode performed by the image generating unit 101 will be described. First, generation of a frame interpolation image will be described. This is performed according to the method specified in 26L. With reference to FIG. 13, an 8-tap FIR filter coefficient is used from a reference image to obtain 1/4, 2 /
A frame interpolation image with 4, 3/4 pixel accuracy is generated, and each or any one of the interpolation images is stored in the frame memory 102.

An example of an 8-tap FIR filter when a frame interpolation image with 1/4 pixel accuracy is generated will be described. (-3 12 -37 229 71 -216-
1) An example of an 8-tap FIR filter when generating a frame interpolation image with // 256 2/4 pixel accuracy is shown. (-3 12 -39 158 158 -39 12
-3) // 256 An example of an 8-tap FIR filter when a frame interpolation image with 3/4 pixel accuracy is generated is shown. (-1-6-21 71 229 -37 12-
3) // 256

An FIR is performed on a frame interpolation image of 1/4 pixel accuracy, 2/4 pixel accuracy, or 3/4 pixel accuracy.
By using a filter or using linear interpolation, a frame interpolation image with 1/8 pixel accuracy is generated.

In this way, an interpolated image with 1/8 pixel accuracy may be generated by using the FIR filter, or 1/8 by linear interpolation between adjacent pixels.
A frame interpolation image with pixel accuracy may be generated. As described above, even when an interpolated image with １ ／ pixel accuracy is generated by linear interpolation (interpolation) between adjacent pixels, first, interpolation with 精度 pixel accuracy is performed from the reference image by linear interpolation. An image is generated, and an interpolated image with 1/8 pixel accuracy is further generated from the generated interpolated image with 1/4 pixel accuracy by linear interpolation.

The interpolated images of 1/4 pixel accuracy, 2/4 pixel accuracy, 3/4 pixel accuracy, and 1/8 pixel accuracy generated in this manner are stored in the frame memory 102. Only one of the interpolated images may be stored.

Next, a method of generating a field interpolation image with 1/8 pixel accuracy performed by the image generation unit 101 will be described. Referring to FIG. 14, first, the reference image is divided into a first field image and a second field image. Next, an 8-tap FIR filter is used for each of the first field image and the second field image, and the number of pixels in both the vertical and horizontal directions is quadrupled.
A field interpolation image with 4-pixel accuracy is generated. Also,
If necessary, 2/4 pixel field interpolation image, 3
A field interpolation image with / 4 pixel accuracy is similarly generated using an 8-tap FIR filter.

As the 8-tap FIR filter, the same filter as that described above is used. That is, 1
8 when a frame interpolation image with / 4 pixel accuracy is generated
An example of a tap FIR filter is as follows. (-3 12 -37 229 71 -216-
1) // 256 An example of an 8-tap FIR filter when a frame interpolation image with 2/4 pixel accuracy is generated is as follows. (-3 12 -39 158 158 -39 12
-3) // 256 An example of an 8-tap FIR filter when a frame interpolation image with 3/4 pixel accuracy is generated is as follows. (-1-6-21 71 229 -37 12-
3) // 256

FIR is performed on a frame interpolated image having 1/4 pixel accuracy, 2/4 pixel accuracy, or 3/4 pixel accuracy.
A filter or a linear interpolation is used to generate a 1 / 8-pixel precision field interpolation image.

In this manner, an interpolated image with 1/8 pixel accuracy may be generated by using the 8-tap FIR filter, or by linear interpolation between adjacent pixels. A field interpolation image may be generated. As described above, even when an interpolated image with 1/8 pixel accuracy is generated by linear interpolation (interpolation) between adjacent pixels, first, 1/4 pixel accuracy with linear interpolation from a reference image is generated. An interpolated image is generated, and an interpolated image with 1/8 pixel accuracy is further generated from the generated interpolated image with 1/4 pixel accuracy by linear interpolation.

The interpolated images of 1/4 pixel accuracy, 2/4 pixel accuracy, 3/4 pixel accuracy, and 1/8 pixel accuracy generated in this manner are stored in the field memory 103. Only one of the interpolated images may be stored.

If necessary, 1/8, 2/8, 3/8, 4
A field image with / 8, 5/8, 6/8, and 7/8 pixel accuracy may be generated using an FIR filter and stored in the field memory 103. Thus, 1/8, 2/8, 3/8, 4/8, 5/8,
Field images with 6/8 and 7/8 pixel accuracy are FI
When an RIR filter is used for generation, examples of coefficients of the FIR filter include the following.

Coefficients of the FIR filter for 1/8 pixel precision {-3, 12, -37, 485, 71, -21, 6,-
1｝ / 512 Coefficients of FIR filter for 2/8 pixel precision {−3, 12, −37, 229, 71, −21, 6, −
1｝ / 256 Coefficients of FIR filter for 3/8 pixel precision {−6,24, −76,387,229, −60,1
8, -4} / 512 Coefficients of FIR filter for 4/8 pixel precision {-3, 12, -39, 158, 158, -39, 1
2, -3} / 256 Coefficients of FIR filter for 5/8 pixel precision {-4,18, -60,229,387, -76,2
4, -6} / 512 Coefficients of FIR filter for 6/8 pixel precision {-1,6, -21,71,229, -37,12,-
3｝ / 256 Coefficients of FIR filter for 7/8 pixel precision {-1, 6, -21,71,485, -37,12,-
3｝ / 512

As described above, the frame memory 102 and the field memory 103 store １ ／, ４, ／
Interpolated images having a plurality of resolutions such as 4, 3/4, and 1/8 are stored, respectively. The reading of the interpolated image stored in this manner, that is, the operation of the selector 104 or the selector 124 will be described. The selector 104 of the image information encoding device 100 and the image information decoding device 12
Since the operation of the selector 124 of 0 is different, the operation of the selector 104 of the image information encoding device 100 will be described first.

First, the motion prediction / compensation unit 24 finds a motion vector that minimizes a prediction error in both the frame motion prediction mode and the field motion prediction mode. Alternatively, for both the frame motion prediction mode and the field motion prediction mode, a motion vector that minimizes the cost evaluation function is obtained.

When processing the frame motion prediction mode, the selector 104 uses the frame memory 10 as a reference image.
2 is selected. On the other hand, when processing the field motion prediction mode, the selector 104 selects an interpolation image stored in the field memory 10 as a reference image.

When the accuracy of the motion vector is the 1/4 pixel accuracy mode, the motion prediction / compensation unit 24 obtains a predicted image as follows. As shown in FIG. 15, the reference image selected by the selector 104 is an interpolated image with half-pixel accuracy, and the interpolated image stored in the frame memory 102 or the field memory 103 is an interpolated image with half-pixel accuracy. In this case, the half-pixel accuracy prediction image is directly read from the half-pixel accuracy interpolation image stored in the frame memory 102 or the field memory 103, and supplied to the motion prediction / compensation unit 24. .

As shown in FIG. 15, the 1/4 pixel precision prediction image is generated by a 6-tap FIR filter or linear interpolation based on the 1/2 pixel precision interpolation image, and then is subjected to motion prediction / compensation. It is supplied to the unit 24.

As shown in FIG. 16, the reference image selected by the selector 104 is an interpolated image with 1/4 pixel accuracy.
When the interpolated image stored in the frame memory 102 or the field memory 103 is an interpolated image with １ ／ pixel accuracy, the predicted image with 画素 pixel accuracy or １ ／ pixel accuracy is stored in the frame memory 102 or field memory. The interpolated image read from the memory 103 is used as it is.

When the accuracy of the motion vector is in the 1/8 pixel accuracy mode, the motion prediction / compensation unit 24 obtains a predicted image as follows. As shown in FIG. 17, the reference image selected by the selector 104 is a １ ／ pixel precision interpolation image, and the interpolation image stored in the frame memory 102 or the field memory 103 is a 画素 pixel precision interpolation image. In the case of 予 測, as the predicted image of 画素 pixel accuracy or 画素 pixel accuracy, the 画素 pixel accuracy interpolation image stored in the frame memory 102 or the field memory 103 as shown in FIG. 17 is used as it is. .

As shown in FIG. 17, a prediction image with 1/8 pixel accuracy is based on an interpolation image with 1/4 pixel accuracy stored in the frame memory 102 or the field memory 103, or an 8-tap FIR filter or Generated by linear interpolation.

As shown in FIG. 18, the reference image selected by the selector 104 is a 1/8 pixel precision interpolation image, and the interpolation image stored in the frame memory 102 or the field memory 103 is a 1/8 pixel precision interpolation image. , The predicted image with の pixel accuracy, １ ／ pixel accuracy, or ８ pixel accuracy is the 1/8 pixel accuracy stored in the frame memory 102 or the field memory 103, respectively. A precision interpolation image is used.

Thus, the image information encoding device 10
The motion prediction / compensation unit 24 of 0 uses the interpolated image selected by the selector 104 to determine a motion vector in which the error of the predicted image is reduced in each of the frame prediction mode and the field prediction mode. Alternatively, the one in which the prediction error in the field prediction mode becomes smaller is determined as the motion vector of the macroblock.

Alternatively, in the above, the smaller the cost evaluation function in the frame prediction mode or the field prediction mode may be determined as the motion vector of the macroblock.

The image information decoding device 1 corresponding to the image information encoding device 100 for determining the motion vector in this manner
The operation of the twenty selectors 124 will be described. The image information decoding device 120 reads an interpolation image from the memory selected by the selector 124 according to the prediction mode flag of the frame or field decoded by the lossless decoding unit 43.

The selector 124 reads the interpolated image stored in the designated frame memory 122 or field memory 123 as a reference image, and the motion prediction / compensation unit 51 uses the read interpolated image and reference image. To obtain a predicted image. The method of obtaining the predicted image is performed in the same manner as in the above-described image information encoding device 100, and thus the description thereof is omitted here.

In addition, H. In 26L, there is a provision that the motion vector may point outside the VOP boundary. For that purpose, the frame memory 102 and the field memory 103 in the image information encoding device 100,
Further, the frame memory 122 and the field memory 123 in the image information decoding device 120 may be provided with a padding area (Padding area) at the VOP boundary as shown in FIG.

In the above-described embodiment, H.264 is used. 26
Although the operating principle in L has been described, the application scope of the present invention is not limited to this, and an interlaced image is used as an input signal, orthogonal transform such as discrete cosine transform, Karhunen-Loeve transform, etc. It goes without saying that the present invention can be applied to any encoding device that realizes image compression by motion prediction compensation with a selectable compensation mode, and a decoding device that decodes the image compression information.

As described above, the frame memory 102 (122) for storing the frame interpolation image and the field memory 103 (123) for storing the field interpolation image correspond to the image information encoding device 100 and the image information decoding device 12, respectively.
By providing 0, it is possible to reduce the amount of calculation for obtaining a predicted image, and to avoid inconvenience that affects other processes.

Further, in the motion prediction processing, many pixels near a predetermined area are repeatedly referred to many times, so that a predicted image can be obtained at a high speed when encoding or decoding an image signal. Can be solved.

The above series of processing can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, it is possible to execute various functions by installing a computer in which the programs constituting the software are embedded in dedicated hardware, or by installing various programs For example, it is installed from a recording medium to a general-purpose personal computer or the like. Before describing the recording medium, a personal computer that handles the recording medium will be briefly described.

FIG. 19 is a diagram showing an example of the internal configuration of a general-purpose personal computer. The CPU (Central Processing Unit) 211 of the personal computer is an RO
Various processes are executed according to a program stored in an M (Read Only Memory) 212. RAM (Random A
The ccess memory 213 stores data, programs, and the like necessary for the CPU 211 to execute various processes. The input / output interface 215 is connected to an input unit 216 including a keyboard and a mouse, and outputs a signal input to the input unit 216 to the CPU 211. The input / output interface 215 is also connected to the output unit 7 including a display, a speaker, and the like.

Further, the input / output interface 215 includes a storage unit 218 composed of a hard disk or the like,
Further, a communication unit 219 for exchanging data with another device via a network such as the Internet is also connected. The drive 220 includes a magnetic disk 231, an optical disk 232, a magneto-optical disk 233, and a semiconductor memory 23.
4 is used when reading data from or writing data to a recording medium such as a recording medium.

As shown in FIG. 19, the recording medium is a magnetic disk 231 (including a flexible disk) on which the program is recorded, which is distributed in order to provide the user with the program, separately from a personal computer. (CD-ROM (Compact Disc-Read Only Mem
ory), a DVD (including a Digital Versatile Disc), a magneto-optical disk 233 (including an MD (Mini-Disc) (registered trademark)), or a package medium including a semiconductor memory 234, as well as a computer. The storage unit 218 includes a hard disk including a ROM 212 storing a program and a storage unit 218, which are provided to a user in a state where the hard disk is incorporated in advance.

[0108] In this specification, the steps of describing a program provided by a medium may be performed in a chronological order according to the described order. Alternatively, it also includes individually executed processing.

In the present specification, the system is
It represents the entire device composed of a plurality of devices.

[0110]

According to the image processing apparatus, method and program of the present invention, image data of a predetermined resolution is obtained by processing using an FIR filter or processing using linear interpolation. Image data necessary for performing prediction compensation is generated, and when the generated image data is image data required for performing the frame prediction compensation mode, the image data is stored in a frame memory that stores the image data. When the image data required when performing the field prediction compensation mode is stored in the field memory, the motion prediction It is possible to reduce the amount of calculation when acquiring, and to perform encoding or decoding of image data at high speed. The ability.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration of an example of a conventional image information encoding device.

FIG. 2 is a diagram illustrating a configuration of an example of a conventional image information decoding device.

FIG. 3 is a diagram for explaining a frame motion prediction compensation mode.

FIG. 4 is a diagram for explaining a field motion prediction compensation mode.

FIG. 5 is a diagram for explaining motion compensation from outside a VOP boundary.

FIG. 6 is a diagram illustrating a motion prediction compensation process with quarter-pixel accuracy.

FIG. 7 is a diagram illustrating a motion prediction compensation process with 1/8 pixel accuracy.

FIG. 8 is a diagram illustrating an interpolation method with 1/8 pixel accuracy.

FIG. 9 is a diagram illustrating a configuration of an embodiment of an image information encoding device to which the present invention has been applied.

FIG. 10 is a diagram illustrating an image information decoding apparatus to which the present invention has been applied.

FIG. 11 is a diagram for describing generation of a frame interpolation image with 1/2 and 1/4 pixel accuracy.

FIG. 12 is a diagram for explaining generation of a field-interpolated image with 1/2 and 1/4 pixel accuracy.

FIG. 13 is a diagram for describing generation of a frame interpolation image with 1/4 and 1/8 pixel accuracy.

FIG. 14 is a diagram for describing generation of a field interpolation image with 1/4 and 1/8 pixel accuracy.

FIG. 15 is a diagram for describing acquisition of a predicted image with 、 and 画素 pixel accuracy.

FIG. 16 is a diagram for describing acquisition of a predicted image with 、 and 画素 pixel accuracy.

FIG. 17 is a diagram for describing acquisition of a predicted image with 、, ４, and ８ pixel accuracy.

FIG. 18 is a diagram for describing acquisition of a predicted image with 、, ４, and ８ pixel accuracy.

FIG. 19 is a diagram illustrating a medium.

[Explanation of symbols]

100 image information encoding device, 101 image generation unit,
102 frame memory, 103 field memory, 104 selector, 120 image information decoding device, 121 image generator, 122 frame memory, 123 field memory, 124 selector

   ────────────────────────────────────────────────── ─── Continuation of front page    (72) Inventor Yoichi Yagasaki             6-7-35 Kita Shinagawa, Shinagawa-ku, Tokyo Soni             ー Inc. F term (reference) 5C059 KK10 KK15 LA05 MA00 MA03                       MA05 MA21 NN21 PP04 SS02                       SS08 SS13 SS20 TA08 TA62                       TA65 TB07 TC03 UA11 UA33                       UA39                 5J064 AA00 AA03 BA04 BA16 BB01                       BB04 BC01 BC06 BC07 BC08                       BC12 BC16 BC25 BC29 BD02                       BD04

Claims (21)

[Claims] 1. An image processing apparatus which receives image information of interlaced scanning and performs motion prediction compensation by selecting a frame prediction compensation mode or a field prediction compensation mode in units of orthogonal transform and macroblocks. Or, by processing using linear interpolation, generating means for generating image data of a resolution set in advance and necessary for performing the motion prediction compensation, and the generating means generated by the generating means When the image data is image data required for performing the frame prediction compensation mode, a frame memory for storing the image data; and the image data generated by the generation unit is the field prediction compensation mode. When the image data is required when performing the above, a field for storing the image data An image processing apparatus comprising: a memory; and a prediction compensation unit that performs the motion prediction compensation using the image data stored in the frame memory or the field memory. 2. When the motion prediction compensation performed by the prediction compensation means is the frame prediction compensation mode, the image data is read from the frame memory. When the motion prediction compensation is the field prediction compensation mode, the image data is read from the field memory. The image processing apparatus according to claim 1, further comprising a reading unit for reading. 3. The image according to claim 1, wherein the prediction compensation unit determines a motion vector that minimizes a prediction error in each of the frame prediction compensation mode and the field prediction compensation mode. Processing equipment. 4. The image processing apparatus according to claim 3, wherein the prediction compensation unit further determines the motion vector having the smallest prediction error as a motion vector of a macroblock to be processed. apparatus. 5. The apparatus according to claim 1, wherein the prediction compensation means determines a motion vector having a minimum value of a cost evaluation function in each of the frame prediction compensation mode and the field prediction compensation mode. The image processing apparatus according to any one of the preceding claims. 6. The method according to claim 5, wherein the prediction compensation unit further determines the motion vector having the smallest value of the cost evaluation function as a motion vector of a macroblock to be processed. Image processing device. 7. The image processing apparatus according to claim 1, wherein said frame memory and said field memory separately store image data of a plurality of resolutions. 8. The image processing apparatus according to claim 1, wherein the resolution is 精度, ４, or ８ pixel accuracy of the input image information. 9. The image processing apparatus according to claim 1, wherein the generation unit generates the image data of at least one of the field images having the accuracy of ま た は or 画素 pixel of the input image information by linear interpolation between fields. The image processing device according to claim 1. 10. The image processing apparatus according to claim 1, wherein the generating unit converts the image data of at least one of the field images having 1/2 and 1/4 pixel precision of the input image information using an FIR filter having a predetermined number of taps between fields. The image processing apparatus according to claim 1, wherein the image processing apparatus generates the image data. 11. The image processing apparatus according to claim 1, wherein when // represents a rounding operation, the generation unit generates 1/1/3 of the input image information.
2. A field image with two-pixel accuracy is generated using an FIR filter that performs an operation represented by the following equation: {1, -5, 20, 20, -5, 1} // 32. An image processing apparatus according to claim 1. 12. The method according to claim 1, wherein the generation unit generates a quarter-pixel precision field image of the input image information from the half-pixel precision field image using linear interpolation between fields. The image processing apparatus according to claim 1. 13. The image processing apparatus according to claim 1, wherein the generation unit converts the image data of at least one field image of 1/4, 2/4, and 3/4 pixel precision of the input image information into a predetermined number of fields between fields. The image processing apparatus according to claim 1, wherein the image is generated using a tap FIR filter. 14. The image forming apparatus according to claim 1, wherein when // represents a rounding operation, 1/1 of the input image information is obtained.
Field images of 4, 2/4, and 3/4 pixel accuracy are calculated as follows: ｛−3, 12, −37, 229, 71, −21, 6, −
1｝ // 256 ｛-3,12, -37,229,71, -21,6,-
1｝ // 256 ｛-1,6, -21,71,229, -37,12,-
The image processing apparatus according to claim 1, wherein the image is generated using an FIR filter that performs an operation represented by an expression of 3｝ / 256. 15. The image processing apparatus according to claim 1, wherein the generation unit converts a field image having 1/8 pixel precision of the input image information into 1/4, 2
2. The image processing apparatus according to claim 1, wherein the image is generated by using linear interpolation between fields from any of the field images having ４ or ／ pixel accuracy. 3. 16. The image processing apparatus according to claim 16, wherein the generation unit is configured to output 1/8, 2/8, 3/8, 4/8, 5/8, 6/6 of the input image information.
2. The image processing method according to claim 1, wherein image data of at least one field image of 8 and 7/8 pixel precision is generated by using an FIR filter having a predetermined number of taps between fields. apparatus. 17. The image processing apparatus according to claim 17, wherein when // represents a rounding operation, // the generation unit outputs 1/1 / of the input image information.
8, 2/8, 3/8, 4/8, 5/8, 6/8, and 7/8 pixel-accurate field images are represented by {-3,12, -37,485,71, -21,6}. , −
1｝ // 512１２-3,12, -37,229,71, -21,6,-
1｝ // 256 ｛−6, 24, −76, 387, 229, −60, 1
8, -4｝ // 512１２-3,12, -39,158,158, -39,1
2, -3｝ // 256２-4,18, -60,229,387, -76,2
4, -6｝ // 512 ｛-1,6, -21,71,229, -37,12,-
3｝ // 256 ｛-1,6, -21,71,485, -37,12,-
The image processing apparatus according to claim 1, wherein the image is generated using an FIR filter that performs an operation represented by an expression of 3｝ / 512. 18. The apparatus according to claim 1, wherein the frame memory and the field memory have a padding area for supporting motion compensation from outside a VOP boundary.
An image processing apparatus according to claim 1. 19. An image processing method for an image processing apparatus for inputting interlaced image information and performing motion prediction compensation by selecting a frame prediction compensation mode or a field prediction compensation mode in units of orthogonal transform and macroblocks. A process using image processing or a process using linear interpolation to generate image data of a resolution set in advance and image data necessary for performing the motion prediction compensation, When the image data generated in the process is image data required when performing the frame prediction compensation mode, a first storage control step of controlling storage of the image data in a frame memory. Wherein the image data generated in the processing of the generation step performs the field prediction compensation mode. When the image data is required in the case, a storage control step of controlling storage in a field memory that stores the image data, using the image data stored in the frame memory or the field memory, A prediction compensation step of performing the motion prediction compensation. 20. A program for an image processing apparatus for inputting image information of interlaced scanning and performing motion prediction compensation by selecting a frame prediction compensation mode or a field prediction compensation mode in units of orthogonal transform and macroblocks. A generation step of generating image data that is image data of a preset resolution and that is necessary for performing the motion prediction compensation, by processing using linear interpolation or processing using linear interpolation. When the image data generated in the process is image data required when performing the frame prediction compensation mode, a first storage control step of controlling storage of the image data in a frame memory. The image data generated in the processing of the generation step performs the field prediction compensation mode When the image data is required, a storage control step of controlling storage in a field memory that stores the image data, and using the image data stored in the frame memory or the field memory, A prediction compensation step of performing motion prediction compensation. 21. A computer that inputs image information of interlaced scanning and controls an image processing apparatus that performs motion prediction compensation by selecting a frame prediction compensation mode or a field prediction compensation mode in units of orthogonal transform and macroblocks. A generation step of generating image data that is image data of a preset resolution and that is necessary for performing the motion prediction compensation, by processing using linear interpolation or processing using linear interpolation. When the image data generated in the process is image data required when performing the frame prediction compensation mode, a first storage control step of controlling storage of the image data in a frame memory. The image data generated in the processing of the generation step performs the field prediction compensation mode When the image data is needed in the case, a storage control step of controlling storage in a field memory that stores the image data, and using the image data stored in the frame memory or the field memory, A prediction compensation step of performing the motion prediction compensation.