JP2003348596A - Image processor and image processing method, recording medium, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a calculation amount for acquiring a predictive image. <P>SOLUTION: When a motion predicting and compensating part 24 needs the predictive image P of a prescribed part in a reference frame and has to read image data from an interpolated image buffer 101 for the predictive image P, the motion predicting and compensating part 24 reads data of an area P' corresponding to the predictive image P from the image data stored in the interpolated image buffer 101. When a divided area S (P) including the area corresponding to the predictive image P has already been written to the interpolated image buffer 101, a value stored in the divided area S (P) is used to acquire the predictive image P. The present invention is applicable to an encoding device and a decoding device which perform processing that needs a predictive image. <P>COPYRIGHT: (C)2004,JPO

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly, to image information (bit stream) compressed by orthogonal transform such as discrete cosine transform or Karhunen-Loeve transform and motion compensation. A, satellite broadcasting, cable television broadcasting,
Encodes and decodes image information used when transmitting and receiving via network media such as the Internet, or processing on storage media such as optical disks, magnetic disks, and flash memories, and converts update frequencies. The present invention relates to an image processing apparatus and method suitable for use in an apparatus, a recording medium, and a program.

[0002]

2. Description of the Related Art In recent years, image information has been handled as digital data.
MPEG (Moving Pic) that compresses by orthogonal transform such as discrete cosine transform and motion compensation using the redundancy inherent in image information
devices based on a system such as a network expert group are becoming widespread in both information distribution of broadcast stations and the like and information reception in ordinary households.

[0003] In particular, MPEG2 (ISO / IEC 13818-2) is a standard defined as a general-purpose image compression method, and is a standard covering both interlaced scan images and progressive scan images, as well as standard resolution images and high definition images. For example, DVD (Digit
al Versatile Disk), and is widely used in a wide range of applications for professional use and consumer use.

[0004] By using this MPEG2 compression method,
For example, for a standard resolution interlaced scan image having 720 × 480 pixels, 4 to 8 Mbps and 1920 × 1
By assigning a code amount (bit rate) of 18 to 22 Mbps to a high-resolution interlaced scan image having 088 pixels, a high compression rate and good image quality can be realized.

[0005] MPEG2 was mainly intended for high-quality encoding suitable for broadcasting, but did not support an encoding system with a higher compression rate. Therefore, the MPEG4 encoding system was standardized. Regarding the image coding method, 1998
In December 2000, the standard was approved as an international standard as ISO / IEC 14496-2.

Further, in recent years, with the initial purpose of image coding for video conferences, ITU-T (International Telecommunication Uni
The standardization of H.26L (ITU-T Q6 / 16VCEG) by the on-telecommunication standardization sector is in progress. H. It is known that 26L requires a greater amount of computation for encoding and decoding than encoding schemes such as MPEG2 and MPEG4, but realizes higher encoding efficiency.

[0007] Currently, as part of MPEG4 activities,
This H. H. 26L. Standardization of coding technology that achieves higher coding efficiency by incorporating functions that are not supported by 26L has been jointly implemented with ITU-T by JVT (Joint
Video Team).

Here, image compression by orthogonal transform such as discrete cosine transform or Karhunen-Loeve transform and motion compensation will be described. FIG. 1 is a diagram showing a configuration of an example of a conventional image information encoding device.

In the image information encoding device 10 shown in FIG. 1, image information composed of an analog signal input from an input terminal 11 is converted into a digital signal by an A / D converter 12. Then, the screen rearrangement buffer 13
The GOP (Gr) of the image information supplied from the A / D converter 12
oup of Pictures) The frames are rearranged according to the structure.

[0010] Here, the screen rearrangement buffer 13 supplies image information of the entire frame to the orthogonal transformation unit 15 for an image on which intra (intra-image) encoding is performed. The orthogonal transform unit 15 performs orthogonal transform such as discrete cosine transform or Karhunen-Loeve transform on the image information, and supplies transform coefficients to the quantization unit 16. The quantization unit 16 performs a quantization process on the transform coefficient supplied from the orthogonal transform unit 15.

The lossless encoding unit 17 determines an encoding mode from the quantized transform coefficients and the quantization scale supplied from the quantization unit 16, and performs variable length encoding or Lossless coding such as arithmetic coding is performed to form information to be inserted into a header portion of an image coding unit. Then, the lossless encoding unit 17 supplies the encoded encoding mode to the accumulation buffer 18 and accumulates the encoded encoding mode. This encoded encoding mode is output to the output terminal 1 as image compression information.
9 is output.

The lossless encoding unit 17 performs lossless encoding such as variable-length encoding or arithmetic encoding on the quantized transform coefficients, and supplies the encoded transform coefficients to the storage buffer 18. And accumulate. The encoded transform coefficient is output from the output terminal 19 as image compression information.

The behavior of the quantization unit 16 is determined by the accumulation buffer 18
Is controlled by the rate control unit 20 based on the data amount of the conversion coefficient accumulated in the. Also, the quantization unit 20
The quantized transform coefficient is supplied to the inverse quantization unit 21, and the inverse quantization unit 21 inversely quantizes the quantized transform coefficient.
The inverse orthogonal transform unit 22 performs an inverse orthogonal transform process on the inversely quantized transform coefficient to generate decoded image information, and supplies the information to the frame memory 23 for storage.

The screen rearrangement buffer 13 supplies image information to the motion prediction / compensation unit 24 for an image to be subjected to inter (inter-image) encoding. The motion prediction / compensation unit 24 extracts image information to be referred to simultaneously from the frame memory 23 and performs motion prediction / compensation processing to generate reference image information. The motion prediction / compensation unit 24 supplies the generated reference image information to the adder 14, and the adder 14
The reference image information is converted into a difference signal from the corresponding image information. The motion prediction / compensation unit 24 supplies the motion vector information to the lossless encoding unit 17 at the same time.

The lossless encoding unit 17 determines an encoding mode based on the quantized transform coefficient and quantization scale supplied from the quantization unit 16, the motion vector information supplied from the motion prediction / compensation unit 24, and the like. Then, lossless coding such as variable length coding or arithmetic coding is performed on the determined coding mode to generate information to be inserted into the header part of the image coding unit. Then, the lossless encoding unit 17 supplies the encoded encoding mode to the accumulation buffer 18 and accumulates the encoded encoding mode. This encoded encoding mode is output as image compression information.

The lossless encoding unit 17 performs a lossless encoding process such as variable-length encoding or arithmetic encoding on the motion vector information to generate information to be inserted into a header portion of an image encoding unit. I do.

Also, unlike the intra coding, in the case of the inter coding, the image information input to the orthogonal transform unit 15 is a difference signal obtained from the adder 14. The other processing is the same as that of the image compression information to be subjected to the intra coding, and the description thereof will be omitted.

Next, FIG. 2 shows an example of the configuration of an image information decoding device corresponding to the above-described image information encoding device 10.
In the image information decoding device 40 shown in FIG. 2, the image compression information input from the input terminal 41 is stored in a storage buffer 42.
, And then transferred to the lossless decoding unit 43.

The lossless decoding unit 43 performs processing such as variable-length decoding or arithmetic decoding on the image compression information based on the determined format of the image compression information, and obtains the encoding mode information stored in the header part. Then, it is supplied to the inverse quantization unit 44 and the like. Similarly, the lossless decoding unit 43 obtains the quantized transform coefficient and supplies it to the inverse quantization unit 44. Furthermore, if the frame to be decoded is inter-coded, the lossless decoding unit 43 also decodes the motion vector information stored in the header part of the image compression information, and decodes the information into the motion prediction / compensation unit. 51.

The inverse quantization section 44 inversely quantizes the quantized transform coefficient supplied from the lossless decoding section 43 and supplies the transform coefficient to the inverse orthogonal transform section 45. The inverse orthogonal transform unit 45 performs an inverse discrete cosine transform or an inverse Karhunen transform on the transform coefficients based on the format of the determined image compression information.
Performs an inverse orthogonal transform such as a Loeve transform.

Here, if the target frame has been intra-coded, the image information subjected to the inverse orthogonal transform processing is stored in the screen rearrangement buffer 47,
The signal is output from the output terminal 49 after the D / A conversion processing in the / A conversion section 48.

When the target frame is inter-coded, the motion prediction / compensation unit 51 converts the motion vector information subjected to the lossless decoding process into the image information stored in the frame memory 50. A reference image is generated based on the reference image and supplied to the adder. The adder 46 combines the reference image and the output from the inverse orthogonal transform unit 45. Note that the other processing is the same as that of the intra-coded frame, and a description thereof will not be repeated.

FIG. 3 is a diagram showing an example of the configuration of an image information conversion device 70 for converting the update frequency of the image information signal by motion prediction. Image information conversion device 70 shown in FIG.
Is a motion prediction unit 71, a selector 72, a frame memory 7
3, an interpolation image generation unit 74 and a delay buffer 75.

In the image information conversion device 70 shown in FIG. 3, a motion estimating unit 71 estimates a motion between frames based on a reference frame stored in a frame memory 73 and input image information. From the motion prediction determined by the motion prediction unit 71, the interpolation image generation unit 74 generates an interpolation image. The generated interpolation image is temporarily stored in the delay buffer 75. The selector 72 outputs image information by appropriately switching between the input image and the interpolated image stored in the delay buffer 75 according to the target update frequency.

By such processing, for example, as shown in FIG. 4, an interpolation frame is inserted between input image information,
The update frequency can be increased by increasing the number of frames. Conversely, as shown in FIG. 5, it is possible to reduce the update frequency by deleting input image information (deleting input frames), inserting interpolation frames, and reducing the number of frames. That is, by performing such processing, the frame rate is converted in the image information conversion device 70.

By the way, in MPEG4, as shown in FIG. 6, a motion vector is a VOP (Video Object P).
lane) It is specified that it may point outside the boundary.
When the region specified by the motion vector is outside the VOP boundary, information on the pixels located on the VOP boundary is used as the predicted value. H. 26L also stipulates that the motion vector may point outside the VOP boundary.

H. In 26L, a high-precision motion prediction compensation process such as 1/4 or 1/8 pixel is specified. In order to generate the decimal precision prediction image, it is prescribed that a several tap filter and linear interpolation are combined.

In the following, H. 26L
The motion prediction compensation processing with 4, 1/8 pixel accuracy will be described. FIG. It is a figure for explaining motion prediction compensation processing of 1/4 pixel precision defined in 26L. First, based on the pixels stored in the frame memory, FIR (Finite) of 6 taps in each of the horizontal direction and the vertical direction is used.
Using an Impulse Response) filter, a pixel value with 1/2 pixel accuracy is generated. The following is defined as an example of the FIR filter coefficient. (1-5 20 20-5 1) // 32 In this FIR filter coefficient, // indicates that the division is a division with rounding (rounding). In this specification, // indicates a rounded division.

A pixel value of 1/4 pixel precision is obtained by linear interpolation from two adjacent pixel values of 1/2 pixel precision obtained above.

FIG. 1 defined in 26L
It is a figure for explaining motion prediction compensation processing of / 8 pixel accuracy. First, based on the pixels stored in the frame memory, FI taps of 8 taps each in the horizontal and vertical directions are used.
Using the R filter, pixel values with １ ／, ／, and ／ pixel accuracy are generated. The following are defined as FIR filter coefficients. (-3 12 -37 229 71 -216-
1) // 256 (-3 12 -39 158 158 -39 12
−3) // 256 (−1−6 −21 71 229 −37 12 −)
3) // 256

The pixel value of 1/8 pixel precision is obtained by dividing the pixel value of 1/4, 2/4, and 3/4 pixel precision generated as described above into two pixel values as shown in FIG. Obtained by linear interpolation.

[0032]

SUMMARY OF THE INVENTION In a coding apparatus capable of selecting frame motion prediction compensation or field motion prediction compensation on a macroblock basis, and in a decoding apparatus for decoding image compression information from the coding apparatus, the motion prediction compensation is used. When obtaining a predicted image, the amount of calculation for obtaining a predicted image with decimal precision becomes a problem. That is, the calculation of the interpolation pixel with decimal precision has been performed by the several tap filter and the linear interpolation as described above. However, there is a problem that performing these calculations for each pixel is a heavy process and may affect other processes.

In particular, in the motion prediction processing, many pixels located in the vicinity of a predetermined area are repeatedly referred to many times. Therefore, when encoding or decoding an image signal, a predicted image is It is important to acquire at high speed, but there is a problem that it is difficult.

The present invention has been made in view of such circumstances, and it is an object of the present invention to reduce processing for calculating an interpolated pixel and to obtain the interpolated pixel at high speed.

[0035]

An image processing apparatus according to the present invention comprises a generating means for generating image data with a predetermined sub-pel precision by processing using an FIR filter or processing using linear interpolation. Storage means for storing image data generated by the generation means, and prediction compensation means for executing prediction compensation processing using the image data stored in the storage means.

[0036] The generating means may be a unit for storing the input image information.
It is possible to generate image data with / 2 pixel accuracy.

When // represents a rounding operation, the generation means converts image data having half-pixel accuracy of the input image information into {1, -5, 20, 20, -5, 1 It can be generated using an FIR filter that performs an operation represented by the formula of｝ // 32.

[0038] The generation means may be a unit for storing one of the input image information.
It is possible to generate image data with / 4 pixel accuracy.

[0039] The generation means may generate image data of 1/4 pixel accuracy of the image information from the image data of 1/2 pixel accuracy using an FIR filter having a predetermined number of taps.

When the // represents a rounding operation, the generation means converts {1, -5, 20, 20, -5, 1} into image data of 1/2 pixel precision of the image information. 32 using an FIR filter that performs the operation represented by the equation
Image data with 4-pixel accuracy can be generated.

[0041] The generating means may generate the image data of 1/4 pixel precision of the image information from the image data of 1/2 pixel precision by linear interpolation.

[0042] The generating means may output 1/4, 2 /
Image data with 4, 3/4 pixel accuracy can be generated using an FIR filter with a predetermined number of taps.

When the // represents a rounding operation, the generation means converts the field image of 1/4, 2/4, and 3/4 pixel precision of the image information into ｛−3, 12, and −37. , 229,71, -21,6,-
1｝ // 256 ｛-3,12, -39,158,158, -39,1
2, -3｝ // 256 ｛-1,6, -21,71,229, -37,12,-
It can be generated using an FIR filter that performs the operation represented by the formula of 3｝ / 256.

[0044] The generating means may generate image data of 1/4 pixel accuracy of the image information by linear interpolation.

[0045] The generation means may generate the image data of 1/8 pixel accuracy of the image information from the image data of 1/4 pixel accuracy using a FIR filter having a predetermined number of taps.

When // represents a rounding operation, the generation means converts {1, -5, 20, 20, -5, 1} into image data of 1/4 pixel precision of image information. 32 using an FIR filter that performs the operation represented by the equation
It is possible to generate image data with 8-pixel accuracy.

[0047] The generating means may generate the image data of 1/8 pixel accuracy of the image information from the image data of 1/4 pixel accuracy by linear interpolation.

[0048] The generating means may generate the image data of 1/8 pixel accuracy of the image information by linear interpolation.

[0049] The generating means may generate image data of 1/4 pixel accuracy of the image information from the image data of 1/2 pixel accuracy using an FIR filter having a predetermined number of taps.

The prediction compensating means stores the image data of 1/2 pixel precision of the image information in the storage means, and stores the image data of 1/2 pixel precision when the prediction compensation requires the image data. The half-pixel precision image data can be read out and used as it is.

In the prediction compensating means, image data of 1/2 pixel precision of the image information is stored in the storage means, and when prediction compensation requires image data of 1/4 pixel precision, the data is stored in the storage means. It is possible to read out image data with 1/2 pixel accuracy and generate image data with 1/4 pixel accuracy by using an FIR filter with a predetermined number of taps on the image data to perform prediction compensation. .

When // represents a rounding operation, the prediction compensation means converts {1, -5, 20, 20, -5, 1} / {1, -5, 20, 20, -5, 1} into image data of 1/2 pixel precision of image information. / 32 using an FIR filter that performs the operation represented by the equation
Image data with 4-pixel accuracy can be generated.

The prediction compensating means stores image data of 1/2 pixel precision of the image information in the storage means, and when the prediction compensation requires image data of 1/4 pixel precision, the data is stored in the storage means. It is possible to read out the image data with 1/2 pixel accuracy and generate the image data with 1/4 pixel accuracy using linear interpolation on the image data to perform prediction compensation.

In the prediction compensating means, image data of 1/4 pixel precision of the image information is stored in the storage means, and when prediction prediction requires image data of 1/4 pixel precision, the image data is stored in the storage means. It is possible to read out the image data of 1/4 pixel precision and use it as it is.

In the prediction compensating means, image data of 1/4 pixel precision of the image information is stored in the storage means, and when prediction compensation requires image data of 1/8 pixel precision, it is stored in the storage means. It is possible to read out image data with 1/4 pixel accuracy and generate image data with 1/8 pixel accuracy by using an FIR filter with a predetermined number of taps on the image data to perform prediction compensation. .

The prediction compensation means, when // represents a rounding operation, outputs {1, -5, 20, 20, -5, 1} / { / 32 using an FIR filter that performs the operation represented by the equation
It is possible to generate image data with 8-pixel accuracy.

In the prediction compensating means, image data of 1/4 pixel accuracy of image information is stored in the storage means, and when prediction compensation requires image data of 1/8 pixel accuracy, the data is stored in the storage means. It is possible to read out the image data with 1/4 pixel precision and generate image data with 1/8 pixel precision by using linear interpolation on the image data to perform prediction compensation.

In the prediction compensating means, image data of 1/8 pixel accuracy of image information is stored in the storage means, and when prediction compensation requires image data of 1/8 pixel accuracy, the data is stored in the storage means. Image data with 1/8 pixel accuracy can be read and used as it is.

[0059] The storage means may have a padding area for coping with motion compensation from outside the VOP boundary.

[0060] The prediction compensation means may perform orthogonal transform by discrete cosine transform or Karhunen-Loeve transform and overlap motion prediction compensation with decimal pixel precision.

The image processing method according to the present invention comprises a generating step of generating image data with a predetermined decimal pixel precision by a processing using an FIR filter or a processing using linear interpolation, and a processing in the generating step. It is characterized by including a storage control step of controlling the storage of the generated image data, and a prediction compensation step of executing a prediction compensation process using the image data whose storage has been controlled in the processing of the storage control step.

The program of the recording medium of the present invention is
A generation step of generating image data with a predetermined sub-pixel accuracy by a processing using a filter or a processing using linear interpolation, and a storage control for controlling storage of the image data generated in the processing of the generation step And a prediction compensation step of executing a prediction compensation process using the image data whose storage has been controlled in the process of the storage control step.

The program of the present invention is stored in a computer.
A generation step of generating image data with a predetermined sub-pixel accuracy by processing using an FIR filter or processing using linear interpolation, and storage for controlling storage of the image data generated in the processing of the generation step A control step and a prediction compensation step of executing a prediction compensation process using the image data whose storage has been controlled in the process of the storage control step are performed.

The second image processing apparatus according to the present invention is characterized in that it comprises a holding means for holding a pixel value subjected to an operation relating to the motion prediction compensation processing.

The update frequency is 25 Hz to 50 Hz
Can be converted to

In the image processing apparatus, method, and program according to the present invention, the generated image data with decimal pixel precision is temporarily stored, and the stored image data is read out as necessary, thereby providing prediction compensation. Is performed.

In the second image processing apparatus according to the present invention,
The pixel value subjected to the operation related to the motion compensation processing is held.

[0068]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 10 is a diagram showing a configuration of an embodiment of an image information encoding device to which the image processing device of the present invention is applied. In the image information encoding device 100 shown in FIG. 10, blocks having the same functions as those of the image information encoding device 10 shown in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

The image information coding apparatus 100 shown in FIG.
Is configured such that data output from the frame memory 23 is supplied to the motion prediction / compensation unit 24 via the interpolation image buffer 101.

The other parts of the configuration are the same as those of the image information encoding device 10 shown in FIG.

The image information coding apparatus 100 shown in FIG.
FIG. 11 shows a configuration of an embodiment of an image information decoding apparatus of an image processing apparatus to which the present invention is applied. In the image information decoding device 120 shown in FIG. 11, blocks having the same functions as those of the image information decoding device 40 shown in FIG. 2 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

The image information decoding device 120 shown in FIG.
Is configured such that data output from the frame memory 50 is supplied to the motion prediction / compensation unit 51 via the interpolation image buffer 121.

The configuration of the other parts is the same as that of the image information decoding device 40 shown in FIG. 2, and the description thereof will be omitted.

In this embodiment, the image data stored in the frame memory 23 of the image information encoding apparatus 100 shown in FIG. 10 is supplied to the motion prediction / compensation unit 24 via the interpolation image buffer 101. Operation and FIG.
Frame memory 50 of the image information decoding apparatus 120 shown in FIG.
The operation performed until the image data stored in the memory is supplied to the motion prediction / compensation unit 51 via the interpolation image buffer 121 is basically performed in the same manner.

Here, in consideration of the above, FIG.
The image data stored in the frame memory 23 of the image information encoding device 100 shown in FIG.
1 will be described as an example, and the image data stored in the frame memory 50 of the image information decoding apparatus 120 shown in FIG. The description of the operation up to the supply to the motion prediction / compensation unit 51 via the.

The interpolation image buffer 101 of the image information encoding device 100 holds the same number of frames as the frame memory 23. Further, the interpolation image buffer 121 of the image information decoding device 120 holds the same number of frames as the frame memory 50.

However, the size of the picture frame per frame depends on the interpolation pixel precision. That is, when the interpolated image buffer 101 (121) holds an interpolated image with half-pixel accuracy, the number of pixels in both the vertical and horizontal directions is twice as large as the size of the image frame stored in the frame memory 23 (50). It becomes a frame with a number.

The interpolation image buffer 101 (121)
However, when an interpolated image with 1/4 pixel precision is held, the image frame has four times the number of pixels in each of the vertical and horizontal directions as compared with the size of the image frame stored in the frame memory 23 (50).

The interpolation image buffer 101 (121)
However, when an interpolated image with 1/8 pixel accuracy is held, the image frame has eight times the number of pixels in each of the vertical and horizontal directions as compared with the size of the image frame stored in the frame memory 23 (50).

As shown in FIG. 12, the interpolated image buffer 101 (121) is divided by a rectangle of a fixed size of M × N. The size of the divided region may be equal to or larger than the size of the block or the macroblock. Alternatively, the size of the divided area may be smaller than a block or a macroblock. That is, the size of the divided area may be set to a size suitable for the system.

First, each divided area is initialized as undefined.

With reference to FIG.
When the compensating unit 24 needs to process the predicted image P of a predetermined portion in the reference frame for processing and needs to read the predicted image P from the interpolated image buffer 101, the compensating unit 24 calculates the predicted image P from the image data stored in the interpolated image buffer 101. Is read out from the area P 'corresponding to. Only the data corresponding to the area P ′ may be read, or the data of the divided area S (P) including the area P ′ may be read.

When reading the divided area S (P), it is necessary to set in advance how many areas the frame is divided into. If it is set, the divided area S (P) including the area P ′ may be read.

When the divided area S (P) is not defined, an interpolated image is generated from the corresponding area of the frame memory 23 by the defined interpolation calculation, and as shown in FIG. (P). The predicted image P requested by the motion prediction / compensation unit 24
Is obtained from the interpolated image buffer 101 as shown in FIG.

When the divided area S (P) including the area corresponding to the predicted image P has already been written in the interpolation image buffer 101, the data stored in the divided area S (P) is used to generate the predicted image P Is acquired.

As shown in FIG. 15, when the region P ′ corresponding to the predicted image P is included in a plurality of divided regions S (P), the above-described processing is performed on each divided region S (P). Just do it.

Here, when the interpolation accuracy is set to the 1/4 pixel accuracy mode and the image data up to the 1/4 image accuracy is set in the interpolation image buffer 101, the 1/4 pixel accuracy mode is set as the prediction image. When accuracy is required, the image data is directly read from the image data stored in the interpolation image buffer 101 and used.

Alternatively, 補 間 is added to the interpolation image buffer 101.
When it is set that image data up to image accuracy is stored, when image data with 1/2 pixel accuracy is required as a predicted image, the image data read from the interpolation image buffer 101 is used as it is, 1 /
When image data with 4-pixel accuracy is required, the image data read from the interpolated image buffer 101 is set as an intermediate value, and 1 / 4-pixel accuracy is obtained by calculation such as linear interpolation.

When the interpolation accuracy is set to the 1/8 pixel accuracy mode and the interpolation image buffer 101 is set to store the image data up to the 1/8 image accuracy, the 1/8 pixel accuracy is set as the prediction image. When requested, the image data is directly read from the image data stored in the interpolation image buffer 101 and used.

Alternatively, 1/4 is stored in the interpolation image buffer 101.
When it is set that image data up to image accuracy is stored, when image data with 1/4 pixel accuracy is required as a predicted image, the image data read from the interpolation image buffer 101 is used as it is, When the prediction image requires image data with 1/8 pixel accuracy, the image data read from the interpolation image buffer 101 is set as an intermediate value, and further calculated by linear interpolation or the like to obtain 1/8 pixel accuracy. An image is required.

When motion compensation from outside the VOP boundary is permitted, a padding area is provided around the interpolation image buffer 101 as shown in FIG.
The padding area may be set so as to be included in the image stored in No. 1), and may be handled in the same manner as described above.

FIG. 17 is a diagram showing the configuration of an embodiment of the image information conversion device 131 to which the present invention is applied. FIG.
The image information conversion device 131 shown in FIG. 7 includes a motion prediction / compensation unit 132, a selector 133, a frame memory 134, an interpolation image buffer 135, and a delay buffer 136.

In the image information conversion device 131 shown in FIG. 17, the motion estimating unit 132 estimates the motion between frames based on the input image information and the reference frame stored in the interpolated image buffer 135, and Generate
The generated interpolation image is temporarily stored in the delay buffer 136. The selector 133 appropriately switches and outputs the input image information and the image information of the interpolation image stored in the delay buffer 136 according to the target update frequency.

By such processing, for example, as shown in FIG. 4, an interpolation frame is inserted between input image information,
The update frequency can be increased by increasing the number of frames. Conversely, as shown in FIG. 5, it is also possible to lower the update frequency by deleting the input image information (deleting the input frame), inserting an interpolation frame, and reducing the number of frames as a whole. That is,
By performing such processing, the image information conversion device 1
At 31, the frame rate is converted.

By inserting one interpolated image for each input image, an image signal having an update frequency of 25 Hz is changed to 50 Hz.
Can be converted into an image signal having the update frequency of

In the above-described embodiment, the operation principle of the block matching method has been described as an example. However, the range to which the present invention can be applied is not limited to this, and may be applied to other motion prediction / compensation methods. Is also applicable. For example, when the size of the divided area S (P) is set to be larger than the size of the macroblock, overlap motion compensation (Michael T. Orchard and Gary J. Sullivan; O
verlapped Block Motion Compensation: An Estimation
-Theoretic Approach; IEEE Transactions on image pr
ocessing, vol 3. No. 5, September 1994).

The above series of processing can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, it is possible to execute various functions by installing a computer in which the programs constituting the software are embedded in dedicated hardware, or by installing various programs For example, it is installed from a recording medium to a general-purpose personal computer or the like. Before describing the recording medium, a personal computer that handles the recording medium will be briefly described.

FIG. 18 is a diagram showing an example of the internal configuration of a general-purpose personal computer. The CPU (Central Processing Unit) 211 of the personal computer is an RO
Various processes are executed according to a program stored in an M (Read Only Memory) 212. RAM (Random A
The ccess memory 213 stores data, programs, and the like necessary for the CPU 211 to execute various processes. The input / output interface 215 is connected to an input unit 216 including a keyboard and a mouse, and outputs a signal input to the input unit 216 to the CPU 211. The input / output interface 215 is also connected to the output unit 7 including a display, a speaker, and the like.

Further, the input / output interface 215 has a storage unit 218 composed of a hard disk or the like,
Further, a communication unit 219 for exchanging data with another device via a network such as the Internet is also connected. The drive 220 includes a magnetic disk 231, an optical disk 232, a magneto-optical disk 233, and a semiconductor memory 23.
4 is used when reading data from or writing data to a recording medium such as a recording medium.

As shown in FIG. 18, the recording medium is a magnetic disk 231 (including a flexible disk) on which the program is recorded, which is distributed in order to provide the user with the program, separately from a personal computer, and an optical disk 232. (CD-ROM (Compact Disc-Read Only Mem
ory), a DVD (including a Digital Versatile Disc), a magneto-optical disk 233 (including an MD (Mini-Disc) (registered trademark)), or a package medium including a semiconductor memory 234, as well as a computer. The storage unit 218 includes a hard disk including a ROM 212 storing a program and a storage unit 218, which are provided to a user in a state where the hard disk is incorporated in advance.

In the present specification, the steps of describing a program provided by a medium are not necessarily performed in chronological order but may be performed in chronological order according to the described order. Alternatively, it also includes individually executed processing.

In this specification, the system is
It represents the entire device composed of a plurality of devices.

[0103]

As described above, according to the image processing apparatus, method, and program of the present invention, the image data with a predetermined sub-pixel accuracy can be obtained by the processing using the FIR filter or the processing using the linear interpolation. Is generated, the image data is stored, and if necessary, the prediction compensation process is performed using the stored image data, so that the already generated image data is repeatedly generated. Can be prevented, the amount of calculation related to the generation can be reduced, and the speed of the prediction compensation process can be increased.

Further, according to the second image processing apparatus of the present invention, since the pixel value subjected to the operation related to the motion compensation processing is stored, the stored pixel value is used. It is possible to reduce the processing amount related to the change of the update frequency of the image and increase the speed.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration of an example of a conventional image information encoding device.

FIG. 2 is a diagram illustrating a configuration of an example of a conventional image information decoding device.

FIG. 3 is a diagram illustrating a configuration of an example of a conversion device that converts an update frequency.

FIG. 4 is a diagram illustrating a case where the number of frames is increased by inserting an interpolation frame.

FIG. 5 is a diagram illustrating a case where the number of frames is reduced by deleting an input frame and inserting an interpolation frame.

FIG. 6 is a diagram for explaining motion compensation from outside a VOP boundary.

FIG. 7 is a diagram illustrating a motion prediction compensation process with quarter-pixel accuracy.

FIG. 8 is a diagram illustrating a motion prediction compensation process with 1/8 pixel accuracy.

FIG. 9 is a diagram illustrating an interpolation method with 1/8 pixel accuracy.

FIG. 10 is a diagram illustrating a configuration of an embodiment of an image information encoding device to which the present invention has been applied.

FIG. 11 is a diagram illustrating a configuration of an embodiment of an image information decoding device to which the present invention has been applied.

FIG. 12 is a diagram for explaining a relationship between an interpolation image buffer and area division.

FIG. 13 shows a predicted image P in a reference frame, a predicted image region P ′ and a divided region S in an interpolated image buffer.
It is a figure for explaining the relation of (P).

FIG. 14 is a diagram illustrating acquisition of a predicted image from a divided area.

FIG. 15 is a diagram for describing a situation in which a predicted image area P ′ is included in a plurality of divided areas.

FIG. 16 is a diagram for describing an interpolated image buffer including a padding area.

FIG. 17 is a diagram showing a configuration of an embodiment of an apparatus for converting an update frequency to which the present invention is applied.

FIG. 18 is a diagram illustrating a medium.

[Explanation of symbols]

23 frame memory, 24 motion prediction / compensation unit,
50 frame memory, 51 motion prediction / compensation unit,
Reference Signs List 100 image information encoding device, 101 interpolation image buffer, 120 image information decoding device, 121 interpolation image buffer, 131 image information conversion device, 132 motion prediction compensation unit, 133 selector, 134 frame memory, 135 interpolation image buffer, 136 delay buffer

   ────────────────────────────────────────────────── ─── Continuation of front page    (72) Inventor Yoichi Yagasaki             6-7-35 Kita Shinagawa, Shinagawa-ku, Tokyo Soni             ー Inc. F term (reference) 5C059 KK10 KK15 LA05 LB13 MA00                       MA21 NN21 PP04 SS02 SS08                       SS20 TA08 TA62 TA65 TB07                       TC03 UA11 UA33 UA39                 5J064 AA00 BA04 BA16 BB01 BB03                       BB04 BC01 BC06 BC08 BC12                       BC16 BC25 BC29 BD04

Claims (31)

[Claims] 1. An image processing apparatus for inputting image information of interlaced scanning or sequential operation and performing motion prediction compensation by orthogonal transformation and block matching with decimal pixel precision, wherein processing using a FIR filter or linear interpolation is used. Generating means for generating image data with a preset sub-pixel accuracy by processing performed, storing means for storing the image data generated by the generating means, and the image data stored in the storing means And a prediction compensation unit for executing a prediction compensation process by using the image processing apparatus. 2. The image processing apparatus according to claim 1, wherein the generation unit generates image data with half-pixel accuracy of the input image information. 3. The image processing apparatus according to claim 1, wherein the generating unit is configured to: when 1/2 represents a rounding operation, 生成 of the input image information.
The image data with pixel accuracy is generated by using an FIR filter that performs an operation represented by an expression of {1, -5, 20, 20, -5, 1} // 32. The image processing apparatus according to any one of the preceding claims. 4. The image processing apparatus according to claim 1, wherein the generation unit generates image data having １ ／ pixel accuracy of the input image information. 5. The image processing apparatus according to claim 1, wherein the generation unit is configured to output a 1/4 of the image information.
The image processing apparatus according to claim 1, wherein the image data with pixel accuracy is generated from the image data with half-pixel accuracy using an FIR filter with a predetermined number of taps. 6. The image processing apparatus according to claim 1, wherein when // represents a rounding operation, the generating means adds ｛1, −5, 20, 20, 20, −5, 1 Ｆ // 32 using an FIR filter that performs the operation represented by the expression
The image processing apparatus according to claim 1, wherein the image data is generated with 4-pixel accuracy. 7. The image processing apparatus according to claim 1, wherein the generation unit is configured to output a 1/4 of the image information.
2. The image data of pixel accuracy is generated by linear interpolation from image data of 1/2 pixel accuracy.
An image processing apparatus according to claim 1. 8. The image processing apparatus according to claim 1, wherein the generation unit is configured to output 1/1 of the image information.
The image processing apparatus according to claim 1, wherein the image data with 4, 2/4, and 3/4 pixel accuracy is generated using an FIR filter having a predetermined number of taps. 9. The image processing apparatus according to claim 9, wherein the generation unit is configured to define a rounding operation by using １ ／, ／, and の of the image information.
And 3/4 pixel precision field images are expressed as ３ ， −3, 12, −37, 229, 71, −21, 6, −
1｝ // 256 ｛-3,12, -39,158,158, -39,1
2, -3｝ // 256 ｛-1,6, -21,71,229, -37,12,-
The image processing apparatus according to claim 1, wherein the image is generated using an FIR filter that performs an operation represented by an expression of 3｝ / 256. 10. The image processing apparatus according to claim 1, wherein the generation unit is configured to output 1/1 of the image information.
The image processing apparatus according to claim 1, wherein the image data with four-pixel accuracy is generated by linear interpolation. 11. The image processing apparatus according to claim 1, wherein the generation unit is configured to output 1/1 of the image information.
2. The image processing apparatus according to claim 1, wherein the image data with 8-pixel accuracy is generated from the image data with 1 / 4-pixel accuracy using an FIR filter with a predetermined number of taps. 12. The image processing apparatus according to claim 1, wherein when // represents a rounding operation, 生成 1, -5, 20, 20, -5, 1 Ｆ // 32 using an FIR filter that performs the operation represented by the expression
2. The image processing apparatus according to claim 1, wherein the image data is generated with 8-pixel accuracy. 13. The image processing apparatus according to claim 13, wherein the generation unit is configured to output 1/1/3 of the image information.
2. The image processing apparatus according to claim 1, wherein the image data with 8-pixel accuracy is generated by linear interpolation from the image data with 1/4 pixel accuracy. 14. The image processing apparatus according to claim 1, wherein the generation unit is configured to output 1/1 of the image information.
The image processing apparatus according to claim 1, wherein the image data with 8-pixel accuracy is generated by linear interpolation. 15. The image processing apparatus according to claim 15, wherein the generation unit is configured to output 1/1 of the image information.
2. The image processing apparatus according to claim 1, wherein the image data with 4-pixel accuracy is generated from the image data with 1 / 2-pixel accuracy using an FIR filter with a predetermined number of taps. 16. The prediction compensating means, wherein the storage means stores image data with half-pixel accuracy of the image information, and when the prediction compensation requires image data with half-pixel accuracy, 2. The image processing apparatus according to claim 1, wherein the half-pixel precision image data stored in a storage unit is read out and used as it is. 17. The prediction compensation means, wherein the storage means stores image data of 1/2 pixel precision of the image information, and when prediction compensation requires image data of 1/4 pixel precision, The image data with 1/2 pixel accuracy stored in the storage means is read out, and image data with 1/4 pixel accuracy is generated from the image data by using an FIR filter with a predetermined number of taps to perform prediction compensation. The image processing apparatus according to claim 1, wherein: 18. The image processing apparatus according to claim 18, wherein when // represents a rounding operation, the prediction compensation means adds に 1, -5, 20, 20, -5, 1 // 32 using an FIR filter that performs the operation represented by the equation
The image processing apparatus according to claim 17, wherein the image data is generated with 4-pixel accuracy. 19. The prediction compensating means, wherein the storage means stores image data of 1/2 pixel precision of the image information, and when prediction compensation requires image data of 1/4 pixel precision, The method is characterized in that the image data of 1/2 pixel accuracy stored in the storage means is read out, and image data of 1/4 pixel accuracy is generated by using linear interpolation on the image data to perform prediction compensation. The image processing device according to claim 1. 20. The prediction compensation means, wherein the storage means stores image data of 1/4 pixel accuracy of the image information, and when the prediction compensation requires image data of 1/4 pixel accuracy, 2. The image processing apparatus according to claim 1, wherein the image data of 1/4 pixel precision stored in a storage unit is read out and used as it is. 21. The prediction compensating means, wherein the storage means stores image data of 1/4 pixel precision of the image information, and when prediction compensation requires image data of 1/8 pixel precision, The image data with 1/4 pixel precision stored in the storage means is read out, and image data with 1/8 pixel precision is generated from the image data by using an FIR filter having a predetermined number of taps to perform prediction compensation. The image processing apparatus according to claim 1, wherein: 22. The prediction compensation means according to claim 1, wherein when // represents a rounding operation, ｛1, -5, 20, 20, -5, 1 // 32 using an FIR filter that performs the operation represented by the equation
22. The image processing apparatus according to claim 21, wherein image data with 8-pixel accuracy is generated. 23. The prediction compensation unit, wherein the storage unit stores image data of の pixel accuracy of the image information, and when prediction compensation requires image data of ８ pixel accuracy, The method is characterized in that the image data of 1/4 pixel accuracy stored in the storage means is read out, and image data of 1/8 pixel accuracy is generated by using linear interpolation on the image data to perform prediction compensation. The image processing device according to claim 1. 24. The prediction compensation means, wherein the storage means stores image data of の pixel accuracy of the image information, and when prediction compensation requires image data of ８ pixel accuracy, 2. The image processing apparatus according to claim 1, wherein the image data of 1/8 pixel accuracy stored in a storage unit is read out and used as it is. 25. The image processing apparatus according to claim 1, wherein the storage unit has a padding area for supporting motion compensation from outside a VOP boundary. 26. The image processing apparatus according to claim 1, wherein the prediction compensation unit performs orthogonal transform by discrete cosine transform or Karhunen-Loeve transform and overlap motion prediction compensation with decimal pixel precision. 27. An image processing method of an image processing apparatus for inputting image information of interlaced scanning or sequential operation and performing motion prediction compensation by orthogonal transformation and block matching with decimal pixel precision, wherein processing using a FIR filter or linear processing is performed. A generation step of generating image data with preset decimal pixel accuracy by a process using interpolation; a storage control step of controlling storage of the image data generated in the processing of the generation step; A prediction compensation step of executing a prediction compensation process using the image data the storage of which has been controlled in the processing of the control step. 28. A program for an image processing apparatus for inputting image information of interlaced scanning or sequential operation and performing motion prediction compensation by orthogonal transformation and block matching with decimal pixel precision, comprising a processing using a FIR filter or a linear processing. A generation step of generating image data with a preset sub-pixel accuracy by a process using interpolation; a storage control step of controlling storage of the image data generated in the processing of the generation step; A prediction compensation step of executing a prediction compensation process using the image data whose storage has been controlled in the control step process. 29. A computer using an FIR filter or a linear processor, which inputs image information of interlaced scanning or sequential operation, and controls an image processing apparatus which performs motion prediction compensation by orthogonal transformation and block matching with decimal pixel precision. A generation step of generating image data with a preset sub-pixel accuracy by a process using interpolation; a storage control step of controlling storage of the image data generated in the processing of the generation step; A prediction compensation step of performing a prediction compensation process using the image data whose storage has been controlled in the process of the control step. 30. Perform motion prediction compensation with sub-pel precision,
An image processing apparatus for converting an update frequency of an input image by inserting an interpolation image, the image processing apparatus comprising: a holding unit for holding a pixel value on which an operation related to the motion prediction compensation processing is performed. . 31. The update frequency is increased from 25 Hz to 50 Hz.
31. The image processing apparatus according to claim 30, wherein the image is converted into Hz.