RU2658891C2 - Device and method for image processing - Google Patents

Abstract

FIELD: electrical communication engineering.

SUBSTANCE: invention relates to image processing. Image processing apparatus is disclosed, comprising: a circuit configured to decode an encoded image; performing the first motion compensation by using the first motion vector of the encoded image to generate the first motion compensation image; performing a second motion compensation by using the second motion vector of the encoded image to generate a second motion compensation image; generating a predicted image by performing the filtering processing and summing processing on the first motion compensation image and the second first motion compensation image.

EFFECT: technical result is to enable the generation of a predicted high-precision image by a small amount of control data.

12 cl, 30 dwg

Description

FIELD OF THE INVENTION

The present invention relates to a device and method for processing images, and, in particular, to a device and method for processing images that can generate a predicted image with high accuracy without increasing the amount of processing.

State of the art

Typically, motion compensation coding methods such as MPEG (Cinematography Expert Group standard) or H.26x and orthogonal transforms such as discrete cosine transforms, Karunen-Loeve transforms or wavelet transforms are commonly used as coding methods for processing dynamic images. In these methods of encoding a dynamic image, the amount of code is reduced by using correlation in the spatial direction and the time direction among the characteristics of the input image signal over which encoding is to be performed.

For example, the H.264 method uses unidirectional or bidirectional prediction to generate an intermediate frame, which is a frame serving the purpose of inter-frame prediction (inter-frame prediction) using time-domain correlation. Inter-frame prediction generates a predicted image based on frames at different points in time.

FIG. 1 is a diagram illustrating an example of unidirectional prediction.

As illustrated in FIG. 1, in the case of generating the frame P ₀ to be encoded, which is the frame of the current time to be encoded, by unidirectional prediction, motion compensation is performed using the encoded frame as the reference frame in the past or future time relative to the current time. The remainder between the predicted image and the actual image is encoded by correlation in the time direction, so that the amount of code can be reduced. The reference frame data and the motion vector are used as data defining the reference frame and data defining a position to be referred to in the reference frame, respectively, and these pieces of data are transmitted from the encoding side to the decoding side.

Here, the number of reference frames is not necessarily 1. For example, in the H.264 method, a plurality of frames can be used as reference frames. When two frames close in time to the frame P ₀ to be encoded are used as reference frames R ₀ and R ₁ , as shown in FIG. 1, the pixel values of an arbitrary macroblock in a frame P ₀ to be encoded can be predicted based on the pixel values of arbitrary pixels in a reference frame R ₀ or R ₁ .

The rectangles shown inside the respective frames in FIG. 1, represent macroblocks. If we assume that the macroblock in the frame P ₀ to be encoded, which is the prediction target, is the macroblock MBP ₀ , then the macroblock in the reference frame R ₀ corresponding to the macroblock MBP ₀ is the macroblock MBR ₀ , which is determined by the motion vector MV ₀ . In addition, the macroblock in the reference frame R ₁ is the macroblock MBR ₁ which is defined by the motion vector MV ₁ .

When it is assumed that the pixel values of the macroblocks MBR ₀ and MBR ₁ (pixel values of the motion compensation images) are equal to MS ₀ (i, j) and MS ₁ (i, j), since the pixel values of any of the motion compensation images are used as the pixel values of the predicted images in unidirectional prediction, the predicted image Pred (i, j) is expressed by the following equation (1). (i, j) denotes the relative position of the pixel in the macroblock, where 0≤i≤16 and 0≤j≤16. In equation (1), ⎪⎢ means that the value taken is MS ₀ (i, j) or MS ₁ (i, j).

In addition, it is possible to divide a single macroblock with a size of 16 × 16 pixels into smaller blocks having a size of, for example, 16 × 8 pixels, and to perform motion compensation in separate blocks formed by division, by reference to different reference frames. By transmitting the motion vector with decimal precision, and not the motion vector with integer precision, and by interpolating with the FIR filter (with a finite impulse response) defined according to the standard, the pixel values for the pixels surrounding the corresponding reference position can be used to compensate movement.

FIG. 2 is a diagram illustrating an example of bidirectional prediction.

As illustrated in FIG. 2, in the case of generating the frame B ₀ to be encoded, which is the current time frame to be encoded, using bi-directional prediction, motion compensation is performed using encoded frames in the past or future time relative to the current time as reference frames. A plurality of coded frames are used as reference frames, and the difference between the predicted image and the actual image is encoded by correlation with these frames, whereby the amount of code can be reduced. In the H.264 method, in addition, it is possible to use many past frames and many future frames as reference frames.

As shown in FIG. 2, when one past frame and one future frame are used as reference frames L ₀ and L ₁ , and the frame B _{0 to} be encoded serves as the basis, the pixel values of an arbitrary macroblock in the frame B ₀ to be encoded can be predicted based on the pixel values of arbitrary pixels in reference frames L ₀ and L ₁ .

In the example of FIG. 2, the macroblock in the reference frame L ₀ corresponding to the macroblock MBB ₀ in the frame B ₀ to be encoded is the macroblock MBL ₀ defined by the motion vector MV ₀ . In addition, the macroblock in the reference frame L ₁ corresponding to the macroblock MBB ₀ in the frame B ₀ to be encoded is the macroblock MBL ₁ defined by the motion vector MV ₁ .

If we assume that the pixel values of the macroblocks MBL ₀ and MBL ₁ are equal to MS ₀ (i, j) and MS ₁ (i, j), respectively, then the value Pred (i, j) of the pixel of the predicted image Pred (i, j) can be obtained as the average of these pixel values, as expressed by the following equation (2).

With the motion compensation using unidirectional prediction described above, the accuracy of the predicted image is increased by increasing the accuracy of the motion vector and reducing the size of the macroblock to reduce the residual relative to the actual image, thereby increasing the coding efficiency.

In addition, when compensating for motion using bidirectional prediction, average pixel values for pixels of near-time reference frames are used as pixel values for pixels of the predicted image, thereby stably reducing the prediction difference in terms of probability.

In addition, as another method, a method is proposed for converting the correlation in the time direction into spatial resolution using motion compensation and FIR filtering of pixel values and their use (for example, see NPL 1).

In the method described in NPL 1, time-correlation is used to process a resolution increase that is performed with respect to the input image sequence. In particular, difference data on the difference between the current image and the past image, relative to which the motion prediction is compensated, is calculated, and the difference data is fed back to the target current image, thereby restoring the high-frequency component included in the input images.

List of links

Non-Patent Literature

NPL 1: “Improving Resolution by Image Registration”, Michal Irani and Simon Peleg, Department of Computer Science, The Hebrew University of Jerusalem, 91904 Jerusalem, Israel, Communicated by Rama Chellapa, Received June 16, 1989 ; accepted May 25, 1990.

SUMMARY OF THE INVENTION

Technical challenge

In the case of conventional unidirectional prediction, even when multiple reference frames can be selected, it is necessary to selectively use the pixel values of any of these reference frames as the pixel values of the frame to be encoded. Thus, since a reference frame that is not selected is not used for motion compensation, the temporal correlation between the reference frame and the frame to be encoded is not used enough, and much can be improved from the point of view of increasing encoding efficiency.

In addition, in the case of conventional bi-directional prediction, the average pixel values of two reference frames are used as the pixel values of the frame to be encoded, so that a temporary low-pass filtering operation is performed and the high-frequency component is lost in the predicted image. As a result, since the difference signal including the high-frequency component cannot be encoded, the image obtained by decoding does not include the high-frequency component, and the resolution is reduced.

Further, the prediction can be performed with higher accuracy than with conventional bidirectional prediction by filtering data on two or more reference frames in the manner described in NPL 2, and using them. In this case, however, it is necessary to transmit motion vector data relating to two or more reference frames to the decoder. That is, to increase the accuracy of prediction, a large amount of control data is required, which from the point of view of coding efficiency can be inefficient.

The present invention is made taking into account these circumstances and is aimed at providing the ability to generate a predicted image of high accuracy using a small amount of control data by reducing the amount of code for motion vectors that are necessary to perform bidirectional prediction or reference to multiple images.

The solution of the problem

An object of the present invention is an image processing apparatus comprising a circuit configured to: decode an encoded image; performing a first motion compensation by using the first motion vector of the encoded image to generate a first motion compensation image; performing a second motion compensation by using the second motion vector of the encoded image to generate a second motion compensation image; generating the predicted image by performing filtering processing and summing processing on the first motion compensation image and the second first motion compensation image.

According to an embodiment of the invention, the filtering processing extracts a high frequency component of the motion compensation image. The first motion compensation image is generated from a reference frame formed from a decoded image; and generating a second motion compensation image is carried out from a reference frame other than the reference frame from which the first motion compensation image is generated. The selection of the high-frequency component of the motion compensation image is accomplished by using the correlation in the time direction included in the motion compensation images.

According to another embodiment of the invention, the second motion compensation image is the same or similar to the first motion compensation image generated from the reference frame using a certain cost function common to an encoding device that encodes the image, the second motion compensation image serving as the motion compensation image, corresponding to the predicted image. The cost function is a function for calculating the total sum of the absolute values of the difference values of individual pixel values between the first image compensation image and the target block for processing the reference frame. The cost function may also be a function for calculating the minimum quadratic error of individual pixel values between the first image compensation image and the target block to be processed by the reference frame.

According to another embodiment of the invention, the predicted image generating process includes: performing low-pass filtering of the differential image between the first motion compensation image and the second motion compensation image; performing high-pass filtering of an image obtained by low-pass filtering; and summing the image obtained by low-pass filtering and the image obtained by high-pass filtering with the first motion compensation image or the second motion compensation image, whereby a predicted image is generated.

The summation process may include summing an image obtained by low-pass filtering and an image obtained by high-pass filtering with a motion compensation image generated from a frame per unit time of the previous time of the predicted image.

According to yet another embodiment of the invention, the image processing apparatus may further comprise a circuit configured to: receive an identification mark for determining whether a predicted image should be generated by unidirectional prediction performed by unidirectional prediction means whether the predicted image should be generated by bidirectional prediction performed bidirectional prediction means, or predicted image adjection should be generated through filtering processing; and evaluating, by checking with the identification mark received by the receiving means, whether the predicted image should be generated by unidirectional prediction, whether the predicted image should be generated by bidirectional prediction, or the predicted image should be generated by filtering processing.

According to another embodiment of the invention, the image processing apparatus may further comprise a circuit configured to: perform unidirectional prediction using a plurality of motion compensation images to generate a predicted image; and performing bidirectional prediction using a plurality of motion compensation images to generate a predicted image.

An object of the present invention is also an image processing method comprising the steps of: decoding an encoded image; performing a first motion compensation by using the first motion vector of the encoded image to generate a first motion compensation image; performing a second motion compensation by using the second motion vector of the encoded image to generate a second motion compensation image; and generating a predicted image by performing filtering processing and summing processing on the first motion compensation image and the second first motion compensation image.

Advantageous Effects of the Invention

According to the present invention, a high precision predicted image can be generated without increasing the amount of transmitted motion vectors in the stream, and high coding efficiency can be achieved.

Brief Description of the Drawings

FIG. 1 is a diagram illustrating an example of unidirectional prediction.

FIG. 2 is a diagram illustrating an example of bidirectional prediction.

FIG. 3 is a diagram describing a principle of generating a predicted image of the present invention.

FIG. 4 is a block diagram illustrating an example configuration of a decoding apparatus according to an embodiment of the present invention.

FIG. 5 is a diagram illustrating the idea of a third prediction mode.

FIG. 6 is a block diagram illustrating an example configuration of the motion prediction-compensation scheme of FIG. 3.

FIG. 7 is a diagram illustrating an example of reference frames.

FIG. 8 is a diagram illustrating another example of reference frames.

FIG. 9 is a block diagram illustrating an example configuration of the prediction circuit of FIG. 6.

FIG. 10 is a block diagram illustrating an example configuration of a filter circuit of FIG. 6.

FIG. 11 is a flowchart describing decoding processing performed by a decoding device.

FIG. 12 is a flowchart describing motion prediction-compensation processing performed in step S9 of FIG. eleven.

FIG. 13 is a flowchart describing an example procedure in extraction processing.

FIG. 14 is a flowchart describing an example procedure in filtering prediction processing.

FIG. 15 is a block diagram illustrating an example configuration of an encoding device.

FIG. 16 is a block diagram illustrating an example configuration of the mode determination circuit of FIG. fifteen.

FIG. 17 is a block diagram illustrating an example configuration of a motion prediction-compensation scheme of FIG. fifteen.

FIG. 18 is a flowchart describing encoding processing performed by an encoding device.

FIG. 19 is a flowchart for describing the mode determination processing performed in step S108 of FIG. 18.

FIG. 20 is a flowchart describing a motion prediction-compensation operation performed in step S111 of FIG. eighteen.

FIG. 21 is a block diagram illustrating another example configuration of a filter circuit.

FIG. 22 is a block diagram illustrating another example configuration of a filter circuit.

FIG. 23 is a block diagram illustrating an example of a case of using three reference frames.

FIG. 24 is a block diagram illustrating an example configuration of a filter circuit in the case of using three reference frames.

FIG. 25 is a block diagram illustrating an example configuration of a personal computer.

FIG. 26 is a block diagram illustrating an example of a basic configuration of a television receiver to which the present invention is applied.

FIG. 27 is a block diagram illustrating an example of a basic configuration of a mobile telephone to which the present invention is applied.

FIG. 28 is a block diagram illustrating an example of a basic configuration of a hard disk recorder to which the present invention is applied.

FIG. 29 is a block diagram illustrating an example of a basic configuration of a camera to which the present invention is applied.

FIG. 30 is a diagram illustrating an example of macroblock sizes.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention (hereinafter referred to as embodiments) will be described below. Note that the description will be in the following order:

1. The first embodiment (decoding operation)

2. Second embodiment (coding operation)

3. Third embodiment (modification of the filter circuit)

First Embodiment

Prediction principle

FIG. 3 is a diagram describing a principle in a method of generating a predicted image to which the present invention is applied.

In the present invention, at least one motion vector (motion vector A) is transmitted in a bitstream to obtain a plurality of motion compensation images from a plurality of reference planes in a decoder.

FIG. 3 illustrates a state in which two frames — frame (N-1) and frame (N-2) —are used as reference planes for motion compensation to decode frame N.

In FIG. 3, a motion vector A indicating coordinates in a frame (N-1) is transmitted in a stream. The decoder receives an MS image using this vector.

Subsequently, the decoder performs motion prediction to perform motion compensation based on the frame (N-2). That is, in the frame (N-2), the image MS 'is searched, which has a close value with respect to the image MS. Search methods, e.g. search algorithm, search range, cost function, etc. can be determined arbitrarily, since they are common to the encoder and decoder in advance. When they are common to the encoder and decoder, the search results in the encoder and decoder, that is, the pixel values of the image MS ', match each other.

In this case, the decoder can obtain a motion prediction image based on the frame (N-1) and the frame (N-2). Accordingly, the motion vector MS 'is not necessary. That is, the amount of motion vector code is reduced. Thus, the decoder and encoder can generate high-precision predicted images using a small amount of control data.

Decoder Configuration

FIG. 4 is a block diagram illustrating an example configuration of a decoding apparatus 1 according to an embodiment of the present invention.

Image data encoded by an encoding device, which will be described later, is input to decoding device 1 via a cable, network, or removable medium. The compressed image data is image data encoded in accordance with, for example, the H.264 standard.

The storage buffer 11 sequentially stores the bit streams input as compressed image data. The data stored in the storage buffer 11, if necessary, is read by the lossless decoding circuit 12 in the image elements of certain elements, such as macroblocks making up the frame. In the H.264 standard, an operation can be performed not only in elements of macroblocks of 16 × 16 pixels in size, but also in elements of blocks of 8 × 8 pixels or 4 × 4 pixels in size, obtained by further division of macroblocks.

The lossless decoding circuit 12 performs a decoding operation corresponding to an encoding method, such as a variable length decoding operation or an arithmetic decoding operation of an image read from the storage buffer 11. The lossless decoding circuit 12 outputs a quantized transform coefficient obtained by decoding processing to the circuit 13 inverse quantization.

Furthermore, the lossless decoding circuit 12 determines, based on the identification mark included in the header of the image to be decoded, whether the image is an intra-frame encoded image or an inter-frame encoded image. If the lossless encoding circuit 12 estimates the image to be decoded as an image with intraframe coding, the lossless encoding circuit 12 outputs the intra prediction mode data stored in the image header to the intra prediction circuit 22. Intra-frame prediction mode data includes intra-frame prediction data, such as the size of a block serving as a processing element.

If the lossless encoding circuit 12 estimates the image to be decoded as inter-frame encoded data, the lossless encoding circuit 12 outputs a motion vector and an identification mark stored in the image header to the motion prediction-compensation circuit 21. Using the identification tag, a prediction mode for generating a predicted image by inter-frame prediction can be determined. An identification mark is set, for example, in macroblock or frame elements.

As a prediction mode, a third prediction mode is prepared for generating a predicted image by filtering motion compensation images that are extracted from a plurality of reference frames arranged in one or two time directions, in addition to the unidirectional prediction mode of FIG. 1 and the bidirectional prediction mode of FIG. 2.

FIG. 5 is a diagram illustrating the idea of a third prediction mode.

In the example of FIG. 5, where the time of the current frame (predicted frame) is taken as the basis, a frame one unit of time preceding it is considered as a reference frame R ₀ , and a frame one unit of time preceding a reference frame R ₀ is considered as a reference frame R ₁ . In this case, according to the third prediction mode, motion compensation images MS ₀ and MC ₁ extracted from the reference frames R ₀ and R ₁ are supplied to the filtering circuit, and the pixel values of the image output from the filtering circuit are considered as pixel values of the predicted image which is the target macroblock.

Next, a prediction mode in which the pixel values of any of the motion compensation images extracted from the plurality of reference frames arranged in one direction are considered as the pixel values of the predicted image, as described above with reference to FIG. 1 is called simply a unidirectional prediction mode. In addition, a prediction mode in which average pixel values of motion compensation images extracted from a plurality of reference frames arranged in two directions are considered as pixel values of the predicted image, as described above with reference to FIG. 2 is called simply a bi-directional prediction mode.

The third prediction mode shown in FIG. 5, in which the pixel values of the predicted image are obtained by filtering individual motion compensation images extracted from a plurality of reference frames arranged in one or two directions, called a prediction filtering mode. The prediction filtration mode will be described in detail below.

In FIG. 4, the inverse quantization circuit 13 performs a method corresponding to the quantization method used by the coding side to inverse quantize the quantized transform coefficient supplied from the lossless decoding circuit 12. The inverse quantization circuit 13 outputs a transform coefficient obtained by performing inverse quantization to the inverse orthogonal transform circuit 14.

The inverse orthogonal transform circuit 14 performs a fourth-order inverse orthogonal transform for the transform coefficient supplied from the inverse quantization circuit 13 in a manner corresponding to the orthogonal transform method used by the encoding side, such as a discrete cosine transform or Karunen-Loeve transform, and outputs the resulting image to a summing Scheme 15.

The summing circuit 15 combines the decoded image supplied from the inverse orthogonal transform circuit 14 and the predicted image supplied from the motion compensation prediction circuit 21 or the intra prediction circuit 22 through the switch 23, and outputs the combined image to the deblocking filter 16.

The deblocking filter 16 eliminates the block noise included in the image filed from the summing circuit 15 and outputs an image from which the block noise is removed. The image output from the deblocking filter 16 is supplied to the reordering buffer 17 and the frame memory 19.

The reordering buffer 17 temporarily stores the image filed from the deblocking filter 16. The reordering buffer 17 generates individual frames, for example, from images in macroblock elements stored in it, reorders the generated frames in a specific order, such as display order, and outputs them to a digital-to-analog converter circuit 18.

The digital-to-analog conversion circuit 18 performs digital-to-analog conversion of the individual frames supplied from the reordering buffer 17 and outputs the signals of the individual frames to the outside.

Frame memory 19 temporarily stores an image filed from deblocking filter 16. The data stored in the frame memory 19 is supplied to the motion prediction-compensation circuit 21 or the intra prediction circuit 22 through the switch 20.

The switch 20 is connected to terminal a1 in the case of generating a predicted image using inter-frame prediction, and in the case of generating it using intra-frame prediction, is connected to terminal b1. The switching of the switch 20 is controlled, for example, by a control circuit 31.

The motion prediction-compensation circuit 21 determines the prediction mode in accordance with an identification mark supplied from the lossless decoding circuit 12, and selects a frame for use as a reference frame among the decoded frames stored in the frame memory 19 in accordance with the prediction mode. The motion compensation prediction-circuit 21 determines the macroblock corresponding to the target predicted image among the macroblocks constituting the reference frame based on the motion vector supplied from the lossless decoding circuit 12, and extracts the specific macroblock as the motion compensation image. The motion compensation prediction-circuit 21 receives the pixel values of the predicted image from the pixel values of the motion compensation image in accordance with the prediction mode, and outputs the predicted image in which the pixel values are obtained to the summing circuit 15 through the switch 23.

The intra prediction circuit 22 performs intra prediction in accordance with the intra prediction mode data supplied from the lossless decoding circuit 12 to generate a predicted image. The intra prediction circuit 22 outputs the generated predicted image to the adder circuit 15 through the switch 23.

The switch 23 is connected to terminal a2 if the predicted image is generated by motion prediction-compensation circuit 21, and to terminal b2 if the predicted image is generated by intra-frame prediction circuit 22. The switching of the switch 23 is also controlled, for example, by a control circuit 31.

The control circuit 31 switches the connection of the switches 20 and 23 and controls the entire operation of the decoding device 1. Whether the image to be processed is an image with intra-frame encoding or an image with inter-frame encoding can be determined by the control circuit 31.

FIG. 6 is a block diagram illustrating an example configuration of the motion prediction-compensation circuit 21 of FIG. 3.

As shown in FIG. 6, the motion compensation prediction-circuit 21 consists of a prediction mode determination circuit 41, a unidirectional prediction circuit 42, a bi-directional prediction circuit 43, a prediction circuit 44, and a filter circuit 45. The motion vector and an identification mark supplied from the lossless decoding circuit 12 are input to a prediction mode determination circuit 41.

The prediction mode determination circuit 41 determines the prediction mode in accordance with an identification mark supplied from the lossless decoding circuit 12. The prediction mode determination circuit 41 outputs a motion vector to the unidirectional prediction circuit 42 if it is determined that the generation of the predicted image will be performed using unidirectional prediction and to the bi-directional prediction circuit 43 if it is determined that the generation of the predicted image will be performed using the bi-directional prediction. In addition, the prediction mode determination circuit 41 outputs a motion vector to the prediction circuit 44 if it is determined that the generation of the predicted image will be performed by filtering prediction.

Thus, in order to ensure the determination of filtering prediction, a value other than a value indicating unidirectional prediction and a value indicating bi-directional prediction, which are defined in the standard H.264 standard, can be set as the value of the identification mark. Alternatively, the prediction mode may be determined in a predetermined manner instead of being determined in accordance with an identification mark in order to reduce the amount of data.

The unidirectional prediction circuit 42 considers a plurality of frames arranged in one time direction as reference frames and determines macroblocks in the reference frames corresponding to the predicted image based on motion vectors, as shown in FIG. 1. In addition, the unidirectional prediction circuit 42 reads certain macroblocks in the respective reference frames as motion compensation images from the frame memory 19 and generates a predicted image using the pixel values of any of the motion compensation images as the pixel values of the predicted image. The unidirectional prediction circuit 42 outputs the predicted image to the adder circuit 15. As the unidirectional prediction performed by the unidirectional prediction circuit 42, for example, the unidirectional prediction defined in the H.264 standard is used.

The bi-directional prediction circuit 43 considers the plurality of frames arranged in two time directions as reference frames and determines macroblocks in the reference frames corresponding to the predicted image based on motion vectors, as shown in FIG. 2. In addition, the bidirectional prediction circuit 43 reads certain macroblocks in the respective reference frames as motion compensation images from the frame memory 19 and generates a predicted image using the average pixel values of the read motion compensation images as the pixel values of the predicted image. The bi-directional prediction circuit 43 outputs the predicted image to the adder circuit 15. As the bi-directional prediction performed by the bi-directional prediction circuit 43, for example, the bi-directional prediction defined in the H.264 standard is used.

Predictive circuit 44 determines a plurality of frames arranged in one or two time directions as reference frames. The frames to be used as reference frames may be predetermined or may be indicated using data transmitted by the encoding side together with the identification mark.

FIG. 7 is a diagram illustrating an example of reference frames.

In the example of FIG. 7, where the time of the predicted frame is taken as the basis, two frames preceding it by one and two units of time are considered as reference frames, as in the example described above with reference to FIG. 5. Of the two reference frames, the frame closest to the predicted frame and one unit of time preceding the predicted frame is considered as the reference frame R ₀ , and the frame, one unit of time preceding the reference frame R ₀ , is considered as the reference frame R ₁ .

FIG. 8 is a diagram illustrating another example of reference frames.

In the example of FIG. 8, where the time of the predicted frame is taken as the basis, two frames are considered as reference frames — one frame of time preceding it and one unit of time following it. Of the two reference frames, a frame that precedes a predicted frame by one unit of time is considered as a reference frame L ₀ , and a frame that follows a predicted frame one unit of time later is considered as a reference frame L ₁ .

Thus, in filtering prediction, a plurality of frames arranged in one time direction or a plurality of frames ordered in two directions are used as reference frames.

In addition, the predictive circuit 44 based on the motion vector supplied from the predictive mode determination circuit 41 determines the macroblock corresponding to the predicted image among the decoded macroblocks in at least one of the reference frames determined by the method shown in FIG. 7 or 8.

Next, the prediction circuit 44 performs motion prediction with respect to the remaining reference frame (at least one reference frame) among the reference frames determined by the method shown in FIG. 7 or 8, using a macroblock corresponding to the predicted image determined based on the motion vector, whereby a macroblock corresponding to the predicted image is determined.

The predictive circuit 44 reads certain macroblocks in the respective reference frames as motion compensation images from the frame memory 19 and outputs the read motion compensation images to the filtering circuit 45.

That is, the predictive circuit 44 extracts a motion compensation image based on the motion vector from a portion of the reference frames and extracts the motion compensation image from the remaining reference frame based on the motion prediction using the motion prediction image.

The motion vector can be performed not in elements of macroblocks with a size of 16 × 16 pixels, but in elements of blocks obtained by further division of macroblocks. Images in macroblock elements are introduced, for example, into a filter circuit 45. In FIG. 6, two arrows extending from the predictive circuit 44 to the filter circuit 45 mean that two motion compensation images are supplied.

The filter circuit 45 receives motion compensation images supplied from the predictive circuit 44, filters them, and outputs the predicted image obtained by performing filtering to the summing circuit 15.

FIG. 9 is a block diagram illustrating an example configuration of the predictive circuit 44 of FIG. 6. In FIG. 9, in the prediction circuit 44, there is a motion compensation circuit 51 and a motion prediction circuit 52.

The motion compensation circuit 51 determines a macroblock corresponding to the predicted image in the reference frame portion using the motion vector supplied from the prediction mode determination circuit 41. The motion compensation circuit 51 reads an image of a specific macroblock from the frame memory 19 and extracts it as a motion compensation image. The motion compensation circuit 51 feeds the extracted motion compensation image MS ₀ to the filter circuit 45 and, in addition, feeds it to the motion prediction circuit 52.

The motion prediction circuitry 52 compares (performs motion prediction) of the motion compensation image MS ₀ supplied from the motion compensation circuit 51 with at least one or more of the remaining reference frames (reference frames other than the reference frame from which the MS image was extracted ₀ motion compensation).

When searching for a part in the reference frame that matches or is similar to the image of the motion compensation MC _{0, the} motion prediction circuitry 52 uses some cost function that is common to the encoder and decoder 1 in advance. For example, the cost function expressed by the following equation is common (3 )

In expression (3), Refi (posX + x, posY + y) represents the pixel value in coordinates (posX + x, posY + y) on the reference plane i. Similarly, the MS [1] (x, y) is the pixel value at the relative position (x, y) from the upper left edge of the motion compensation image block MS [1] (that is, the motion compensation image MS ₀ ). bk _width and bk _height indicate the width and height of the block, respectively.

As expressed by equation (3), by summing the absolute value of the difference with all the pixels in the block, the degree of similarity between the motion compensation image MS [1] and the block (posX, posY) in the reference plane i can be calculated.

The degree of similarity is determined by the cost function common to the encoder and decoder. That is, the determination of the degree of similarity can be changed by changing the cost function. Another example of a cost function is expressed by the following equation (4). Equation (4) is an example in which the minimum quadratic error is used as the cost function.

Many cost functions can be defined and can be switched for each block or each sequence. Adaptive change of the cost function to optimize the result of further filtering processing leads to improved coding efficiency

With such a cost function, when (posX, posY) changes, the macroblock image that minimizes Cost is most similar to the motion compensation image MC [1] (that is, the motion compensation image MC ₀ ).

The range within which (posX, posY) varies is a search range, and it is necessary that the search range be common for the encoder and decoder 1. Note that an arbitrary value can be used, for example, a set value can be used, or the value can adaptively change for each sequence, each frame, each region, etc. If the value changes, the corresponding label can be described separately in the stream, or the encoding device and decoding device 1 can make a change based on the decision according to a certain algorithm.

The motion compensation image MS [1] is obtained so that it is similar to the encoded image of the current block (the macroblock to be processed), and thus, the motion compensation image MS [1], which is obtained by motion prediction using such a cost function, similar to the encoded image of the current block.

The decoder performs motion prediction in this way, whereby a plurality of motion compensation images MC [i] (i> 1) can be obtained with a single motion vector.

The motion prediction circuitry 52 reads a macroblock image from the frame memory 122 that matches or resembles the motion compensation image MS ₀ in the reference frame, and extracts it as the motion compensation image. The motion prediction circuitry 52 supplies the extracted motion compensation image MC ₁ to the filter circuit 45.

FIG. 10 is a block diagram illustrating an example configuration of a filter circuit 45. In the filter circuit 45 configured in FIG. 10, the signal is filtered in a time interval.

As shown in FIG. 10, the filtering circuit 45 consists of a difference calculating circuit 61, a low-pass filter circuit 62, a gain tuning circuit 63, a high-pass filter circuit 64, a gain tuning circuit 65, a summing circuit 66, and a summing circuit 67. The motion compensation image MS ₀ from the prediction circuit 44 is input to the difference calculating circuit 61 and the summing circuit 67, and the motion compensation image MC ₁ is input to the difference calculating circuit 61.

In the case of generating a predicted image using unidirectional prediction, as shown in FIG. 7, an image extracted from the reference frame R ₀ , which is believed to have a higher correlation with the predicted image, is considered, for example, as a motion compensation image MS ₀ , and an image extracted from the reference frame R ₁ is considered, for example, as the image of the motion compensation MC ₁ . The image extracted from the reference frame R ₀ can be considered as a motion compensation image MC ₁ , and the image extracted from the reference frame R ₁ can be considered as a motion compensation image MS ₀ .

On the other hand, in the case of generating a predicted image using bidirectional prediction, as shown in FIG. 8, an image extracted from a reference frame L ₀ that precedes one unit of time is considered as an image of the motion compensation MC ₀ , and an image extracted from a reference frame L ₁ that follows one unit of time is considered, for example, as image MS ₁ motion compensation. The image extracted from the reference frame L ₀ can be considered as the image of the motion compensation MS ₁ , and the image extracted from the reference frame L ₁ can be considered as the image of the motion compensation MS ₀ .

The difference calculation circuit 61 calculates the difference between the motion compensation image MS ₀ and the motion compensation image MS ₁ and outputs the difference image to the low-pass filter circuit 62. The difference image D is expressed by the following equation (5).

In equation (5) (i, j) denotes the relative position of a pixel in a motion compensation image. When the operation is to be performed in elements of macroblocks with a size of 16 × 16 pixels, the conditions 0≤i≤16 and 0≤j≤16 are satisfied. The same is true for the following.

In the lowpass filter circuit 62, there is an FIR filter circuit. The low-pass filter circuit 62 performs low-pass filtering of the differential image D supplied from the difference calculating circuit 61, and outputs the obtained image to the gain tuning circuit 63 and the high-pass filter circuit 64. The difference image D ′, which is an image obtained by performing low-pass filtering, is expressed by the following equation (6). In equation (6), LPF (X) indicates that low-pass filtering is performed relative to the input image X using a two-dimensional FIR filter.

The gain tuning circuit 63 adjusts the gain of the differential image D ′ supplied from the low-pass filter circuit 62 and outputs the image in which the gain is tuned to the summing circuit 66. The image X (i, j) derived from the gain tuning circuit 63 is expressed by the following equation ( 7).

In the high pass filter circuit 64, there is an FIR filter circuit. The high-pass filter circuit 64 performs high-pass filtering of the differential image D ′ supplied from the low-pass filter circuit 62 and outputs the obtained image to the gain tuning circuit 65. The difference image D ″, which is the image obtained by performing high-pass filtering, is expressed by the following equation (8). In equation (8), HPF (X) indicates that high-pass filtering is performed relative to the input image X using a two-dimensional FIR filter.

The gain tuning circuit 65 adjusts the gain of the differential image D ″ supplied from the high-pass filter circuit 64 and outputs the image in which the gain is tuned to the summing circuit 66. The output image Y (i, j) of the gain tuning circuit 65 is expressed by the following equation ( 9).

For the values of α in expression (7) and β in expression (9), for example, the values α = 0.8 and β = 0.2, but other values can be used to increase the accuracy of the predicted image. In addition, the values can adaptively change in accordance with the properties of the entered sequence.

The summing circuit 66 summarizes the image X (i, j) and the image Y (i, j), in which the gain is adjusted, and outputs the image obtained by summing. The output image Z (i, j) of the summing circuit 66 is expressed by the following equation (10).

The output image Z (i, j) is a high-frequency image component that can be obtained from the difference between the motion compensation image MS ₀ and the motion compensation image MS ₁ , i.e., from the correlation between them.

Summing circuit 67 summarizes the output image Z (i, j) supplied from summing circuit 66 with motion compensation image MS ₀ and outputs the resulting image as a predicted image to summing circuit 15. The predicted image S (i, j), which is final the output of the summing circuit 67 is expressed by the following equation (11).

Thus, according to the filtration prediction mode, an image is generated as the predicted image, which is obtained by summing the image representing the high-frequency component and the motion compensation image MS ₀ . This predicted image includes a larger volume of the high-frequency component than the predicted image, which is obtained in the case of simple bi-directional prediction. As described above, since average pixel values of the plurality of motion compensation images are obtained as pixel values in the predicted image generated by performing bidirectional prediction, the high-frequency component is lost.

In addition, since in the summing circuit 15, a predicted image including a large volume of the high-frequency component is added to the decoded image, the image that is finally output from the decoding apparatus 1 is a high-resolution image including a large volume of the high-frequency component.

Further, the predicted image can be generated by making more efficient use of the temporal correlation of the images compared to the case of simple unidirectional prediction. A predicted image generated by unidirectional prediction is not considered as an image generated by the sufficient use of temporal correlation of images, because the pixel values of any of a plurality of motion compensation images are used, as described above.

Thus, the decoding device 1 is able to increase the encoding efficiency, while reducing the increase in data volume.

Description of decoding processing algorithm

Now, processing performed by the decoding apparatus 1 having the above configuration will be described.

First, with reference to the flowchart of FIG. 11, a decoding operation performed by the decoding apparatus 1 will be described.

The processing of FIG. 11 begins when an image of some size, such as a 16 × 16 pixel macroblock, is read out by the lossless decoding circuit 12, for example, from data stored in the storage buffer 11. The processing in successive steps of FIG. 11, if necessary, is performed in parallel with processing at another stage or in a modified order. The same is true for processing in sequential steps on separate flowcharts described below.

In step S1, the lossless decoding circuit 12 performs the decoding operation of an image read from the storage buffer 11 and outputs the quantized transform coefficient to the inverse quantization circuit 13. In addition, the lossless decoding circuit 12 outputs the intra prediction mode data to the intra prediction circuit 22 if the image to be decoded is an intra-coded image and outputs a motion vector and an identification mark to the motion prediction-compensation circuit 21 if the image to be decoding, is an image with inter-frame encoding.

In step S2, the inverse quantization circuit 13 performs inverse quantization in a manner corresponding to the quantization method used by the encoding side, and outputs the transform coefficient to the inverse orthogonal transform circuit 14.

In step S3, the inverse orthogonal transform circuit 14 performs the inverse orthogonal transform of the transform coefficient supplied from the inverse quantization circuit 13 and outputs the obtained image to the summing circuit 15.

In step S4, the summing circuit 15 combines the decoded image supplied from the inverse orthogonal transform circuit 14 and the predicted image supplied from the motion prediction-compensation circuit 21 or from the intra prediction circuit 22 and outputs the combined image to the deblocking filter 16.

In step S5, the deblocking filter 16 filters to eliminate block noise included in the composite image and outputs an image from which block noise is removed.

In step S6, the frame memory 19 temporarily stores the image supplied from the deblocking filter 16.

In step S7, the control circuit 31 evaluates whether the target image is an intra-frame encoded image.

If it is determined in step S7 that the target image is an intra-frame encoded image, the intra-frame prediction circuit 22 in step S8 performs intra-frame prediction to generate the predicted image, and outputs the generated predicted image to the adder circuit 15.

On the other hand, if it is determined in step S7 that the target image is not an intra-frame encoded image, that is, it is an inter-frame encoded image, in step S9, the motion prediction-compensation circuit 21 performs a motion-prediction-compensation operation. The predicted image generated by performing the motion prediction-compensation processing is output to the adder circuit 15. The motion-prediction-compensation operation will be described below with reference to the flowchart of FIG. 12.

In step S10, the control circuit 31 evaluates whether the above operation has been performed with respect to the macroblocks in the entire frame. If the control circuit 31 determines that the operation has not been performed with respect to the macroblocks in the entire frame, the operation is repeated for another macroblock from step S1.

On the other hand, if it is determined in step S10 that the operation was performed with respect to the macroblocks in the entire frame, in step S11, the reordering buffer 17 outputs the generated frame to the digital-analog converter circuit 18 in accordance with the control action by the control circuit 31.

In step S12, the digital-to-analog conversion circuit 18 performs digital-to-analog conversion of the frame supplied from the reordering buffer 17 and outputs the analog signal to the outside. The operation described above is performed with respect to individual frames.

Next, the motion prediction-compensation operation that is performed in step S9 of FIG. 11 will be described with reference to the block diagram of FIG. 12.

In step S31, the prediction mode determination circuit 41 in the motion prediction-compensation circuit 21 evaluates whether the identification tag supplied from the lossless decoding circuit 12 means that the operation should be performed in the filter prediction mode.

If it is determined in step S31 that the identification mark means that the operation should be performed in the filter prediction mode, the processing proceeds to step S32. In step S32, the predictive circuit 44 performs an operation of extracting motion compensation images. Details of the extraction processing will be described below.

After the motion compensation images are extracted, in step S33, the filtering circuit 45 performs a filtering prediction operation.

After the processing is completed in step S33, the motion prediction-compensation operation is completed, and the processing returns to step S9 of FIG. 11 and proceeds to step S10.

In addition, if it is determined in step S31 that the identification tag does not mean that the operation should be performed in the filter prediction mode, in step S32, unidirectional or bidirectional prediction is performed and a predicted image is generated.

That is, if the identification mark means that the operation should be performed in the unidirectional prediction mode, the motion vector is supplied from the prediction mode determination circuit 41 to the unidirectional prediction circuit 42, and in the unidirectional prediction circuit 42, unidirectional prediction is performed. In addition, if the identification mark means that the operation is to be performed in the bidirectional prediction mode, the motion vector is supplied from the prediction mode determination circuit 41 to the bidirectional prediction circuit 43, and in the bidirectional prediction circuit 43, bidirectional prediction is performed. After the predicted image is output to the summing circuit 15, the motion prediction-compensation operation is completed, the processing returns to step S9 of FIG. 11 and proceeds to step S10.

Next, an example of the extraction processing procedure performed in step S32 of FIG. 12 will be described with reference to the flowchart of FIG. 13.

When the extraction operation is started, the predictive circuit 44 in step S51 sets the value of the variable i to zero. In step S52, the motion compensation circuit 51 performs motion compensation of the ith reference frame, that is, the reference plane 0, thereby extracting the motion compensation image MS ₀ . In step S53, the motion compensation circuit 51 outputs an image of the motion compensation MC ₀ to the filter circuit 45.

In step S54, the predictor circuit 44 evaluates whether the value of the variable i satisfies the condition “less than or equal to N”. If it is determined that the value of the variable i is equal to some natural number N or less, the processing proceeds to step S55.

In step S55, the motion prediction circuitry 52 increments the variable i. In step S56, the motion prediction circuit 52 performs motion prediction, such as matching, to the reference plane i using the motion compensation image MC ₀ , whereby a motion compensation image MC _{i is} generated. In step S57, the motion prediction circuit 52 outputs a motion compensation image MC _i . After the processing in step S57 is completed, the processing returns to step S54, and a further operation is performed.

If it is determined in step S54 that the value of the variable i is greater than some natural number N, the extraction operation is completed, and the processing returns to step S32 of FIG. 12 and proceeds to step S33.

The following is an example of a filtering processing procedure that is performed in step S33 of FIG. 12 will be described with reference to the block diagram of FIG. fourteen.

When filtering processing is started after extraction of the motion compensation image, the difference calculating circuit 61 of the filtering circuit 45 calculates a difference between the motion compensation image MS ₀ and the motion compensation image MC _1, and outputs a differential image to the low pass filter circuit 62 in step S71.

In step S72, the low-pass filter circuit 62 performs low-pass filtering of the differential image supplied from the difference calculating circuit 61 and outputs the image obtained as a result to the gain tuning circuit 63 and the high-pass filter circuit 64.

In step S73, the gain adjusting circuit 63 adjusts the gain of the image supplied from the low-pass filter circuit 62 and outputs the image in which the gain is tuned to the summing circuit 66.

In step S74, the high-pass filter circuit 64 performs high-pass filtering of the differential image supplied from the low-pass filter circuit 62 and outputs the image obtained as a result to the gain tuning circuit 65.

In step S75, the gain tuning circuit 65 adjusts the gain of the differential image supplied from the high-pass filter circuit 64 and outputs the image in which the gain is tuned to the summing circuit 66.

In step S76, the summing circuit 66 summarizes the image supplied from the gain tuning circuit 63 (low-pass filter output) and the image fed from the gain tuning circuit 65 (high-pass filter output), thereby obtaining a high-frequency image component. The resulting high-frequency component is supplied from the summing circuit 66 to the summing circuit 67.

In step S77, the summing circuit 67 summarizes the image supplied from the summing circuit 66 (high-frequency component) with the motion compensation image MS ₀ and outputs the resulting image serving as the predicted image to the summing circuit 15. In step S78, the filtering circuit 45 estimates, whether all motion compensation images have been processed. If it is determined that there is an unprocessed motion compensation image, the processing returns to step S71, and further processing is repeated.

On the other hand, if it is determined in step S78 that all the motion compensation images are processed, the filtering prediction operation is completed, the processing returns to step S33 of FIG. 12, the motion prediction-compensation operation is completed, the processing returns to step S9 of FIG. 11 and proceeds to step S10.

In this case, decoding is performed using the predicted image, which is generated by the filtering prediction, so that a high-resolution decoded image can be obtained. Moreover, at this moment, part of the motion compensation images is obtained using the motion vector, and the remaining motion compensation image is obtained by motion prediction (matching or the like) of the motion compensation image obtained using the motion vector. Accordingly, the number of motion vectors to be encoded can be reduced. That is, the decoding device 1 is capable of generating a predicted image of high accuracy using a small amount of control data.

Second Embodiment

Encoder Configuration

Next will be described the configuration and operation of the device on the encoding side.

FIG. 15 is a block diagram illustrating a configuration example of an encoder 101. Compressed image data that is obtained by encoding performed by the encoder 101 is input to the decoder 1 of FIG. four.

An analog-to-digital conversion circuit 111 performs an analog-to-digital conversion of the input signal and outputs the image to the reordering buffer 112.

The reordering buffer 112 reorders the frames according to the structure of the picture group for compressed image data and outputs the images in specific blocks, such as macroblocks. The image output from the reordering buffer 112 is supplied to a summing circuit 113, a mode determination circuit 123, a motion compensation prediction circuit 125, and an intra prediction circuit 126.

The summing circuit 113 receives the difference between the image supplied from the reordering buffer 112 and the predicted image that is generated by the motion compensation / prediction circuit 125 or the intra prediction circuit 126 and which is supplied through the switch 127 and outputs the remainder to the orthogonal transform circuit 114. Since the predicted image is more similar to the original image, and since the remainder obtained here is less, the amount of code assigned to the remainder is less. And thus, the coding efficiency is higher.

The orthogonal transform circuit 114 performs an orthogonal transform, such as a discrete cosine transform or the Karunen-Loeve transform of the remainder supplied from the summing circuit 113, and outputs the transform coefficient obtained by performing the orthogonal transform to the quantizing circuit 115.

The quantizing circuit 115 quantizes the transform coefficient supplied from the orthogonal transform circuit 114 in accordance with the control action of the speed control circuit 118 and outputs the quantized transform coefficient. The transform coefficient quantized by the quantizing circuit 115 is supplied to the lossless coding circuit 116 and the inverse quantization circuit 119.

The lossless encoding circuit 116 compresses the transform coefficient supplied from the quantizing circuit 115 by performing lossless encoding, such as variable-length encoding or arithmetic encoding, and outputs the data to the storage buffer 117.

In addition, the lossless encoding circuit 116 sets the value of the identification mark in accordance with the data supplied from the mode determination circuit 123, and describes the identification mark in the image header. Based on the identification tag described by the lossless encoding circuit 116, a prediction mode is determined in the decoding apparatus 1, as described above.

The lossless encoding circuit 116 also describes the data supplied from the motion compensation prediction circuit 125 or the intra prediction circuit 126 in the image header. Motion vectors and so on, detected when performing inter-frame prediction, are supplied from the motion prediction-compensation circuit 125, and data about the applied intra-frame prediction mode is supplied from the intra-frame prediction circuit 126.

A storage buffer 117 temporarily stores data supplied from the lossless encoding circuit 116 and outputs them as compressed image data at a specific point. The storage buffer 117 outputs data on the amount of generated code to the speed control circuit 118.

The speed control circuit 118 calculates a quantization level based on the amount of code output from the storage buffer 117, and controls the quantization circuit 115 so that quantization is performed with the calculated quantization level.

The inverse quantization circuit 119 performs inverse quantization of the transform coefficient quantized by the quantizing circuit 115, and outputs the transform coefficient to the inverse orthogonal transform circuit 120.

The inverse orthogonal transform circuit 120 performs the inverse orthogonal transform of the transform coefficient supplied from the inverse quantization circuit 119 and outputs the resulting image to the deblocking filter 121.

The deblocking filter 121 removes the block noise that appears in the locally decoded image and outputs the image from which the block noise is removed to the frame memory 122.

Frame memory 122 stores an image filed from deblocking filter 121. The image stored in the frame memory 122 is, if necessary, read by the mode determination circuit 123.

The mode determination circuit 123 determines whether intra-frame coding or inter-frame coding should be performed based on the image stored in the frame memory 122 and the original image supplied from the reordering buffer 112. In addition, if the mode determination circuit 123 determines that inter-coding is to be performed , the mode determination circuit 123 assigns any of three modes: a unidirectional prediction mode, a bi-directional prediction mode, and a filtering prediction mode. The mode determination circuit 123 outputs data indicative of the determination result as mode data to the lossless encoding circuit 116.

If the mode determination circuit 123 determines that inter-frame coding is to be performed, the mode determination circuit 123 outputs a frame stored in the frame memory 122 and obtained by local decoding to the motion prediction-compensation circuit 125 through the switch 124.

In addition, if the mode determination circuit 123 determines that intra-frame coding is to be performed, the mode determination circuit 123 outputs a frame stored in the frame memory 122 and obtained by local decoding to the intra-frame prediction circuit 126.

When interframe coding is performed, the switch 124 is connected to terminal a11, and when intra-frame coding is performed with terminal b11. The switch 124 is controlled, for example, by a control circuit 131.

The motion prediction-compensation circuit 125 detects motion vectors based on the original image supplied from the reordering buffer 112 and the reference frames read from the frame memory 122, and outputs the detected motion vectors to the lossless encoding circuit 116. In addition, the motion prediction-compensation circuit 125 performs motion compensation using the detected motion vectors and reference frames to generate the predicted image, and outputs the generated predicted image to the adder circuit 113 via the switch 127.

The intra-frame prediction circuit 126 performs intra-frame prediction based on the original image supplied from the reordering buffer 112 and the reference frames that are locally decoded and stored in the frame memory 122 to generate the predicted image. The intra prediction circuit 126 outputs the generated predicted image to the adder circuit 113 through the switch 127 and outputs the intra prediction mode data to the lossless encoding circuit 116.

A switch 127 is connected to terminal a12 or terminal b12 and outputs the predicted image generated by motion prediction-compensation circuitry 125 or intra-frame prediction circuitry 126 to summing circuit 113.

The control circuit 131 switches the connection of the switches 124 and 127 in accordance with the mode defined by the mode determination circuit 123, and controls the entire operation of the encoder 101.

FIG. 16 is a block diagram illustrating an example configuration of the mode determination circuit 123 of FIG. fifteen.

As shown in FIG. 16, the mode determination circuit 123 consists of an intra-frame prediction circuit 141, an inter-frame prediction circuit 142, a prediction error calculating circuit 143, and a determining circuit 144. In the mode determination circuit 123, intra-frame prediction and inter-frame prediction are performed with respect to blocks differing in size from each other , and based on the result, the prediction mode used for prediction is determined. As for inter-frame prediction, the processing is performed in individual prediction modes: unidirectional prediction mode, bidirectional prediction mode or filter prediction mode. The original image filed from the reordering buffer 112 is input to the intra prediction circuit 141, the inter prediction circuit 142, and the prediction error calculation circuit 143.

Intra-frame prediction circuitry 141 performs intra-frame prediction on block elements of different sizes from each other based on the original image and the image read from the frame memory 122, and outputs the generated predicted image to the prediction error calculation circuit 143. In the 4x4 prediction scheme 151-1, intra-frame prediction is performed on 4x4 pixel block elements. In the 8 × 8 prediction scheme 151-2, intra-frame prediction is performed on block elements of 8 × 8 pixels. In the 16 × 16 prediction scheme 151-3, intra-frame prediction is performed on block elements of 16 × 16 pixels.

The prediction circuit 161 in the inter-frame prediction circuit 142 detects motion vectors in block elements differing in size from the original image and reference frames read from the frame memory 122. In addition, the prediction circuit 161 performs motion compensation based on the detected motion vectors and outputs motion compensation images used to generate the predicted image.

In 16 × 16 prediction scheme 161-1, processing is performed with respect to images in block elements of 16 × 16 pixels. In 16 × 8 prediction scheme 161-2, processing is performed with respect to images in block elements of 16 × 8 pixels. In addition, in the 4 × 4 prediction scheme 161- (n-1), processing is performed with respect to images in block elements of 4 × 4 pixels. In the hopping / forward prediction circuit 161-n, the motion vectors are detected in the hopping mode or the direct prediction mode, and motion compensation is performed using the detected motion vectors.

Motion compensation images extracted from a plurality of reference frames arranged in one direction with respect to the current frame are supplied from the respective schemes in the prediction circuit 161 to the unidirectional prediction circuit 162. In addition, motion compensation images extracted from a plurality of reference frames arranged in two directions relative to the current frame are supplied from the respective circuits in the prediction circuit 161 to the bidirectional prediction circuit 163.

In the event that the filter prediction is performed using motion compensation images extracted from a plurality of reference frames arranged in one direction, as described above, motion compensation images extracted from a plurality of reference frames arranged in one direction are supplied from the respective circuits in scheme 161 predictions to the filtering prediction circuit 164. In the event that the filter prediction is performed using motion compensation images extracted from a plurality of reference frames arranged in two directions, motion compensation images extracted from a plurality of reference frames arranged in two directions are supplied from the corresponding circuits in the prediction circuit 161 to the circuit 164 filtration prediction.

The unidirectional prediction circuit 162 performs unidirectional prediction using motion compensation images of different sizes supplied from the respective circuitry of the prediction circuit 161, whereby a predicted image is generated and outputs the generated predicted image to the prediction error calculation circuit 143. For example, the unidirectional prediction circuit 162 generates a predicted image by considering the pixel values of any of a plurality of 16 × 16 pixel motion compensation images supplied from the prediction circuit 161-1 as the pixel values of the predicted image.

The bidirectional prediction circuit 163 performs bidirectional prediction using motion compensation images of different sizes supplied from the respective circuitry of the prediction circuit 161, whereby a predicted image is generated and outputs the generated predicted image to the prediction error calculation circuit 143. For example, the bidirectional prediction circuit 163 generates a predicted image by considering average pixel values of a plurality of 16 × 16 pixel motion compensation images supplied from the prediction circuit 161-1 as the pixel values of the predicted image.

The filter prediction circuit 164 performs filter prediction using motion compensation images of different sizes supplied from the respective circuit of the prediction circuit 161, whereby a predicted image is generated, and outputs the generated predicted image to the prediction error calculation circuit 143. The filter prediction circuit 164 corresponds to the filter circuit 45 of the decoding apparatus 1 and has the same configuration as that shown in FIG. 10.

For example, in the case of generating a predicted image using 16 × 16 pixel motion compensation images MS ₀ and MC ₁ supplied from the prediction circuit 161-1, the filtering circuit 164 obtains the difference between the motion compensation images MS ₀ and MC ₁ and performs low-pass filtering in the resulting difference image. In addition, the filter circuit 164 performs high-pass filtering in the output of the low-pass filtering and sums the image output after it in which the gain is adjusted, and the image output after the low-pass filter in which the gain is set. The filter circuit 164 summarizes the image as a summing result, which is a high-frequency component, with a motion compensation image MS ₀ , whereby a predicted image is generated, and outputs the generated predicted image to the prediction error calculation circuit 143.

The prediction error calculation circuit 143 receives the differences between the original image and the corresponding predicted images supplied from the corresponding circuits in the intra prediction circuit 141, and outputs the residual signal representing the differences obtained to the determining circuit 144. In addition, the prediction error calculation circuit 143 receives the differences between the original image and the corresponding predicted images filed from the unidirectional prediction circuit 162, the bidirectional circuit 163 prediction and filter circuit 164 in the inter prediction circuit 142, and outputs the residual signal representing the differences obtained to the determining circuit 144. In addition, the prediction error calculation circuit 143 receives the differences between the original image and the corresponding predicted images filed from the unidirectional circuit 162 predictions, bidirectional prediction circuits 163, and filter circuit 164 of inter prediction circuit 142, and outputs a residual signal representing s difference in the determining circuit 144.

The determination circuit 144 measures the intensity of the residual signals supplied from the prediction error calculation circuit 143, and determines the prediction method used to generate the predicted image having a small difference with the original image as the prediction method for generating the predicted image for use in encoding. The determining circuit 144 outputs data representing the result of the determination, which are mode data, to the lossless encoding circuit 116. The mode data includes data representing a block size for use as a processing unit.

In addition, if the determining circuit 144 determines that the predicted image will be generated using inter-frame prediction (determines that inter-frame coding will be performed), the determining circuit 144 outputs reference frames read from the frame memory 122 to the motion prediction-compensation circuit 125 together with mode data. If the determining circuit 144 determines that the predicted image will be generated using intra-frame prediction (determines that intra-frame coding will be performed), the determining circuit 144 outputs an image read from the frame memory 122 and to be used for intra-frame prediction to the intra-frame prediction circuit 126 together with mode data.

FIG. 17 is a block diagram illustrating an example configuration of the motion prediction-compensation circuit 125 of FIG. fifteen.

As shown in FIG. 17, the motion compensation prediction-circuit 125 consists of a motion vector detection circuit 181, a unidirectional prediction circuit 182, a bi-directional prediction circuit 183, a prediction circuit 184 and a filter circuit 185. The motion-compensation prediction circuit 125 has a configuration similar to that of the prediction circuit 21 motion compensation shown in FIG. 8, except that instead of the prediction mode determination circuit 41, there is a motion vector detection circuit 181.

The motion vector detection circuit 181 detects motion vectors by matching blocks or the like based on the original image supplied from the reordering buffer 112 and the reference frames supplied from the mode determination circuit 123. The motion vector detection circuit 181 is checked with the mode data supplied from the mode determination circuit 123 and outputs the motion vectors together with the reference frames to any of three circuits: the unidirectional prediction circuit 182, the bi-directional prediction circuit 183, or the predictive circuit 184.

The motion vector detection circuit 181 outputs the motion vectors together with the reference frames to a unidirectional prediction circuit 182 if a unidirectional prediction is selected, and outputs these pieces of data to a bi-directional prediction circuit 183 if a bi-directional prediction is selected. The motion vector detection circuit 181 outputs the motion vectors together with the reference frames to the predictive circuit 184 if filtering prediction is selected.

Like the unidirectional prediction circuit 42 of FIG. 8, the unidirectional prediction circuit 182 generates a predicted image by performing unidirectional prediction. The unidirectional prediction circuit 182 outputs the generated predicted image to the adder circuit 113.

Like the bidirectional prediction circuit 43 of FIG. 8, a bidirectional prediction circuit 183 generates a predicted image by performing bidirectional prediction. The bidirectional prediction circuit 183 outputs the generated predicted image to the adder circuit 113.

Like the predictive circuit 44 of FIG. 8, the predictive circuit 184 extracts motion compensation images from a plurality of (eg, two) reference frames and outputs the extracted plurality of motion compensation images to a filtering circuit 185.

Like the filter circuit 45 of FIG. 8, the filtering circuit 185 generates a predicted image by performing filtering prediction. The filtering circuit 185 outputs the generated predicted image to the summing circuit 113. Note that the filtering circuit 185 has a configuration similar to that of the filtering circuit 45 shown in FIG. 12. Further, when describing as the configuration of the filter circuit 185, the configuration of the filter circuit 45 shown in FIG. 12.

The predicted image generated by the filtering prediction includes a large volume of the high-frequency component compared to the predicted image generated by the unidirectional or bidirectional prediction, and is an image having a small difference with the original image. Thus, the amount of code assigned to the remainder is small, and thus it is possible to increase the coding efficiency.

In addition, filtering prediction can be performed if the number of reference frames is at least two, and thus, such an increase in coding efficiency can be achieved without complicating the processing. For example, the remainder of the original image can be reduced, and the coding efficiency is increased by generating and using a predicted high-precision image using a large number of reference frames in inter-frame prediction. In this case, however, the processing is complicated because the number of reference frames is large.

Note that when it is necessary to select a prediction method, a weight can be added to the residual signal intensity according to the code size, taking into account the amount of data code, such as the motion vectors required for the prediction and the encoding mode, so that the optimal prediction method is selected. Accordingly, coding efficiency can then be improved. In addition, to simplify the encoding processing, the prediction method can be adaptively selected using the number of properties in the temporal and spatial directions of the input source image.

Description of the encoding processing procedure

Next, processing performed by the encoder 101 with the above configuration will be described.

The encoding operation performed by the encoder 101 will be described with reference to the flowchart of FIG. 18. This operation begins when the image is output from the reordering buffer 112 by some unit, such as a macroblock.

In step S101, the adder circuit 113 obtains the difference between the image supplied from the reordering buffer 112 and the predicted image generated by the motion compensation prediction circuit 125 or the intra prediction circuit 126 and outputs the remainder to the orthogonal transform circuit 114.

In step S102, the orthogonal transform circuit 114 performs orthogonal transform of the remainder supplied from the summing circuit 113 and outputs the transform coefficient to the quantizing circuit 115.

In step S103, the quantizing circuit 115 quantizes the transform coefficient supplied from the orthogonal transform circuit 114 and outputs the quantized transform coefficient.

In step S104, the inverse quantization circuit 119 performs inverse quantization of the transform coefficient quantized by the quantizing circuit 115, and outputs the transform coefficient to the inverse orthogonal transform circuit 120.

In step S105, the inverse orthogonal transform circuit 120 performs the inverse orthogonal transform of the transform coefficient supplied from the inverse quantization circuit 119 and outputs the obtained image to the deblocking filter 121.

In step S106, the deblocking filter 121 performs filtering to remove block noise and outputs an image from which block noise is removed to the frame memory 122.

In step S107, the frame memory 122 stores the image supplied from the deblocking filter 121.

In step S108, the mode determination operation 123 performs the mode determination operation. The prediction mode for use in generating the predicted image is determined by the mode determination processing. The processing of the mode definition will be described below.

In step S109, the control circuit 131 judges whether intra-frame prediction should be performed based on a determination made by the mode determination circuit 123.

If it is determined in step S109 that intra-frame prediction should be performed, in step S110, the intra-frame prediction circuit 126 performs intra-frame prediction and outputs the predicted image to the adder circuit 113.

On the other hand, if it is determined in step S109 that intra-frame prediction should not be performed, that is, inter-frame prediction should be performed, in step S111, the motion prediction-compensation circuit 125 performs the motion-prediction-compensation operation, and the predicted image is output to the summing circuit 113 The processing of prediction-motion compensation will be described below.

In step S212, the lossless encoding circuit 116 compresses the transform coefficient supplied from the quantizing circuit 115 and outputs it to the storage buffer 117. In addition, the lossless encoding circuit 116 describes an identification mark in the image header in accordance with the data supplied from the determination circuit 123 mode, and describes the motion vector filed from the motion prediction-compensation circuit 125 in the image header.

In step S113, the storage buffer 117 temporarily stores data supplied from the lossless encoding circuit 116.

In step S114, the control circuit 131 evaluates whether the processing described above is performed with respect to the macroblocks in the entire frame. If it is determined that processing has not been performed with respect to the macroblocks in the whole frame, the processing is repeated for another macroblock, starting from step S111.

On the other hand, if it is determined in step S114 that the processing was performed with respect to the macroblocks in the whole frame, the storage buffer 117 outputs compressed image data in accordance with the control action of the control circuit 131 in step S115. The processing described above is performed with respect to individual frames.

Next, the mode determination operation that is performed in step S108 of FIG. 18 will be described with reference to the block diagram of FIG. 19.

In step S131, the intra prediction circuit 141 and the inter prediction circuit 142 perform intra prediction and inter prediction, respectively, with respect to blocks differing in size, thereby generating predicted images. The generated predicted images are supplied to the prediction error calculation circuit 143.

In step S132, the prediction error calculating circuit 143 obtains the differences between the original image and the corresponding predicted images supplied from the respective circuits in the intra prediction circuit 141 and from the unidirectional prediction circuit 162, the bidirectional prediction circuit 163 and the filtering circuit 164 in the inter prediction circuit 142. The prediction error calculation circuit 143 outputs a residual signal to the determining circuit 144.

In step S133, the determining circuit 144 determines the prediction method for generating the predicted image to be supplied to the summing circuit 113 based on the intensity of the residual signal supplied from the prediction error calculating circuit 143.

In step S134, the determining circuit 144 outputs the mode data, that is, data about the determined prediction method, to the lossless encoding circuit 116. After that, the processing returns to step S108 of FIG. 18, and further processing is performed.

Next, the motion prediction-compensation operation that is performed in step S111 of FIG. 18 will be described with reference to the block diagram of FIG. twenty.

In step S151, the motion vector detection circuit 181 detects motion vectors based on the original image and reference frames.

In step S152, the motion vector detection circuit 181 evaluates whether the mode determination circuit 123 has determined that the operation should be performed in the filter prediction mode.

If it is determined that the processing in the filtration prediction mode has been determined, the processing proceeds to step S153. The corresponding processing in steps S153 and S154 are performed in a manner similar to steps S32 and S33 of FIG. 12. That is, in step S153, the extraction operation is performed in the manner described above with reference to the flowchart of FIG. 13. In step S154, the filter prediction operation is performed in the manner described above with reference to the flowchart of FIG. fourteen.

After the processing is completed in step S154, the motion prediction-compensation operation is completed, the processing returns to step S111 of FIG. 18 and proceeds to step S112.

Furthermore, if in step S152 of FIG. 20, it is established that the processing in the filtering prediction mode is not defined, the processing proceeds to step S155. In step S155, the unidirectional prediction circuit 182 or the bidirectional prediction circuit 183 performs unidirectional or bidirectional prediction, whereby a predicted image is generated.

That is, if the execution of processing in the unidirectional prediction mode is determined, the motion vectors are supplied from the motion vector detection circuit 181 to the unidirectional prediction circuit 182, and the unidirectional prediction is performed in the unidirectional prediction circuit 182. In addition, if it is determined that the processing in the bidirectional prediction mode is performed, the motion vectors are supplied from the motion vector detection circuit 181 to the bidirectional prediction circuit 183, and bidirectional prediction is performed in the bidirectional prediction circuit 183. After the predicted image is output to the summing circuit 113 and the processing is completed in step S155 of FIG. 20, motion prediction-compensation processing ends, the processing returns to step S111 of FIG. 18 and proceeds to step S112.

As described above, by performing encoding with the predicted image generated by the filtering prediction, encoding efficiency can be increased.

Third Embodiment

Modification of the filter circuit

In the above description, the filter circuits 45 and 185 have the configuration shown in FIG. 10, but this configuration can be changed if necessary.

FIG. 21 is a block diagram illustrating another example configuration of the filter circuit 45. Configurations corresponding to the configurations shown in FIG. 10 are denoted by the same reference numerals. Excessive descriptions will be omitted whenever possible.

The difference calculation circuit 61 of FIG. 21 calculates the difference between the motion compensation image MS ₀ and the motion compensation image MC ₁ and outputs the differential image to the low pass filter circuit 62.

The low-pass filter circuit 62 performs low-pass filtering in the difference image filed from the difference calculating circuit 61 and outputs the obtained image to the summing circuit 67.

The summing circuit 67 summarizes the image supplied from the low-pass filter circuit 62 with the motion compensation image MS ₀ and outputs the obtained image as a predicted image.

Using the configuration shown in FIG. 21, the amount of processing can be reduced compared with the case of using the configuration of FIG. 10, and a high speed operation can be performed.

FIG. 22 is a block diagram illustrating another example configuration of the filter circuit 45. Configurations corresponding to the configurations shown in FIG. 10 are denoted by the same reference numerals. Excessive descriptions will be omitted whenever possible.

In the filter circuit 45 of FIG. 22, the signal is filtered in the frequency domain, and not in the time domain. Both filter circuits 45 shown in FIG. 10 and 21, filter the signal in the time domain.

The difference calculation circuit 61 of FIG. 22 calculates the difference between the motion compensation image MS ₀ and the motion compensation image MS ₁ and outputs the difference image to the orthogonal transform circuit 201.

The orthogonal transform circuit 201 performs the orthogonal transform, which is a discrete cosine transform (DCT) (DCT), a Hadamard transform, or a Karunen-Loeve transform for a difference image, and outputs the signal after the orthogonal transform to a bandpass filter circuit 202. Orthogonal conversion and filtering are performed with respect to the signal in the frequency domain, thereby the high-precision filtering operation can be performed more flexibly in comparison with performing signal filtering in the time domain.

When DCT is used as the orthogonal transform, the output DF after the orthogonal transform is expressed by the following equation (12). In equation (12), DCT (X) means that a two-dimensional DCT operation is performed on signal X.

The bandpass filter circuit 202 filters the output of the orthogonal transform circuit and outputs a signal in a certain range.

The gain tuning circuit 203 adjusts the gain of the output data of the bandpass filter circuit 202 by multiplying them by α, and also tunes the frequency component. The output XF of the gain tuning circuit 203 is expressed by the following equation (13). In equation (13), BPF (X) means that the bandpass filtering operation is performed on the signal X.

The inverse orthogonal transform circuit 204 performs the inverse orthogonal transform in a manner corresponding to the orthogonal transform performed by the orthogonal transform circuit 201 to convert the signal in the frequency domain supplied from the gain tuning circuit 203 to a signal in the time domain. For example, when DCT is used as the orthogonal transform in the orthogonal transform circuit 201, the inverse discrete cosine transform (IDCT) (ODCT) is performed in the inverse orthogonal transform circuit 204. The output X of the inverse orthogonal transform circuit 204 is expressed by the following equation (14). In equation (14), IDCT (X) denotes that a two-dimensional ODKP operation is performed on signal X.

Adder circuit 57 summarizes the signal X supplied from the inverse orthogonal transform circuit 204 with the time-domain motion compensation image MS ₀ and outputs the obtained image as a predicted image. The predicted image S (i, j), which is the final output of the summing circuit 57, is expressed by the following equation (15).

Moreover, a predicted image of high accuracy can also be generated by performing signal filtering in the frequency domain.

In addition, in the above description, filter prediction is performed using two reference frames, but two or more frames can be used as reference frames.

FIG. 23 is a diagram illustrating an example of a case of using three reference frames.

In the example of FIG. 23, where the time of the predicted frame is taken as a basis, three frames preceding it by one unit of time, by two units of time and by three units of time are considered as reference frames. The frame closest to the predicted frame and preceding it by one unit of time is considered as the reference frame R ₀ , the frame one unit of time preceding the reference frame R ₀ is considered as the reference frame R ₁ , the frame one unit of time preceding the reference frame R ₁ , is considered as a reference frame R ₂ .

FIG. 24 is a block diagram illustrating an example configuration of a filter circuit in the case of using three reference frames.

As shown in FIG. 24, the filter circuit 211 consists of a filter circuit 221 and a filter circuit 222. Both filter circuits 221 and 222 have the configuration shown in FIG. 10, 21 or 22. That is, the filter circuit 211 is configured to operate as a circuit with three inputs and one output by cascading the filter circuit 45, which is used for two inputs and one output.

Here, in the description, the motion compensation image extracted from the reference frame R ₀ will be considered as the motion compensation image MS _{0, the} motion compensation image extracted from the reference frame R ₁ as the motion compensation image MC ₁ , and the motion compensation image extracted from reference frame R ₂ - as the image of the MS ₂ motion compensation. The motion compensation images MC ₁ and MS ₂ are input to the filter circuit 221, and the motion compensation image MS ₀ is input to the filter circuit 222.

The filter circuit 221 performs filtering by considering the motion compensation images MC ₁ and MS _{2 as the} motion compensation images MC ₀ and MC ₁ of FIG. 10, etc., respectively, and outputs the intermediate output X, which are the result of filtering, to the filtering circuit 222.

The filter circuit 221 performs filtering by considering the intermediate output X and the motion compensation image MS _{0 as the} motion compensation images MS ₀ and MS ₁ of FIG. 10, etc., respectively, and outputs a filtering result serving as a predicted image.

It is also possible that a filtering circuit 211 that supports three reference frames is present in the decoding apparatus 1 of FIG. 4 or in the encoder 101 of FIG. 15 instead of the filter circuit 45.

In addition, filter circuits 221 and 222 do not necessarily have the same configuration, and individual configurations may differ from each other. For example, one has the configuration shown in FIG. 10, and the other has the configuration shown in FIG. 21. In addition, it is possible to vary the parameter used for the filter, depending on the I / O characteristics before and after filtering.

Filtering may be performed by filtering circuit 211 with respect to motion compensation images extracted from three reference frames arranged in two time directions, and not with respect to motion compensation images extracted from reference frames arranged in one time direction.

Note that in the case of using frames previous in time and following the predicted frame as reference frames, including the case described above with reference to FIG. 8, a parameter such as the tap coefficient used for filtering can be dynamically changed in accordance with the time direction or distance of the reference frames.

The transmission of compressed image data from the encoding device 101 to the decoding device 1 is accomplished through various types of media, such as a recording medium, including an optical disk, a magnetic disk and flash memory, satellite broadcasting, cable TV, the Internet and a mobile telephone network.

The above-described series of treatments can be performed using hardware and software. In the case of performing a number of operations using software, the program making up the software is installed by means of a recording medium onto a computer built into specialized hardware, a general purpose personal computer capable of performing various types of functions if various types of programs are installed on it, or so on. P.

FIG. 25 is a block diagram illustrating an example hardware configuration of a computer 300 that performs the above series of processes in accordance with a program.

A CPU (central processing unit) 301, ROM (read-only memory) 302 and RAM (random access memory) 303 are interconnected via bus 304.

The input / output interface 310 is further connected to a bus 304. An input unit 311, consisting of a keyboard, mouse, microphone, etc., an output unit 312, consisting of a display, a speaker, etc., a memory unit 313, consisting of a hard a disk, non-volatile memory, etc., a communication unit 314 consisting of a network interface, etc. and a drive 315 that controls removable media 321, such as an optical disk or semiconductor memory, is connected to an input / output interface 310.

In the computer 300 having the above configuration, the CPU 301, for example, downloads the program stored in the memory unit 313 to the RAM 303 via the input / output interface 310 and the bus 304 and performs it, thereby performing the above series of processes.

A program executed by the CPU 301 is provided, being recorded, for example, on a removable medium 321 or via a wired or wireless transmission medium such as a local area network, the Internet or digital broadcasting, and is installed in the memory unit 313.

Additionally, the program executed by the computer may be a program in which the processing is performed in a time sequence in accordance with the procedure described in this description, or it may be a program in which the processing is performed in parallel or at the right time, such as the moment of a telephone call.

In addition, in this description, the steps describing the program recorded on the recording medium, of course, include processing performed in a time sequence in accordance with the described order, and also include processing performed in parallel or separately, not in a time sequence.

In addition, in this description, the system is an entire device consisting of many devices.

In addition, with respect to the above description, described as a single device (or data processing unit), the configuration can be divided into many devices (or data processing units). Conversely, a configuration described as a plurality of devices (or data processing units) can be combined into a single device (or data processing unit). Alternatively, of course, a configuration different from the configuration described above can be added to each device (or each data processing unit). Further, part of the configuration of a device (or data processing unit) can be included in the configuration of another device (or data processing unit), as long as the configuration and operation of the entire system is almost the same. That is, embodiments of the present invention are not limited to the embodiments described above, and various changes are possible without departing from the spirit of the present invention.

For example, the above-described decoding apparatus 1 and encoding apparatus 101 may be applied to arbitrary electronic devices. Next, their examples will be described.

FIG. 26 is a block diagram illustrating an example of a basic configuration of a television receiver using a decoding apparatus 1 to which the present invention is applied.

The television receiver 1000 shown in FIG. 26 includes an air tuner 1013, a video decoder 1015, a video signal processing circuit 1018, a graphics generating circuit 1019, a display control circuit 1020, and a display board 1021.

The air tuner 1013 receives the wave broadcast signal of analogue broadcasting via the antenna, demodulates it, receives the video signal and feeds it to the video decoder 1015. The video decoder 1015 decodes the video signal supplied from the air tuner 1013 and supplies the received digital composite signal to the video signal processing circuit 1018.

The video signal processing circuit 1018 performs a specific operation, such as eliminating noise, with respect to the video data supplied from the video decoder 1015, and provides the received video data to the graphics generating circuit 1019.

The graphics generating circuit 1019 generates video data of a program to be displayed on a display board 1021, image data based on processing based on a program filed over a network, and the like. and supplies the generated video data and image data to the circuit board 1020. In addition, the graphics generating circuit 1019 performs the operation of generating video data (graphics) to display a screen to be used by a user to select a menu item, superimpose it on the program video data and feed the thus obtained video data to the board control circuit 1020, if necessary.

The board control circuit 1020 controls the display board 1021 based on data supplied from the graphics generating circuit 1019, and causes the display board 1021 to display video programs and the various types of screens described above.

The display board 1021 consists of an LCD (liquid crystal display) or the like. and displays a video program or the like. in accordance with the control action carried out by the circuit board 1020.

In addition, the television receiver 1000 includes an A / D converter circuit 1014, an audio signal processing circuit 1022, an echo / audio synthesis compensation circuit 1023, an audio amplifier circuit 1024, and a speaker 1025.

The air tuner 1013 demodulates the received wave broadcast signal, thereby receiving not only a video signal, but also an audio signal. The air tuner 1013 supplies the received audio signal to the analog-to-digital audio converter circuit 1014.

An analog-to-digital audio signal converter circuit 1014 performs an analog-to-digital conversion operation of the audio signal supplied from the broadcast tuner 1013, and supplies the received digital audio signal to the audio signal processing circuit 1022.

The audio signal processing circuit 1022 performs a certain operation, such as eliminating noise, with respect to the audio data supplied from the analog-to-digital audio converter circuit 1014 and supplies the received audio data to the echo / audio synthesis circuit 1023.

The echo / audio synthesis compensation circuit 1023 supplies the audio data supplied from the audio signal processing circuit 1022 to the audio amplifier circuit 1024.

The audio amplifier circuit 1024 performs a digital-to-analog conversion operation and an amplification operation of audio data supplied from the echo / audio synthesis circuit 1023 to adjust them according to the volume level, and causes the loudspeaker 1025 to output an audio signal.

In addition, the television receiver 1000 includes a digital tuner 1016 and an MPEG decoder 1017.

The digital tuner 1016 receives a wave broadcast digital broadcast signal (digital terrestrial broadcasting, digital broadcasting from the Sun (broadcast satellite) or SS (communication satellite)) through the antenna, demodulates it, receives MPEG transport stream (TP) and feeds it to MPEG decoder 1017 .

The MPEG decoder 1017 removes the scrambling of the MPEG TP supplied from the digital tuner 1016, and extracts a stream including data of the program to be played back (view-listen). The MPEG decoder 1017 decodes the audio packets constituting the extracted stream, and supplies the received audio data to the audio signal processing circuit 1022, and also decodes the video packets constituting the stream, and supplies the received video data to the audio signal processing circuit 1018. In addition, MPEG decoder 1017 provides electronic TV guide data extracted from MPEG TPs to CPU 1032 via a path that is not shown.

The television receiver 1000 uses the decoding device 1 described above as an MPEG decoder 1017 that decodes video packets in this manner. Note that MPEG TP transmitted from a broadcast station or the like is encoded using an encoder 101.

As with decoder 1, MPEG decoder 1017 performs decoding using the predicted image that is generated by filtering prediction. In addition, at this moment, as in the case of the decoding apparatus 1, the MPEG decoder 1017 receives a portion of the motion compensation images using the motion vector and obtains the remaining image (remaining images) by predicting the motion of the motion compensation image obtained by the motion vector. Thus, the MPEG decoder 1017 can reduce the number of motion vectors to be encoded.

The video data supplied from the MPEG decoder 1017 is subjected to certain processing on the video signal processing circuit 1018, as is the case with the video data supplied from the video decoder 1015, if necessary, video data or the like generated in the graphics generating circuit 1019 is superimposed, these video data is supplied to a display board 1021 through a board control circuit 1020, and an image related thereto is displayed.

The audio data supplied from the MPEG decoder 1017 is subjected to certain processing on the audio signal processing circuit 1022, as is the case with the audio data supplied from the analog-to-digital audio conversion circuit 1014, supplied to the audio amplifier circuit 1024 via the echo / audio synthesis circuit 1023 and are subjected to digital-to-analog conversion processing and gain processing. As a result, the volume-adjusted audio signal is output from the loudspeaker 1025.

In addition, the television receiver 1000 includes a microphone 1026 and an analog-to-digital conversion circuit 1027.

An analog-to-digital conversion circuit 1027 receives an audio signal from a user captured by a microphone 1026, which is available in the television receiver 1000 for voice communication, performs an analog-to-digital conversion operation of the received audio signal, and provides the received digital audio data to the echo / audio synthesis circuit 1023.

In the event that the audio signal from the user (user A) of the television receiver 1000 is supplied from the analog-to-digital conversion circuit 1027, the echo / audio synthesis compensation circuit 1023 performs the echo compensation of the audio data of user A and outputs these audio data that are obtained by synthesis with other audio data from speaker 1025 through a 1024 audio amplifier circuit.

Further, the television receiver 1000 has an audio encoder 1028, an internal bus 1029, and a synchronous dynamic random access memory (SDRM) 1030, flash memory 1031, CPU 1032, USB interface 1033, and network interface 1034.

An analog-to-digital conversion circuit 1027 receives an audio signal from a user captured by a microphone 1026, which is available in a television receiver 1000 for voice communication, performs an analog-to-digital conversion operation on a received audio signal and supplies the received digital audio data to an audio decoder 1028.

The audio codec-decoder 1028 converts the audio data supplied from the analog-to-digital conversion circuit 1027 into data of a specific format for transmission over a network and feeds them to a network interface 1034 via an internal bus 1029.

The network interface 1034 is connected to the network through a cable attached to the network terminal 1035. The network interface 1034 transmits audio data supplied from the audio encoder-decoder 1028, for example, to another device connected to the network. In addition, the network interface 1034 through the network terminal 1035 receives audio data transmitted, for example, from another device connected via the network, and feeds them to the audio codec 1028 via the internal bus 1029.

The audio codec 1028 converts the audio data supplied from the network interface 1034 into data of a specific format and feeds it to the echo / audio synthesis compensation circuit 1023.

The echo / audio synthesis compensation circuit 1023 compensates for the echo of the audio data supplied from the audio encoder-decoder 1028, and outputs the audio data obtained by synthesis with other audio data from the speaker 1025 through the audio amplifier circuit 1024.

SDOZU 1030 stores various types of data required by the CPU 1032 to perform processing.

The flash memory 1031 stores the program executed by the CPU 1032. The program stored in the flash memory 1031 is read by the CPU 1032 at a certain point, for example, when the television receiver 1000 is turned on. The flash memory 1031 also stores the data of the electronic TV guide received through digital broadcasting, and data received from a specific server over the network.

For example, flash memory 1031 stores MPEG TPs, including content data received from a specific server over a network, under control carried out, for example, by CPU 1032.

MPEG decoder 1017 processes MPEG TPs, as is the case with MPEG TPs supplied from digital tuner 1016. Thus, the television receiver 1000 is capable of receiving video, audio, and other content data. over the network, decode them using MPEG decoder 1017 and display video or sound output.

In addition, the television receiver 1000 has a light signal receiving unit 1037 for receiving an infrared signal from a remote control 1051.

The light signal receiving unit 1037 receives the infrared beam from the remote control 1051 and outputs a control code representing a user-supplied command obtained by demodulation to the CPU 1032.

The CPU 1032 executes the program stored in the flash memory 1031, and controls the entire operation of the television receiver 1000 in accordance with a control code or the like supplied from the light signal receiving unit 1037. The CPU 1032 is connected to individual units of the television receiver 1000 via paths that are not shown.

The USB interface 1033 transmits data to or receives data from an external device of the television receiver 1000, the device being connected via a USB cable connected to the USB terminal 1036. The network interface 1034 is connected to the network via a cable connected to the network terminal 1035, and transmits data other than audio data to various types of devices connected to the network, or receives this data from them.

The television receiver 1000 uses the decoding device 1 as the MPEG decoder 1017, as a result of which it is able to generate a predicted image of high accuracy using a small amount of control data during decoding performed in relation to the video packets forming the stream. As a result, the television receiver 100 is able to increase the coding efficiency, while suppressing the increase in processing volume.

FIG. 27 is a block diagram illustrating an example of a basic configuration of a mobile telephone that uses decoding apparatus 1 and encoding apparatus 101 to which the present invention is applied.

The mobile telephone apparatus 1100 shown in FIG. 27, includes a main control unit 1150 configured to jointly control the individual units, a power supply unit 1151, an instruction input control unit 1152, an image encoder 1153, a camera interface unit 1154, an LCD control unit 1155, an image decoder 1156, an 1157 unit multiplexer-demultiplexer, block 1162 recording-playback, block 1158 circuit modulation-demodulation and audio codec decoder 1159. They are mutually connected via bus 1160.

In addition, the mobile telephone 1100 includes an operation key 1119, a CCD camera (charge-coupled device) 1116, a liquid crystal display 1118, a storage unit 1123, a transmit-receive circuit unit 1163, an antenna 1114, a microphone 1121, and a speaker 1117.

When the call is completed or the user presses the “on” key, the power circuit unit 1151 supplies power from the battery pack to the individual units, thereby bringing the mobile telephone 1100 to working condition.

The mobile telephone 1100 performs various types of actions, such as transmitting / receiving an audio signal, transmitting / receiving email or image data, receiving an image or recording data in various types of modes, such as a sound call mode or a digital communication mode, based on a control action, carried out by the main control unit 1150, including a CPU, ROM, RAM, etc.

For example, in the sound call mode, the mobile telephone 1100 converts the audio signal picked up by the microphone 1121 into digital audio data using an audio encoder-decoder 1159, performs a spreading operation using a modulation-demodulation block 1158, and performs a digital-to-analog conversion operation and a frequency conversion operation with using block 1163 of the transmission-reception scheme. The mobile telephone 1100 transmits a signal to be transmitted obtained by conversion operations to an unshown base station via an antenna 1114. The signal to be transmitted (audio signal) transmitted to the base station is supplied to the interlocutor’s mobile telephone through a public telephone network.

In addition, for example, in the sound call mode, the mobile telephone 1100 amplifies the reception signal received by the antenna 1114 using the transmit-receive unit 1163, then performs the frequency conversion operation and the analog-to-digital conversion operation, performs the opposite operation of spreading using block 1158 of the modulation-demodulation circuit and converts it into an analog audio signal using an audio encoder-decoder 1159. The mobile telephone 1100 outputs an analog audio signal obtained by converting me, through loudspeaker 1117.

In addition, for example, in the case of electronic mail transmission in digital communication mode, the mobile telephone 1100 receives text email data entered by the operation key 1119 on the command input control unit 1152. The mobile telephone 1100 processes the text data on the main control unit 1150 and causes them to be displayed as image quality on the liquid crystal display 1118 through the LCD control unit 1155.

In addition, the mobile telephone 1100 in the main control unit 1150 generates email data based on text data or user commands received by the command input control unit 1152. The mobile telephone 1100 performs the operation of expanding the range of e-mail data using block 1158 of the modulation-demodulation scheme and performs the operation of digital-to-analog conversion and the operation of frequency conversion using block 1163 of the transmit-receive circuit. The mobile telephone 1100 transmits a signal to be transmitted obtained by conversion operations to an unproven base station via an antenna 1114. The signal to be transmitted (e-mail) transmitted to the base station is supplied to a specific destination via a network and a mail server or the like.

In addition, for example, in the case of receiving electronic mail in digital communication mode, the mobile telephone 1100 receives the signal transmitted from the base station through the antenna 1114 using the transmit-receive circuit unit 1163, amplifies it, and then performs the frequency conversion operation and the analog- digital conversion. Mobile telephone 1100 performs the opposite of expanding the spectrum of the received signal using block 1158 of the modulation-demodulation scheme to restore the original email data. The mobile telephone 1100 displays the recovered email data on the liquid crystal display 1118 through the LCD control unit 1155.

Additionally, the mobile telephone 1100 is capable of recording received e-mail data in a storage unit 1123 through a recording / reproducing unit 1162.

The storage unit 1123 is an arbitrary rewritable recording medium. The storage unit 1123 may be a semiconductor memory, such as RAM or built-in flash memory, or removable media, such as a magnetic disk, magneto-optical disk, optical disk, USB memory or memory card. Of course, other types of media may be used.

Further, for example, in the case of transmitting image data in digital communication mode, the mobile telephone 1100 generates image data by fixing with a CCD camera 1116. The CCD camera 1116 has optical devices such as a lens and aperture, and the CCD serving as an element photoelectric conversion, captures the image of the object, converts the intensity of the received light into an electrical signal and generates image data for the image of the object. The CCD camera 1116 encodes the image data using the image encoder 1153 through the camera interface unit 1154, whereby the image data is converted to encoded image data.

Mobile telephone 1100 uses the encoder 101 described above as an image encoder 1153 that performs such an operation. As with the encoder 101, the image encoder 1153 performs encoding using the predicted image that is generated by the filter prediction. In addition, at this point, as in the case of the encoder 101, the image encoder 1153 receives a portion of the motion compensation images using the motion vector and obtains the remaining motion compensation image (s) by predicting the motion of the motion compensation image obtained by the motion compensation. Accordingly, image encoder 1153 can reduce the number of motion vectors to be encoded.

Additionally, at the same time, the mobile telephone 1100 performs, on an audio encoder / decoder 1159, an analog-to-digital conversion of the sound captured by the microphone 1121 during image capturing using the CCD camera 1116, and then encodes it.

The mobile telephone 1100 at the multiplexing-demultiplexing unit 1157 multiplexes the encoded image data supplied from the image encoder 1153 and the digital audio data supplied from the audio encoder-decoder 1159 in a specific manner. Mobile telephone 1100 performs the operation of expanding the spectrum of multiplexed data obtained as a result using block 1158 of the modulation-demodulation circuit and performs the digital-to-analog conversion operation and the frequency conversion operation using block 1163 of the transmit-receive circuit. The mobile telephone 1100 transmits a signal to be transmitted obtained by conversion operations to an unshown base station via an antenna 1114. The signal to be transmitted (image data) transmitted to the base station is supplied to the opposite end of a communication line over a network or the like.

Note that in the case where image data is not transmitted, the mobile telephone 1100 can display the image data generated by the CCD camera 1116 on the liquid crystal display 1118 through the LCD control unit 1155, and not through the image encoder 1153.

In addition, for example, in the case of receiving data of a dynamic image file that is connected to a simple web page or the like, in digital communication mode, the mobile telephone 1100 receives a signal transmitted from the base station through the antenna 1114 using block 1163 transmission-reception scheme, amplifies it and then performs the frequency conversion operation and the analog-to-digital conversion operation. The mobile telephone 1100 performs the reverse operation of expanding the spectrum of the received signal to restore the original multiplexed data using block 1158 of the modulation-demodulation scheme. Mobile telephone 1100 demultiplexes the multiplexed data to encoded image data and audio data using the multiplexing-demultiplexing unit 1157.

The mobile telephone 1100 decodes the encoded image data using the image decoder 1156 to generate reproduced dynamic image data and displays the data on the liquid crystal display 1118 through the LCD control unit 1155. Accordingly, for example, dynamic image data included in a dynamic image file connected to a simple web page is displayed on the liquid crystal display 1118.

Mobile telephone 1100 uses the decoding device 1 described above as image decoder 1156 to perform such processing. That is, as with the decoding apparatus 1, the image decoder 1156 obtains a portion of the motion compensation images using the motion vector and obtains the remaining motion compensation image (s) by predicting the motion of the motion compensation image obtained by the motion compensation. Accordingly, image decoder 1156 can reduce the number of motion vectors to be encoded.

At this point, the mobile telephone 1100 converts the digital audio data into an analog audio signal using an audio encoder-decoder 1159, and outputs it from a speaker 1117. Accordingly, for example, audio data included in a dynamic image file connected to a simple web page is reproduced.

Note that, as with email, the mobile telephone 1100 can also record (store) received data connected to a simple web page or the like in a storage unit 1123 through a recording-reproducing unit 1162.

In addition, the mobile telephone 1100 can analyze the two-dimensional code obtained by the CCD camera 1116 by capturing images, and obtain data recorded in the two-dimensional code using the main control unit 1150.

In addition, the mobile telephone 1100 can communicate with an external device via infrared rays using the infrared communication unit 1181.

Using the encoder 101 as the image encoder 1153, the mobile telephone 1100 can reduce the number of motion vectors to be transmitted during encoding the image data generated by the CCD camera 1116 and transmitting the image data, thereby increasing the encoding efficiency.

Furthermore, using the decoding apparatus 1 as the image decoder 1156, the mobile telephone 1100 can generate a high-precision predicted image using a small amount of control data during decoding, which is performed while receiving data (encoded data) of a dynamic image file connected to a simple web page or the like As a result, mobile telephone 1100 can increase coding efficiency by suppressing an increase in processing volume.

Note that although the above description says that the mobile telephone 1100 uses a CCD camera 1116, an image sensor using a complementary metal oxide semiconductor (CMOS image sensor) can be used instead of the CCD camera 1116. In this case, the mobile telephone 1100 can also capture the image of the object and generate image data for the image of the object, as is the case with the use of a CCD camera 1116.

In addition, although the mobile telephone 1100 is described above, the decoding device 1 and the encoding device 101 can be applied to any device having image capturing and communication functions similar to those of the mobile telephone 1100, such as a PDA (personal electronic assistant), smartphone , UMPK (ultramobile personal computer), netbook or personal laptop, as is the case with the mobile phone 1100.

FIG. 28 is a block diagram illustrating an example of a basic configuration of a hard disk recorder that uses decoding apparatus 1 and encoding apparatus 101 to which the present invention is applied.

The hard disk recorder 1200 (HDD recorder) shown in FIG. 28 is a device that stores audio and video data of broadcast programs included in a wave broadcast signal (television signal) that is transmitted from a satellite, broadcast antenna, or the like, and which is received by a tuner on a hard disk included therein, and which provides the stored data to the user at the moment corresponding to the command received from the user.

The hard disk recorder 1200 can extract audio and video data from a wave broadcast signal, decode them accordingly, and save, for example, to the included hard disk. In addition, the hard disk recorder 1200 may receive audio and video data from another device over the network, decode them accordingly, and save, for example, to the included hard disk.

In addition, the hard disk recorder 1200 can decode the audio and video data recorded on the included hard disk, supply them, for example, to the monitor 1260, display the image related thereto on the monitor 1260, and output sound related thereto from monitor speaker 1260. In addition, the hard disk recorder 1200 can decode audio and video data extracted from a wave broadcast signal received through a tuner, or audio and video data received from another device over a network, for example, on the monitor 1260, display the image relating to them on the screen of the monitor 1260 and output related sound from the speaker of the monitor 1260.

Of course, other actions may be performed.

As shown in FIG. 28, the hard disk recorder 1200 includes a receiving unit 1221, a demodulating unit 1222, a demultiplexer 1223, an audio decoder 1224, a video decoder 1225, and a recording device control unit 1226. Further, the hard disk recorder 1200 includes an electronic TV guide data memory 1227, program memory 1228, working memory 1229, display converter 1230, additional information display control unit (ODI) 1231, display control unit 1232, recording-reproducing unit 1233, digital-to-analog a converter 1234 and a communication unit 1235.

In addition, the display converter 1230 includes a video encoder 1241. The recording / reproducing unit 1233 includes an encoder 1251 and a decoder 1252.

A receiving unit 1221 receives an infrared signal from a remote control (not shown), converts it into an electrical signal, and outputs it to a recording device control unit 1226. The recorder control unit 1226 consists, for example, of a microprocessor or the like. and performs various types of operations in accordance with a program stored in program memory 1228. At this point, the recorder control unit 1226 uses working memory 1229 if necessary.

The communication unit 1235 is connected to the network and performs a connection operation with another device over the network. For example, the communication unit 1235, controlled by the recording device control unit 1226, connects to a tuner (not shown) and outputs a channel selection control signal mainly to the tuner.

Demodulating unit 1222 demodulates the signal supplied from the tuner and outputs to demultiplexer 1233. Demultiplexer 1233 demultiplexes the data supplied from demodulating unit 1222 to audio data, video data and electronic TV guide data and outputs them to audio decoder 1224, video decoder 1225 and recording control unit 1226 respectively.

The audio decoder 1224 decodes the audio data entered therein and outputs it to the recording-reproducing unit 1233. Video decoder 1225 decodes the input video data into it and outputs it to a display converter 1230. The recorder control unit 1226 supplies the electronic TV guide data entered therein to the ETG data memory 1227 for storage.

The display converter 1230, using a video encoder 1241, encodes the video data supplied from the video decoder 1225 or the recorder control unit 1226 into video data corresponding, for example, to the NTSC system (National Committee for Television Standards), and outputs to the recording-reproducing unit 1233. In addition, the display converter 1230 converts the screen size of the video data supplied from the video decoder 1225 or the recorder control unit 1226 into a size corresponding to the size of the monitor 1260, converts them into video data corresponding to the NTCS system using the video encoder 1241, converts them into analog signals and displays them on a display control unit 1232.

The display control unit 1232, under the control of the recorder control unit 1226, superimposes a signal by displaying additional information (ODI) output from the ODI control unit 1231 onto the video signal input from the display converter 1230, displays it on the monitor display 1260, and displays it.

In addition, audio data is output to monitor 1260 that is output from audio decoder 1224 and converted into an analog signal by a digital to analog converter 1234. Monitor 1260 outputs this audio signal from a built-in speaker.

The recording / reproducing unit 1233 includes a hard disk as a recording medium for storing video, audio, and the like on it.

The recording and reproducing unit 1233 by means of the encoder 1251 encodes, for example, the audio data supplied from the audio decoder 1224. In addition, the recording and reproducing unit 1233 by the encoder 1251 encodes the video data supplied from the video encoder 1241 of the display converter 1230. A recording / reproducing unit 1233, using a compressor, combines encoded audio data and encoded video data. A recording / reproducing unit 1233 performs channel coding of the combined data to amplify it and writes the data to the hard disk using a recording head.

The recording-reproducing unit 1233 reproduces the data recorded on the hard disk using the reproducing head, amplifies them and demultiplexes them to the audio and video data using the demultiplexer. Block 1233 recording-playback using the decoder 1252 decodes the audio and video data. The recording-reproducing unit 1233 performs digital-to-analog conversion of the decoded audio data and outputs them to the monitor speaker 1260. In addition, the recording-reproducing unit 1233 performs the digital-to-analog conversion of decoded video data and displays them on the monitor display 1260.

The recording device control unit 1226 reads the latest electronic TV guide data from the electronic TV guide data memory 1227 based on a user command indicated by the infrared signal that is received from the remote control and received through the receiving unit 1221 and provides them to the ODI control unit 1231. The ODI control unit 1231 generates image data corresponding to the entered electronic TV guide data and outputs it to the display control unit 1232. The display control unit 1232 outputs the video data inputted from the ODI control unit 1231 to the monitor display 1260 and displays them. Accordingly, the electronic TV guide (ETG) is displayed on the monitor 1260.

In addition, the hard disk recorder 1200 may receive various types of data, such as video data, audio data, or ETG data supplied from another device over a network, such as the Internet.

The communication unit 1235 is controlled by the recording device control unit 1226, receives encoded video data, audio data, and electronic TV guide data transmitted from another device over the network, and provides to the recording device control unit 1226. The recording device control unit 1226 supplies the obtained encoded video and audio data to the recording / reproducing unit 1233 and, for example, stores it on the hard disk. At this point, the recorder control unit 1226 and the recording / reproducing unit 1233 can, if necessary, perform an operation such as re-encoding.

In addition, the recorder control unit 1226 decodes the received encoded video and audio data and provides the received video data to the display converter 1230. The display converter 1230 processes the video data supplied from the recorder control unit 1226, as well as the video data supplied from the video decoder 1225, provides to the monitor 1260 through the display control unit 1232 and displays an image.

In addition, in accordance with the display of this image, the recorder control unit 1226 can supply the decoded audio data to the monitor 1260 through a digital-to-analog converter 1234 and output sound from the speaker.

Further, the recorder control unit 1226 decodes the received encoded data of the electronic TV guide data and supplies the decoded data of the electronic TV guide to the electronic TV guide data memory 1227.

The hard disk recorder 1200 described above uses a decoding apparatus 1 as a video decoder 1225, a decoder 1252, and a decoder included in the recorder control unit 1226. That is, the video decoder 1225, the decoder 1252, and the decoder included in the recorder control unit 1226 obtain a portion of the motion compensation images using the motion vector, and the remaining motion compensation image (s) are obtained by predicting the motion compensation image motion obtained by the vector motion, as in the case of decoding apparatus 1. Accordingly, video decoder 1225, decoder 1252, and a decoder included in the recorder control unit 1226 can reduce a layer of motion vectors to be encoded.

Thus, the hard disk recorder 1200 can generate a high-precision predicted image with a small amount of control data during decoding, which is performed when the tuner or communication unit 1235 receives video data (encoded data), or when the recording / reproducing unit 1233 reproduces video data (encoded data) from the hard drive. As a result, the hard disk recorder 1200 can increase coding efficiency by suppressing an increase in processing volume.

In addition, the hard disk recorder 1200 uses the encoder 101 as an encoder 1251. Thus, the encoder 1251 obtains a portion of the motion compensation images using the motion vector and obtains the remaining motion compensation image (s) by predicting the motion of the motion compensation image obtained using the motion vector, as in the case of the encoder 101.

Therefore, the hard disk recorder 1200 can reduce the number of motion vectors by writing encoded data to the hard disk, thereby increasing the encoding efficiency.

Note that although the above is a description of a hard disk recorder 1200 for recording video and audio data to a hard disk, of course, any type of recording medium may be used. For example, decoding device 1 and encoding device 101 can be applied to a recording device that uses a recording medium other than a hard disk, for example, a flash memory, an optical disk or a video cassette, as is the case with the above-described device 1200 recording to a hard disk .

FIG. 29 is a block diagram illustrating an example of a basic configuration of a camera that uses a decoding apparatus 1 and an encoding apparatus 101 to which the present invention is applied.

The camera 1300 shown in FIG. 29 captures an image of an object, displays an image of an object on an LCD 1316, and writes it onto a recording medium 1333 as image data.

The lens unit 1311 transmits light (i.e., an image of the object) to the CCD / CMOS 1312. The CCD / CMOS 1312 is an image sensor using a CCD or CMOS, it converts the received light intensity into an electrical signal and feeds it to the camera signal processing unit 1313.

The camera signal processing unit 1313 converts the electrical signal supplied from the CCD-CMOS 1312 into color difference signals Y, Cr, and Cb and supplies them to the image signal processing unit 1314. The image signal processing unit 1314 performs certain image processing with respect to the image signal supplied from the camera signal processing unit 1313 and encodes the image signal using an encoder 1341 under the control of a control device 1321. The image signal processing unit 1314 provides encoded data that is generated by encoding the image signal to the decoder 1315. In addition, the image signal processing unit 1314 receives the data to be displayed generated on the additional display unit 1320 tional data (DAR), and supplies them to the decoder 1315.

In the processing described above, the camera signal processing unit 1313 appropriately uses the DOS (Dynamic RAM) 1318 connected via the bus 1317 and, if necessary, stores image data, encoded data obtained by encoding the image data, and the like in the DOS 1318.

The decoder 1315 decodes the encoded data supplied from the image signal processing unit 1314, and provides the received image data (decoded image data) to the LCD 1316. In addition, the decoder 1315 supplies the data to be displayed supplied from the image signal processing unit 1314 to the LCD 1316 The LCD 1316 appropriately combines the image of the decoded image data supplied from the decoder 1315 and the image of the data to be displayed, and displays the combined image.

The additional information display unit 1320, under the control of the control device 1321, outputs data to be displayed, such as a menu screen consisting of characters, letters or numbers, and icons, to the image signal processing unit 1314 via the bus 1317.

The control device 1321 performs various types of operations based on a signal indicating the contents of the command issued by the user using the command unit 1322, and controls the image signal processing unit 1314, the DOS 1318, the external interface 1319, the additional information display unit 1320, the drive 1323 for carriers, etc. .d. via bus 1317. Programs, data, etc., which are necessary for the control device 1321 to perform various types of operations are stored on a flash ROM 1324.

For example, the control device 1321 can encode image data stored on the DOS 1318 and decode the encoded data stored on the DOS 1318 for the image signal processing unit 1314 or decoder 1315. At this point, the control device 1321 may perform an encoding-decoding operation in a manner similar to with the encoding-decoding method of the image signal processing unit 1314 or the decoder 1315, or may perform the encoding-decoding operation in a manner incompatible with the signal processing unit 1314 ia or decoder 1315.

In addition, for example, if a command was issued from the command unit 1322 to start printing the image, the control device 1320 reads the image data from the DOS 1318 and feeds them to a printer 1334 connected to the external interface 1319 via the bus 1317 for printing.

In addition, for example, if the command to record the image was received from the command unit 1322, the control device 1321 reads the encoded data from the DOSE 1318 and supplies it via the bus 1317 for saving to the recording medium 1333 installed in the media drive 1323.

The recording medium 1333 is an optional removable and readable medium, such as a magnetic disk, magneto-optical disk, optical disk, or semiconductor memory. Of course, the recording medium 1333 can be any type of removable medium, it can be a storage device on a tape, disk or memory card. Of course, the recording medium 1333 may be a contactless IC card or the like.

In addition, the media drive 1323 and the recording medium 1333 can be combined and implemented as a non-portable recording medium, such as an internal hard drive or solid state drive (SSD, solid state drive) and the like.

The external interface 1319 consists of, for example, a USB I / O terminal or the like. and connected to the printer 1334 in the case of printing the image. In addition, the drive 1331 is optionally connected to an external interface 1319 onto which removable media 1322, such as a magnetic disk, optical disk or magneto-optical disk, is appropriately mounted, and the computer program read from it is installed on flash ROM 1324 if necessary.

Further, the external interface 1319 includes a network interface connected to a specific network, such as a local area network or the Internet. The control device 1321 can read the encoded data from the DOSE 1318 and send it from the external interface 1319 to another device connected via the network, in accordance with the command received, for example, from the command unit 1322. In addition, the control device 1321 can through the external interface 1319 receive encoded data or image data filed from another device over the network, and save them to DOSE 1318 or submit them to the image signal processing unit 1314.

The camera 1300 described above uses the decoding apparatus 1 as a decoder 1315. That is, the decoder 1315 obtains a portion of the motion compensation images using the motion vector, and obtains the remaining image (remaining images) by predicting the motion of the motion compensation image obtained by the motion vector as in the case of decoding apparatus 1. Accordingly, decoder 1315 can reduce the number of motion vectors to be encoded.

Therefore, the camera 1300 can generate a high-precision predicted image using a small amount of control data when reading image data generated on a CCD / CMOS 1312, or encoded video data from a DOS 1318 or recording medium 1333, or when receiving encoded video data over a network. As a result, camera 1300 can increase coding efficiency by suppressing an increase in processing volume.

In addition, the camera 1300 uses the encoder 101 as an encoder 1341. The encoder 1341 receives a portion of the motion compensation images using the motion vector, and obtains the remaining image (remaining images) by predicting the motion of the motion compensation image obtained by the motion vector, as in the case encoder 101. Accordingly, encoder 1341 may reduce the number of motion vectors to be encoded.

Therefore, the camera 1300 can reduce the number of motion vectors when writing encoded data to the DOS 1318 or recording medium 1333 or when providing encoded data to another device, for example, and can increase the encoding efficiency.

In addition, the decoding method of decoding device 1 can be applied to decoding processing performed by control device 1321. Similarly, the encoding method of encoding device 101 can be applied to encoding processing performed by control device 1321.

In addition, image data captured by the camera 1300 may be a dynamic or still image.

Of course, the decoding device 1 and the encoding device 101 can be applied to a device or system other than the devices described above.

In addition, the size of macroblocks can be set arbitrarily. The present invention can be applied to various sizes of macroblocks, for example, as shown in FIG. 30. For example, the present invention can be applied to an expanded macroblock of 32 × 32 pixels (extended macroblock), as well as to a conventional macroblock of 16 × 16 pixels.

At the top of FIG. 30 from left to right shows macroblocks consisting of 32 × 32 pixels, which are divided into blocks (compartments) of 32 × 32 pixels, 32 × 16 pixels, 16 × 32 pixels and 16 × 16 pixels. In addition, in the middle part, blocks consisting of 16 × 16 pixels, which are divided into blocks of 16 × 16 pixels, 16 × 8 pixels, 8 × 16 pixels and 8 × 8 pixels, are shown from left to right. Further, in the lower part from left to right, blocks of 8 × 8 pixels are shown, which are divided into blocks of 8 × 8 pixels, 8 × 4 pixels, 4 × 8 pixels, and 4 × 4 pixels.

That is, a macroblock of 32 × 32 pixels can be processed as blocks of 32 × 32 pixels, 32 × 16 pixels, 16 × 32 pixels and 16 × 16 pixels shown in the upper part.

A block of 16 × 16 pixels, shown in the upper right, can be processed as blocks of 16 × 16 pixels, 16 × 8 pixels, 8 × 16 pixels and 8 × 8 pixels, shown in the middle part, as in the H.264 method / Avc.

A block of 8 × 8 pixels, shown in the middle part on the right, can be processed as blocks of 8 × 8 pixels, 8 × 4 pixels, 4 × 8 pixels and 4 × 4 pixels, shown in the lower part, as in the H.264 method / Avc.

These blocks can be divided into the following three levels. That is, blocks of 32 × 32 pixels, 32 × 16 pixels and 16 × 32 pixels shown at the top of FIG. 30 are assigned to the first level. The 16 × 16 pixel block shown at the top right and the 16 × 16 pixel blocks, 16 × 8 pixels and 8 × 16 pixels shown in the middle portion are assigned to the second level. A block of 8 × 8 pixels, shown in the middle part on the right, and blocks of 8 × 8 pixels, 8 × 4 pixels, 4 × 8 pixels and 4 × 4 pixels, shown in the lower part, to the third level.

By adopting such a hierarchical structure for blocks of 16 × 16 pixels or less, you can define a larger block as a superset of blocks, while maintaining compatibility with the H.264 / AVC method.

For example, decoding apparatus 1 and encoding apparatus 101 may generate predicted images for individual layers. In addition, for example, the predicted image generated by the decoding apparatus 1 and the encoding apparatus 101 in the first level, which is a level of a larger block size than in the second level, can also be used for the second level.

Macroblocks in which encoding is performed using a relatively large block size, as in the first and second level, have a relatively small volume of the high-frequency component. On the contrary, it is believed that macroblocks in which coding is performed using a relatively small block size, as in the third level, have a relatively large volume of high-frequency component.

Accordingly, by individually generating predicted images in accordance with corresponding levels of different block sizes, an improvement in performance convenient for local image properties can be achieved.

List of Reference Items

1 - Decoding device

21 - Prediction-compensation scheme

41 - Prediction mode determination circuit

42 - Unidirectional prediction scheme

43 - Bidirectional prediction scheme

44 - Prediction Scheme

45 - Filter circuit

51 - Motion compensation circuit

52 - Motion prediction scheme

61 - Difference calculation circuit

62 - Low-pass filter circuit

63 - Gain tuning circuit

64 - High pass filter circuit

65 - gain tuning circuit

66 - Summing scheme

67 - Summarizing scheme.

Claims (29)

1. An image processing device comprising a circuit configured to: decoding an encoded image; performing a first motion compensation by using the first motion vector of the encoded image to generate a first motion compensation image; performing a second motion compensation by using the second motion vector of the encoded image to generate a second motion compensation image; generating the predicted image by performing filtering processing and summing processing on the first motion compensation image and the second first motion compensation image. 2. The image processing apparatus of claim 1, wherein the filtering processing extracts a high-frequency component of the motion compensation image. 3. The image processing device according to claim 1, wherein the first motion compensation image is generated from a reference frame formed from a decoded image; and the second motion compensation image is generated from a reference frame different from the reference frame from which the first motion compensation image is generated. 4. The image processing apparatus according to claim 2, wherein the high-frequency component of the motion compensation image is extracted by using the correlation in the time direction included in the motion compensation images. 5. The image processing device according to claim 1, in which the second motion compensation image is the same or similar to the first motion compensation image generated from the reference frame using a certain cost function common to an encoding device that encodes the image, the second motion compensation image serving as the motion compensation image corresponding to the predicted image. 6. The image processing device according to claim 5, in which the cost function is a function for calculating the total sum of the absolute values of the difference values of individual pixel values between the first image compensation image and the target block for processing the reference frame. 7. The image processing device according to claim 5, in which the cost function is a function for calculating the minimum quadratic error of individual pixel values between the first image compensation image and the target block for processing the reference frame. 8. The image processing apparatus of claim 1, wherein the predicted image generating process includes: performing low-pass filtering of the differential image between the first motion compensation image and the second motion compensation image; performing high-pass filtering of an image obtained by low-pass filtering; and summing the image obtained by low-pass filtering and the image obtained by high-pass filtering with the first motion compensation image or the second motion compensation image, whereby a predicted image is generated. 9. The image processing apparatus of claim 8, wherein the summing process includes summing the image obtained by low-pass filtering and the image obtained by high-pass filtering with the motion compensation image generated from the frame per unit time of the previous time of the predicted image . 10. The image processing device according to claim 1, further comprising a circuit configured to: receiving an identification mark for determining whether the predicted image should be generated by the unidirectional prediction performed by the unidirectional prediction means, whether the predicted image should be generated by the bi-directional prediction performed by the bidirectional prediction means, or the predicted image should be generated by filtering processing; and judging by checking with the identification mark received by the receiving means whether the predicted image should be generated by unidirectional prediction, whether the predicted image should be generated by bidirectional prediction, or the predicted image should be generated by filtering processing. 11. The image processing device according to claim 10, further comprising a circuit configured to: performing unidirectional prediction using a plurality of motion compensation images to generate a predicted image; and performing bidirectional prediction using a plurality of motion compensation images to generate a predicted image. 12. An image processing method comprising the steps of: decode the encoded image; performing a first motion compensation by using the first motion vector of the encoded image to generate a first motion compensation image; performing a second motion compensation by using the second motion vector of the encoded image to generate a second motion compensation image; and generating a predicted image by performing filtering processing and summing processing on the first motion compensation image and the second first motion compensation image.

Images