JP2009071586A - Image resolution converting device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform resolution conversion of even an encoded image encoded by an encoding system using spatial prediction at high speed. <P>SOLUTION: A variable-length decoding part 11 variable-length-decodes an encoded image and extracts code information including a conversion coefficient and an encoding mode. A converting part 12 uses the conversion coefficient to perform resolution conversion of a residual in spatial prediction and generate a residual to be referred to by other blocks. A pixel to be referred to generating part 13 generates a value of a pixel to be referred to, which is referred to by the other blocks from the encoding mode and the residual to be referred to. An estimation value stored in a buffer 14 is updated by the value of the pixel to be referred to. A prediction value generating part 15 generates a prediction value to a corresponding block from the encoding mode and the prediction value. An adding part 16 adds the residual and the prediction value and outputs the value of a resolution-converted pixel. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image resolution conversion device, and more particularly to an image resolution conversion device that converts the resolution of an encoded image by combining transform coding and predictive coding.

  As a method for converting the resolution of video content, a method for generating a reduced image from code information has been proposed.

Patent Document 1 describes an image playback device that plays back a reduced image (thumbnail) without performing DCT inverse transform. This substitutes the luminance DCT coefficient at the coordinates (0,0) of each block of the coding unit for the I picture, that is, the specific DCT coefficient in each block of the coding unit for the I picture. Is adopted as it is, and for P picture and B picture, thumbnail data is constructed by adding a motion compensation value to the DCT coefficient.
JP 2003-219420 A

  In the image reproduction device described in Patent Document 1, a reduced image whose resolution is converted by adopting a method of extracting only the DC component of the coordinates (0,0) in the DCT coefficient of each block of the encoding unit. Is generated. However, this method has a problem that it cannot be applied to an encoding method using spatial prediction. For example, MPEG-4 AVC / H.264 introduces intra prediction that is spatial prediction. In intra prediction, a prediction value is adaptively generated from surrounding pixel values, and the prediction value and Error (prediction error) and a prediction method are encoded. In this case, since the transform coefficient stored as a result of encoding is obtained by encoding a prediction error, a reduced image is obtained even if only a specific transform coefficient is extracted from each block as in Patent Document 1. Cannot be generated.

  An object of the present invention is to solve the above-described problems of the prior art and to provide an image resolution conversion apparatus capable of converting resolution at high speed even for an encoded image encoded by an encoding method using spatial prediction. .

  In order to solve the above-described problems, the present invention provides an image resolution conversion apparatus that converts the resolution of a transform-coded image, and performs variable-length decoding on an input encoded image, and includes code information including a transform coefficient and a coding mode. A variable-length decoding unit that extracts a signal, and a transform that converts a resolution in the spatial prediction using the transform coefficient extracted by the variable-length decoding unit and generates a referenced residual that is referenced by another block A reference pixel generation unit configured to generate a value of a reference pixel referred to by another block from the encoding mode extracted by the variable length decoding unit and the reference residual generated by the conversion unit; A buffer in which the accumulated prediction value is updated with the value of the referenced pixel generated by the referenced pixel generation unit, the encoding mode extracted by the variable length decoding unit, and the buffer A prediction value generating unit that generates a prediction value in the encoding mode when the resolution of the block is converted from a prediction value stored in the fa, a prediction value generated by the prediction value generating unit, and A first feature is that an addition means for adding the residual of the resolution converted by the conversion means and outputting the value of the pixel whose resolution has been converted is provided.

  In the present invention, the conversion unit receives the conversion coefficient extracted by the variable length decoding unit as an input, and performs a process in which conversion and resolution conversion opposite to encoding are integrated, and the resolution of the block is converted. A second feature is that the difference is generated.

  The present invention is also characterized in that the conversion means generates a referenced residual that is referenced by another block by weighted averaging the conversion coefficients extracted by the variable length decoding means in accordance with resolution conversion. There are three features.

  In addition, the present invention has a fourth feature in that the transforming unit generates a plurality of referenced residuals according to directions in which other blocks refer to the transform coefficient extracted by the variable length decoding unit. .

  Further, the present invention provides the reference relationship represented by the encoding mode, wherein the referenced pixel generation means receives the encoding mode extracted by the variable length decoding means and the referenced residual generated by the conversion means. The fifth feature is that the value of the referenced pixel to be referenced by another block is generated according to the resolution.

  Further, according to the present invention, the referenced pixel generating unit refers to a referenced pixel that is referenced by another block from the referenced residual generated by the converting unit and one or more predicted values accumulated in the buffer. The sixth feature is that the value of is approximately generated according to the prediction mode.

  Further, according to the present invention, the reference pixel generation unit has a plurality of approximation expressions, and adaptively applies one of the plurality of approximation expressions according to a prediction value stored in the buffer. The seventh feature is that the value of the referenced pixel referenced by another block is approximately generated.

  Further, according to the present invention, the prediction value generation means receives the encoding mode extracted by the variable length decoding means and the value of the prediction value stored in the buffer, and stores the one stored in the buffer. The eighth feature is that the predicted value of the referenced pixel when the resolution of the block is converted from the predicted value is approximately generated according to the prediction mode.

  According to the present invention, it is possible to perform resolution conversion in a code area without decoding up to a pixel area, even in an encoded image that has been transcoded by an encoding method using spatial prediction. Furthermore, since decoding and resolution conversion processing are integrated, resolution conversion can be performed at high speed.

  The present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a basic configuration of an image resolution conversion apparatus according to the present invention. The image resolution conversion apparatus includes a variable length decoding unit 11, a conversion unit 12, a referenced pixel generation unit 13, a buffer 14, a predicted value generation unit 15, and an addition unit 16. Each of these units can be realized by hardware or software.

  In the following, an example will be described in which an input image is encoded in a format according to H.264, and the resolution is converted to 1/4 in both vertical and horizontal directions. First, the outline of the processing in each unit will be described, and the processing in the conversion unit 12, the referenced pixel generation unit 13, and the predicted value generation unit 15 will be specifically described later.

  The variable length decoding unit 11 performs variable length decoding on the input encoded image and extracts code information necessary for resolution conversion. The decoding here is variable length decoding, that is, partial decoding up to code information. Also, the code information necessary for resolution conversion is a coding mode (prediction direction for each block) and conversion coefficient in intra-frame prediction. Of the code information extracted by the variable length decoding unit 11, the transform coefficient is sent to the transform unit 12, and the coding mode is sent to the referenced pixel generation unit 13 and the predicted value generation unit 15.

  The conversion unit 12 generates a residual Li subjected to resolution conversion from the conversion coefficient sent from the variable length decoding unit 11 and a referenced residual hi, vi, di to which other blocks refer. Here, as the referenced residuals hi, vi, di, the average value of the lowest row, the average value of the rightmost column, and the lower right value of the transform coefficient of the block are generated. The average value may be a plurality of partial average values depending on the resolution. The resolution-converted residual Li is sent to the adder 16, and the referenced residuals hi, vi, and di are sent to the referenced pixel generator 13.

  The referenced pixel generation unit 13 refers to the encoding mode transmitted from the variable length decoding unit 11 and the predicted values hr ′, vr ′, dr ′ stored in the buffer 14, and is transmitted from the conversion unit 12. The reference residuals hi, vi, di are compensated to generate reference pixel values hi ′, vi ′, di ′. The predicted values hr ′, vr ′, dr ′ stored in the buffer 14 are updated with the newly generated reference pixel values hi ′, vi ′, di ′. The updated predicted values hr ′, vr ′, dr ′ are referred to in the processing for the following blocks.

  The prediction value generation unit 15 uses the encoding mode transmitted from the variable length decoding unit 11 and the prediction values hr ′, vr ′, dr ′ stored in the buffer 14 to predict the prediction value in the prediction mode of the block. Li ′ is generated. The predicted value Li ′ thus generated is sent to the adding unit 16.

  The adding unit 16 receives the residual Li decoded and resolution-converted by the converting unit 12 and the predicted value Li ′ generated by the predicted value generating unit 15, and adds the two to obtain a resolution-converted pixel value P ( = Li + Li ′).

  Next, processing in the conversion unit 12 will be described. The H.264 intra-frame encoded image is obtained by transforming the residual by Intra prediction into the frequency domain by 4x4 integer transform using spatial correlation. In order to generate a pixel value that has undergone resolution conversion, first, the conversion unit 12 performs resolution conversion on a conversion coefficient that is a residual in spatial prediction. This resolution conversion can be speeded up by performing a process in which the resolution conversion is represented by a matrix and reverse conversion (conversion opposite to encoding) and resolution conversion are integrated.

  When the 4x4 conversion coefficient is X, the resolution conversion matrix is R, and the 4x4 inverse transformation matrix is T, the residual Li when the resolution is converted (reduced) to 1/4 in the vertical and horizontal directions by the average value is given by Equation (1) expressed.

Li = R ^t T ^t XTR (1)

  Here, t represents a transposition operation. Since the matrices R and T are fixed values, they can be calculated in advance, thereby reducing the processing load during resolution conversion.

  Since the value Li obtained by the equation (1) is a residual, using the value Li as it is does not result in an image whose resolution is converted from the original image. In H.264, a prediction value is generated from neighboring pixels by Intra prediction, which is spatial prediction, and therefore it is necessary to realize processing equivalent to this in the code region. However, when resolution conversion is performed, information necessary for generating a predicted value is lost, and there is a problem that a correct predicted value cannot be generated as it is. For example, in mode0, the predicted value is generated using the block above the current block, but if the bottom row of the upper block (4x4) is white and the rest is black, only the part used for prediction (white area) is Since the reduced value is different from the value when the entire block is reduced, a correct predicted value cannot be generated from the value when the entire block is reduced by resolution conversion.

  In order to solve the above problem, the conversion unit 12 generates a reduced value for each region used for prediction, separately from the value when the entire block is reduced by resolution conversion. The value reduced for each region used for prediction can be generated by weighted averaging the transform coefficients sent from the variable length decoding unit 11 according to the resolution conversion.

  For example, when the matrix for extracting the rightmost column is S, the average value h of the rightmost column of the transform coefficients of the block whose resolution is converted to 1/4 can be obtained by Expression (2).

h = R ^t T ^t XTS (2)

  Similarly, the average value v and the lower right value d of the bottom row of the transform coefficients of the block whose resolution has been converted to 1/4 can be obtained by Expressions (3) and (4), respectively.

v = S ^t T ^t XTR (3)
d = S ^t T ^t XTS (4)

  In the above description, the resolution is converted to 1/4 resolution. However, by changing the resolution conversion matrix R, it is possible to cope with resolution conversion at other magnifications. Also, when reducing the resolution, the original high-frequency component cannot be expressed, so it is possible to eliminate the high-frequency component from equations (1) to (4) in advance, thereby reducing the amount of calculation. can do.

  As described above, the transform unit 12 does not completely decode the transform-coded image up to the pixel region, but the resolution-converted residual Li and the object that other blocks refer to in intra-frame prediction in the code region. Generate reference residuals hi, vi, di. That is, the conversion unit 12 obtains the resolution-converted residual Li by integrating inverse conversion and resolution conversion, and further calculates the average value hi of the rightmost column and the average value vi of the bottom row of the conversion coefficients of the block. , The lower right value di is obtained as a plurality of referenced residuals corresponding to the directions to which other blocks refer in the prediction mode.

  Since Li, hi, vi, and di are all residuals, the residual Li is sent to the adder 16 to reconstruct the pixel value P, and the referenced residuals hi, vi, di are used as the referenced pixels. The value is sent to the generation unit 13 to reconstruct the reference pixel values hi ′, vi ′, di ′. The adder 16 is provided with the predicted value Li ′ from the predicted value generator 15 in order to reconstruct the pixel value P, and the referenced pixel generator 13 receives the reference pixel values hi ′, vi ′. , di ′ are provided with predicted values hr ′, vr ′, dr ′ from the buffer 14.

  Next, processing in the referenced pixel generation unit 13 will be described. The referenced pixel generation unit 13 includes the prediction mode extracted by the variable length decoding unit 11 and the reference residuals hi, vi, di that have undergone resolution conversion by the conversion unit 12 and the prediction value hr ′ accumulated in the buffer 14. , vr ′, dr ′ are input. The referenced pixel generator 13 generates the reference pixel values hi ′, vi ′, di ′ from the referenced residuals hi, vi, di and the predicted values hr ′, vr ′, dr ′ according to the prediction mode. Then, the predicted values hr ′, vr ′, dr ′ accumulated in the buffer 14 are updated with the reference residuals hi ′, vi ′, di ′ generated thereby.

FIG. 2 is an explanatory diagram of a referenced pixel referred to in spatial prediction. As shown in the figure, among the transform coefficients of blocks adjacent to the block, the average value of the transform coefficients in the left column for the block is h _L , the value of the transform coefficient in the lower left is d _L , and the transform coefficient in the upper left the value d _uL, an average value of the transform coefficients of the ascending v _u, the value of d _u in the upper right of the transform _{coefficients,} the mean value of the transform coefficients in the upper right line v _uR, the value of the upper-right row right transform coefficients d _uR Then, these values are stored in the buffer 14 as predicted values.

FIG. 3 shows a prediction mode in intra prediction, and FIG. 4 shows a reference relationship in generation of a prediction value. As shown in FIG. 3, each prediction mode indicates one of nine types with different prediction directions. Note that mode2 is prediction based on an average value. As shown in FIG. 4, a 4x4 predictive value R _{N (N} = A~G), to represent the value of the pixels neighboring in _{P N (N = A~L, X} ), mode3 of predicting diagonally If it may generate an R _N using a P _N and equation (5) to (11). Note that adding 2 and dividing by 4 is a rounding process.

_{R A = (P A + 2P} B + P C +2) / 4 (5)
R _B = (P _B + 2P _C + P _D +2) / 4 (6)
R _C = (P _C + 2P _D + P _E +2) / 4 (7)
R _D = (P _D + 2P _E + P _F +2) / 4 (8)
R _E = (P _E + 2P _F + P _G +2) / 4 (9)
R _F = (P _F + 2P _G + P _H +2) / 4 (10)
R _G = (P _G + 3P _H +2) / 4 (11)

Here, the predicted value for the referenced pixel in the rightmost column corresponds to the average value of R _D , R _E , R _F , and R _G. Assuming P _D ≒ P _E, (average of formula (8) ~ (11)) R D, R E, R F, the average value of R _G can be approximated by v _uR. In addition, since the predicted value for the reference pixel in the bottom row corresponds to the average of R _D , R _E , R _F , and R _G , it can be similarly approximated by v _uR . The predicted value for the lower right pixel d is R _G , but can be approximated by d _uR assuming P _{H ≈P G.}

Therefore, according to the equations (12) to (14), the reference pixel values hi ′, vi ′ are obtained by adding the respective predicted values v _uR , v _uR , d _uR for each of the referenced residuals hi, vi, di. , di ′ can be calculated.

hi ′ ＝ hi ＋ v _uR (12)
vi '= vi + v _uR (13)
di '= di + d _uR (14)

As described above, the reference pixel generation unit 13 generates the reference residuals hi, vi, di generated by the conversion unit 12 and one or more predicted values v _uR , v _uR , d accumulated in the buffer 14. _The reference pixel values hi ′, vi ′, and di ′ that other blocks refer to from _uR are approximately generated according to the prediction mode.

  Referenced pixel values hi ′, vi ′, and di ′ referenced by other blocks can be approximated by interpolation from the values of one or more other referenced pixels, and are stored in the buffer 14. It is also possible to approximately obtain the reference pixel values hi ′, vi ′, and di ′ by selectively applying an approximate expression corresponding to the distribution of predicted values.

FIG. 5 is an explanatory diagram of processing for approximately obtaining the value of the referenced pixel referenced by another block by interpolation. Here, a quadratic curve (1) is formed based on d _uL , d _u , d _uR (FIG. 2), and the pixel value P _E is determined according to this. It is possible to obtain the v _uR if Motomare pixel value P _E according to a quadratic curve (1), formula (12) can be obtained approximately the hi ', vi' (13). In this case, the method of interpolating the pixel value P _E, other quadratic curve (1), shown (2), as shown in (3), can be set as appropriate. Although illustration (2) corresponds to the case (assuming that P _D ≒ P _E) utilized directly as the pixel value P _E pixel value P _D. In addition, a plurality of approximate expressions for approximately obtaining the values of the reference pixels hi ′, vi ′, and di ′ are prepared, and one of them is used as a predicted value (h _L , d stored in the buffer 14. _L , d _uL , v _u , d _u , v _uR , d _uR , or some of the distributions), and this is applied to reference pixels that other blocks refer to It is also possible to approximate the value of.

Although the above is the case of mode 3, the values hr ′, vr ′, and dr ′ of the reference pixel can be reconstructed by approximating the other prediction modes by the same method. In short, the referenced pixel generation unit 13 calculates the reference pixel value hr ′ from the referenced residuals hi, vi, di and the predicted values h _L , d _L , d _uL , v _u , d _u , v _uR , d _uR. , vr ′ and dr ′ may be generated approximately. The reference pixel values hr ′, vr ′, and dr ′ reconstructed as described above are stored in the buffer 14 as predicted values that are referred to in subsequent blocks.

  Next, processing in the predicted value generation unit 15 will be described. The prediction value generation unit 15 receives the prediction mode extracted by the variable length decoding unit 11 and the prediction values hr ′, vr ′, dr ′ accumulated in the buffer 14, and calculates the prediction value Li ′ for the resolution-converted block. Generate.

  In vertical / horizontal 1/4 resolution conversion, one pixel is generated as a predicted value Li ′ from the values of neighboring reference pixels in accordance with the encoding mode. For example, when the prediction mode is mode 3, the prediction value L ′ can be obtained by Expression (15).

Li ′ ＝ (hr ′ ＋ vr ′) / 2 (15)

  In the case of other prediction modes as well, the prediction value Li ′ can be generated by approximating similarly. The predicted value Li ′ thus generated is sent to the adding unit 16.

  As described above, the prediction value generation unit 15 receives the encoding mode extracted by the variable length decoding unit 11 and the prediction value stored in the buffer 14 as input, and one or more prediction values stored in the buffer. From this, the predicted value of the referenced pixel when the resolution of the block is converted is approximately generated according to the prediction mode.

  The adding unit 16 receives the residual Li subjected to resolution conversion by the converting unit 12 and the predicted value Li ′ generated by the predicted value generating unit 15, and adds both to add a pixel value P (= Li + Li ′) is generated.

  Although the embodiment has been described above, the present invention is not limited to the above embodiment. For example, in the above embodiment, the input image is in a format according to H.264 and is encoded by Intra prediction. However, in other formats, if the image is encoded by spatial prediction, the processing target is You can also An image encoded by spatial prediction is easy to edit.

1 is a block diagram showing a basic configuration of an image resolution conversion apparatus according to the present invention. It is explanatory drawing of the to-be-referenced pixel referred by spatial prediction. It is explanatory drawing of the prediction mode in intra prediction. It is explanatory drawing of the reference relationship in the production | generation of a predicted value. It is explanatory drawing of the process which calculates | requires the value of a reference pixel approximately.

Explanation of symbols

11 ... variable length decoding unit, 12 ... conversion unit, 13 ... referenced pixel generation unit, 14 ... buffer, 15 ... predicted value generation unit, 16 ... addition unit

Claims (8)

In an image resolution converter for converting the resolution of a transform-coded image,
Variable-length decoding means for variable-length decoding an input encoded image and extracting code information including a transform coefficient and an encoding mode;
Transform means for converting the residual in spatial prediction using the transform coefficient extracted by the variable length decoding means and generating a referenced residual that is referenced by another block;
A referenced pixel generating means for generating a value of a referenced pixel referenced by another block from the encoding mode extracted by the variable length decoding means and the referenced residual generated by the converting means;
A buffer in which the accumulated predicted value is updated with the value of the referenced pixel generated by the referenced pixel generation means;
Prediction value generation means for generating a prediction value in the encoding mode when the resolution of the block is converted from the encoding mode extracted by the variable length decoding means and the prediction value stored in the buffer; ,
Image resolution conversion characterized by comprising addition means for adding the predicted value generated by the predicted value generation means and the residual subjected to resolution conversion by the conversion means to output a pixel value subjected to resolution conversion apparatus.   The conversion means receives the conversion coefficient extracted by the variable length decoding means as input, and generates a resolution-converted residual of the block by processing that integrates conversion and resolution conversion opposite to encoding. The image resolution conversion apparatus according to claim 1, wherein:   3. The conversion unit according to claim 2, wherein the conversion unit generates a referenced residual that is referred to by another block by performing weighted averaging of the conversion coefficient extracted by the variable length decoding unit according to resolution conversion. The image resolution conversion apparatus described.   3. The image resolution conversion according to claim 2, wherein the conversion unit generates a plurality of referenced residuals according to directions in which other blocks refer to the conversion coefficient extracted by the variable length decoding unit. apparatus.   The referenced pixel generation means receives the encoding mode extracted by the variable length decoding means and the referenced residual generated by the conversion means, and depends on the reference relationship and resolution represented by the encoding mode. The image resolution conversion apparatus according to claim 1, wherein a value of a referenced pixel that is referenced by another block is generated.   The referenced pixel generation means sets the value of the referenced pixel referenced by another block from the reference residual generated by the conversion means and one or more prediction values stored in the buffer to a prediction mode. 6. The image resolution conversion apparatus according to claim 5, wherein the image resolution conversion apparatus generates the signal approximately in response.   The referenced pixel generation means has a plurality of approximation formulas, and adaptively applies one of the plurality of approximation formulas according to a prediction value stored in the buffer to be referenced by another block. The image resolution conversion apparatus according to claim 6, wherein the reference pixel value to be generated is approximately generated.   The prediction value generation means receives the encoding mode extracted by the variable length decoding means and the value of the prediction value stored in the buffer, and based on one or more prediction values stored in the buffer The image resolution conversion apparatus according to claim 1, wherein the prediction value of the reference pixel when the resolution of the block is converted is approximately generated according to the prediction mode.

Images