JP2009118303A - Image encoding apparatus, image encoding method, image decoding apparatus, and image decoding method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent deterioration in image quality while reducing a code amount in image encoding using multiple resolution analysis. <P>SOLUTION: An interpolation processing unit 104 selects a pixel to be processed and performs interpolation prediction thereon in a plurality of directions (horizontal direction, vertical direction). A direction selecting unit 105 selects an interpolation prediction value in a direction where an interpolation error is smaller. A pixel separating unit 106 separates, as a high frequency component, a difference between the interpolation prediction value and a pixel value of the pixel to be processed. Furthermore, an interpolation processing unit 107 selects a pixel to be processed from among pixels that are not defined as an object to be processed, and performs interpolation prediction thereon in a plurality of directions (oblique direction to right, oblique direction to left). A direction selecting unit 108 and a pixel separating unit 109 select an interpolation prediction value and separate a high frequency component of the target pixel similarly. A conversion/quantization unit 110 converts the separated high frequency component into encoded data. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image encoding device and an image encoding method for efficiently encoding image information, and an image decoding device and an image decoding method for decoding encoded image information.

  The JPEG2000 standard is defined as one of methods for recording and transmitting image information by compression encoding. In the JPEG2000 system, the whole or part of an image is separated into a low-frequency component and a high-frequency component by repeatedly using the Wavelet transform in the horizontal direction and the vertical direction, and these components are quantized and encoded respectively. . As described above, when the frequency components of the image are separated in the horizontal direction and the vertical direction by multi-resolution analysis, the same number of frequency coefficients as the pixels of the image are obtained.

  Furthermore, as a technique for separating frequency components in directions other than the horizontal direction and the vertical direction, there is a technique described in Non-Patent Document 1, for example.

Ero Simoncelli, William Freeman: “The Steerable Pyramid: A Flexible Architecture for Multi-scale Derivative Computation”, 2nd IEEE Intercontinental Promotion. vol III, pp 444-447. October, 1995.

  In the Wavelet conversion technique such as the existing JPEG2000 system, Wavelet conversion is performed in the horizontal direction and the vertical direction, and therefore, in the case of an image having a strong correlation in other directions, frequency components cannot be efficiently separated. Therefore, if frequency components are separated not only in the horizontal and vertical directions but also in other directions, encoding can be performed with less degradation of image quality. However, there is a problem that the number of frequency coefficients becomes larger than the number of pixels of the image, the code amount increases, and the encoding efficiency deteriorates. Even in the method described in Non-Patent Document 1, it is expected that the number of coefficients increases and the coding efficiency deteriorates, but this is not considered.

  The present invention has been made in view of the above problems, and an object of the present invention is to prevent deterioration of image quality while reducing the code amount in image encoding using multi-resolution analysis.

  The image encoding device of the present invention selects a first processing target pixel from the input image at a predetermined interval, and uses the other reference pixels not to be processed, and uses the first plurality of pixels for each processing target pixel. A first interpolation processing unit that performs an interpolation prediction with respect to the interpolation direction, and an interpolation prediction value in a direction with a small interpolation error among the interpolation prediction values for a plurality of directions that are interpolated and predicted by the first interpolation processing unit. A first direction selection unit that selects a first pixel separation unit that separates a difference between an interpolation prediction value selected by the first direction selection unit and a pixel value of the processing target pixel as a high-frequency component; A high-frequency component of the first processing target pixel separated by the first pixel separation unit, and a conversion / quantization unit that converts / quantizes pixel values of other reference pixels that are not to be processed, Coefficients transformed and quantized by the transform / quantization unit are output as encoded data. And a code output section for.

  Furthermore, with respect to the pixels that have not been processed by the first interpolation processing unit, a second processing target pixel is selected at a predetermined interval, and each processing target is used using other reference pixels that are not to be processed. A second interpolation processing unit that performs interpolation prediction with respect to the second plurality of interpolation directions for each pixel, and an interpolation error among the interpolation prediction values for the plurality of directions that have been subjected to interpolation prediction by the second interpolation processing unit. A second direction selection unit that selects an interpolation prediction value in a small direction, and a difference between the interpolation prediction value selected by the second direction selection unit and the pixel value of the processing target pixel is separated as a high frequency component A second pixel separation unit, wherein the transform / quantization unit has a high frequency component of the second processing target pixel separated by the second pixel separation unit together with a high frequency component of the first processing target pixel. Convert components and pixel values of other reference pixels that were not processed To quantization.

  In the image encoding method of the present invention, as the first resolution decomposition step, the first processing target pixel is selected from the input image at a predetermined interval, and each processing is performed using other reference pixels not to be processed. Interpolation prediction is performed with respect to the first plurality of interpolation directions for each target pixel, and an interpolation prediction value in a direction with a small interpolation error is selected from the interpolation prediction values for the plurality of interpolation prediction directions, and the selected interpolation is performed. A step of separating the difference between the predicted value and the pixel value of the processing target pixel as a high frequency component, and the high frequency component of the first processing target pixel separated by the first resolution decomposing step is not set as a processing target The pixel values of the other reference pixels are converted and quantized, and the converted and quantized coefficients are output as encoded data.

  Further, as the second resolution decomposition step, the second processing target pixel is selected at a predetermined interval with respect to the pixels that were not processed in the first resolution decomposition step, and other references that are not the processing target. Interpolation prediction is performed with respect to the second plurality of interpolation directions for each processing target pixel using the pixels, and interpolation prediction values in a direction with a small interpolation error among interpolation prediction values for the plurality of interpolation prediction directions are obtained. Selecting, and separating the difference between the selected interpolated prediction value and the pixel value of the processing target pixel as a high frequency component, and the first processing target pixel separated in the first resolution decomposition step Along with the high-frequency component, the high-frequency component of the second pixel to be processed separated in the second resolution decomposition step and the pixel values of the other reference pixels not to be processed are converted and quantized.

  The image decoding apparatus according to the present invention performs inverse quantization and inverse transform on the coding coefficient of the coded stream, and obtains a pixel value or a high frequency component, and a high frequency component is obtained. A second pixel to be processed is selected at a predetermined interval from the pixel, and interpolation prediction is performed with respect to the second plurality of interpolation directions for each pixel to be processed using another reference pixel from which a pixel value is obtained. A second interpolation processing unit to perform, and a second direction selection unit that selects an interpolation prediction value in a direction with a small interpolation error among interpolation prediction values for a plurality of directions predicted to be interpolated by the second interpolation processing unit A second pixel synthesizing unit that synthesizes a pixel value by adding a high-frequency component of the processing target pixel to the interpolated prediction value selected by the second direction selecting unit, and a second pixel synthesizing unit And an image output unit that outputs the synthesized image.

  Furthermore, for the pixel from which the high-frequency component that has not been processed by the second interpolation processing unit is obtained, another reference is obtained by selecting the first processing target pixel at a predetermined interval and obtaining the pixel value. A first interpolation processing unit that performs interpolation prediction with respect to the first plurality of interpolation directions for each processing target pixel using the pixels, and a plurality of directions that are interpolated and predicted by the first interpolation processing unit. A first direction selection unit that selects an interpolation prediction value in a direction with a small interpolation error among the interpolation prediction values, and a high-frequency component of the pixel to be processed is added to the interpolation prediction value selected by the first direction selection unit. In addition, a first pixel synthesis unit that synthesizes pixel values, and the image output unit is synthesized by the first pixel synthesis unit together with the image synthesized by the second pixel synthesis unit. Output an image.

  The image decoding method of the present invention performs inverse quantization and inverse transform on the coding coefficient of the coded stream, obtains a pixel value or a high frequency component, and obtains a high frequency component as the second resolution synthesis step. A second pixel to be processed is selected at a predetermined interval from the pixel, and interpolation prediction is performed with respect to the second plurality of interpolation directions for each pixel to be processed using another reference pixel from which a pixel value is obtained. The interpolation prediction value in the direction with the smaller interpolation error is selected from the interpolation prediction values for a plurality of directions predicted by interpolation, and the pixel value is synthesized by adding the high-frequency component of the processing target pixel to the selected interpolation prediction value. And outputting the image synthesized in the second resolution synthesis step.

  Further, as a first resolution synthesis step, a first processing target pixel is selected at a predetermined interval for a pixel from which a high frequency component that has not been processed in the second resolution synthesis step is obtained, and a pixel value Interpolation prediction is performed with respect to the first plurality of interpolation directions for each processing target pixel using the other reference pixels obtained, and among the interpolation prediction values for the plurality of interpolation prediction directions, the interpolation error value is calculated. An image synthesized in the second resolution synthesis step, comprising: selecting an interpolation prediction value in a small direction; and adding a high-frequency component of the processing target pixel to the selected interpolation prediction value to synthesize a pixel value. At the same time, the image synthesized in the first resolution synthesis step is output.

  ADVANTAGE OF THE INVENTION According to this invention, in the encoding and decoding process of an image, it becomes possible to reduce the code amount of encoding data and to prevent the degradation of the image quality of a decoded image.

  Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of an image coding apparatus according to the present invention.
The image encoding apparatus 1 includes an image input unit 101, an image prediction unit 102, a resolution decomposition memory 103, a transform / quantization unit 110, and a code output unit 111 as an encoding block 10, and the resolution decomposition memory 103 includes 2 The system interpolation processing units 104 and 107, the direction selection units 105 and 108, and the pixel separation units 106 and 109 are connected. Further, an inverse quantization / inverse transform unit 112, a resolution synthesis memory 113, and a decoded image memory 120 are connected to the output terminal of the transform / quantization unit 110 as a decoding block 20 and input to the image prediction unit 102. Two systems of interpolation processing units 114 and 117, direction selection units 115 and 118, and pixel synthesis units 116 and 119 are connected to the resolution synthesis memory 113.

  The operation of each component of the image encoding device 1 will be described in detail. In addition, although the following operation | movement is set as the autonomous operation | movement of each component, it can also be implement | achieved by the software which the control part and memory | storage part which are not shown in figure memorize | store.

First, the encoding block 10 will be described.
An original image is input to the image input unit 101. The image prediction unit 102 performs prediction using the original image and the decoded image (past, future, or current frame) from the decoded image memory 120, and calculates the difference. Note that the original image may be processed as it is without predicting the image. In that case, the image prediction unit 10 and the decoding block 20 can be omitted.

  The resolution decomposition memory 103 is a memory that holds the image data subjected to the multi-resolution decomposition. The multi-resolution decomposition in this embodiment is performed by two resolution resolution steps. The processing performed by the interpolation processing unit 104, the direction selection unit 105, and the pixel separation unit 106 is referred to as a first step, and the processing performed by the interpolation processing unit 107, the direction selection unit 108, and the pixel separation unit 109 is referred to as a second step. To.

  First, the first step will be described. The interpolation processing unit 104 performs interpolation prediction on the processing target pixel using the reference pixel. For the processing target pixel at that time, when the pixel coordinates are (x, y), for example, pixels at positions where only one of x and y is odd are selected. Interpolation prediction is performed for each processing target pixel in a plurality of directions, for example, in two directions, a horizontal direction and a vertical direction. As the reference pixel, a pixel that is in the vicinity of the processing target pixel and is arranged in the interpolation direction is used, but a pixel that is not selected as the processing target pixel is used. This is because the data of the pixel to be processed is replaced with the difference value after the interpolation process. The reference pixels used for the interpolation are, for example, four points arranged in a line across the processing target pixel. As the interpolation method (interpolation calculation formula), linear interpolation, quadratic or cubic polynomial interpolation, or the like is used. Note that a plurality of interpolation methods may be prepared, and the interpolation method may be switched according to the characteristics of the reference pixel to be used. In this case, it is preferable to select an optimal interpolation method by paying attention to the average value and the variance value of the reference pixels as the characteristics of the reference pixels.

  The direction selection unit 105 selects a prediction value in one direction from prediction values for a plurality of interpolation directions (horizontal direction and vertical direction) interpolated and predicted by the interpolation processing unit 104. The prediction value selection method uses, for example, the total value of the L1 norm, which is the sum of the absolute value errors of the pixel values between adjacent pixels, or the like, in a direction in which this value is minimized (or a value within a predetermined range). Direction) predicted value. That is, the direction in which the predicted value continues more smoothly with respect to the value of the reference pixel is selected.

  The pixel separation unit 106 takes the difference between the pixel value of the processing target pixel and the predicted value selected by the direction selection unit 105, and stores this difference as a high frequency component ΔP1 of the processing target pixel in another region of the resolution decomposition memory 103. . Thus, the high frequency component data ΔP1 is separated and stored for the processing target pixel.

  Next, the second step will be described. The interpolation processing unit 107 targets pixels that are not selected as processing targets in the first step. Among them, for example, a pixel at an odd position in both coordinates x and y is selected as a processing target, and interpolation prediction is performed using the other pixels as reference pixels. Interpolation prediction is performed in a plurality of directions. In this case, for example, two directions of 45 ° right diagonal direction (hereinafter, right diagonal direction) and 45 ° left diagonal direction (hereinafter, left diagonal direction) are different from the first step. To do. This is because the pixels arranged in the same direction (horizontal direction and vertical direction) as the first step already include the pixel to be processed in the first step and replaced with a high-frequency component, so that it can be used as a reference pixel. It is not possible. In this case as well, for example, four reference pixels arranged in a line are used as reference pixels for interpolation. As the interpolation method, linear interpolation, quadratic or cubic polynomial interpolation or the like is used. However, a plurality of interpolation methods may be prepared and an optimum method may be selected.

  The direction selection unit 108 selects a prediction value in one direction from prediction values for a plurality of interpolation directions (right diagonal direction and left diagonal direction) predicted by interpolation by the interpolation processing unit 107. As in the first step, the prediction value selection method uses, for example, the total value of the L1 norm, which is the sum of absolute value errors between adjacent pixels, and selects the prediction value in the direction in which this value is minimized.

  The pixel separation unit 109 takes the difference between the value of the processing target pixel and the predicted value selected by the direction selection unit 108, and stores this difference as a high frequency component data ΔP2 of the processing target pixel in another area of the resolution decomposition memory 103. . Thereby, the high frequency component data ΔP1 and ΔP2 are separated and stored for the pixels to be processed in the first step and the second step.

Next, the transform / quantization unit 110 encodes the image data subjected to multi-resolution decomposition according to the characteristics of each data. For example, the pixel data P0 that has not been processed is divided into blocks, subjected to discrete cosine transform, and the coefficients are quantized to generate encoded data. Coefficients are quantized for the high frequency component data ΔP1 and ΔP2 of the processing target pixel to generate encoded data.
The code output unit 111 outputs the generated encoded data.

  As described above, in the present embodiment, in the resolution decomposition process, the interpolation processing units 104 and 107 that perform interpolation prediction with respect to a plurality of directions, and the direction selection units 105 and 108 that select prediction values in a direction with a smaller interpolation error are provided. It is characterized by the provision. In order to perform efficient processing in consideration of the pixel arrangement, it is characterized by comprising two resolution resolution steps.

Next, the decoding block 20 will be described.
The inverse quantization / inverse transform unit 112 performs inverse quantization and inverse transform on the encoded data created by the transform / quantization unit 110 to return to pixel data P0 and high frequency component data ΔP1, ΔP2.

  The resolution synthesizing memory 113 is a memory that holds the image data that has been subjected to the multiresolution decomposition in order to restore the original image. The restoration of the multiresolution decomposed image is performed by a two-stage resolution synthesis step corresponding to the two-stage resolution decomposition step in the encoded block. The processing performed by the interpolation processing unit 114, the direction selection unit 115, and the pixel separation unit 116 returns the first step to the original, and is called a reverse first step. The interpolation processing unit 117, the direction selection unit 118, and the pixel separation unit The process performed by 119 is to return the second step to the original, and will be referred to as a reverse second step.

First, processing is performed from the reverse second step. This is because the reference pixel data used for the interpolation prediction is acquired without shortage.
As in the second step, the interpolation processing unit 117 selects a pixel at a position where both coordinates x and y are odd numbers as a processing target, and performs interpolation prediction. As in the second step, the interpolation directions are two directions, a right diagonal direction and a left diagonal direction. The reference pixels in the interpolation use four points arranged in a line, and the interpolation method uses linear interpolation, quadratic or cubic polynomial interpolation, or the like.

  The direction selection unit 118 selects a prediction value in one direction from prediction values for a plurality of directions (right diagonal direction and left diagonal direction) predicted by interpolation by the interpolation processing unit 117. As in the second step, the prediction value selection method selects, for example, the prediction value in the direction in which this value is minimum using the total value of the L1 norm, which is the sum of absolute value errors between adjacent pixels.

  The pixel synthesis unit 119 calculates the sum of the high-frequency component ΔP2 of the pixel to be processed and the predicted value selected by the direction selection unit 118, and stores this in the resolution synthesis memory 113 as the original pixel value. As a result, the original image is restored by synthesizing the high-frequency component ΔP2 of the processing target pixel.

Next, reverse first step processing is performed.
As in the first step, the interpolation processing unit 114 selects a pixel at a position where only one of the coordinates x and y is an odd number as a processing target, and performs interpolation prediction. The interpolation directions are two directions, the horizontal direction and the vertical direction, as in the first step. The reference pixels in the interpolation use four points arranged in a line, and the interpolation method uses linear interpolation, quadratic or cubic polynomial interpolation, or the like.

  The direction selection unit 115 selects a predicted value in one direction from predicted values for a plurality of directions (horizontal direction and vertical direction) interpolated and predicted by the interpolation processing unit 114. As in the first step, the prediction value selection method uses, for example, the total value of the L1 norm, which is the sum of absolute value errors between adjacent pixels, and selects the prediction value in the direction in which this value is minimized.

  The pixel synthesis unit 116 calculates the sum of the high-frequency component ΔP1 of the processing target pixel and the predicted value selected by the direction selection unit 115, and stores this as the original pixel value in the resolution synthesis memory 113. As a result, the original image is restored by synthesizing the high-frequency component ΔP1 for the processing target pixel.

  The decoded image synthesized by the reverse second step and the reverse first step is stored in the decoded image memory 120 and used for prediction by the image prediction unit 102.

In this embodiment, image coding is performed by the above configuration and operation, but the following modifications are possible.
First, the order of resolution decomposition processing in the coding block may be such that the second step is performed first and then the first step is performed. Alternatively, the case where the first step is performed first may be compared with the case where the second step is performed first, and either one may be selected and encoded together with the information of the order. However, the order of resolution composition processing in the decoding block is made to correspond to the order of resolution decomposition processing in the coding block. That is, when the second step is performed first in the coding block, the reverse first step is performed first in the decoding block.

  Further, in this embodiment, the interpolation directions in the interpolation processing units 104 and 107 are selected in the horizontal direction and the vertical direction, and the right diagonal direction and the left diagonal direction in consideration of the orthogonal pixel arrangement, but are not limited thereto. Alternatively, four or more directions may be selected.

  FIG. 3 is a diagram for explaining in detail the multiresolution decomposition method in the present embodiment. In this embodiment, the multi-resolution decomposition process is performed by dividing into (a) the first step and (b) the second step, and the selection of the pixels at that time is illustrated.

  In the first step (a), horizontal / vertical interpolation prediction 301 is performed. Here, a position where one of x and y is an odd number is selected as the processing target pixel (x mark) as the horizontal axis position x and the vertical axis position y. Here, reference numeral 310 denotes a processed pixel, and reference numeral 311 denotes a pixel currently being processed. Others are reference pixels (circles) 312, but as reference pixels for the currently processed pixel 311, four neighboring pixels (circles) 313 arranged in the horizontal direction and four neighboring pixels arranged in the vertical direction. Pixels (● marks) 314 are used. Interpolation prediction in the horizontal direction and the vertical direction is performed using these reference pixels. As an interpolation method, linear interpolation, quadratic or cubic polynomial interpolation, or the like is used. The predicted value in the horizontal direction is compared with the predicted value in the vertical direction, and a value with a smaller interpolation error is selected.

  The difference between the value of the processing target pixel 311 and the selected predicted value is taken and becomes the high frequency component data ΔP1 of the half-point difference image 302. The half-point difference image 302 has half the number of pixels as the original image, and the high-frequency component data ΔP1 has a smaller value as a result of selecting the smaller interpolation error. The unprocessed pixels 312 in the first step remain as the half-point original image 303, and the number of pixels is ½ of the original image.

  Next, in the second step (b), interpolation prediction 304 in the diagonally right / left diagonal direction is performed on the pixel 312 of the half-point original image 303 that has not been selected as the processing target in the first step (a). I do. In the diagram of (b), the pixel to be processed in the first step is not displayed. Here, a position where both x and y are odd numbers is selected as a processing target pixel (x mark). Reference numeral 320 denotes a processed pixel, reference numeral 321 denotes a pixel currently being processed, and other reference pixels 322 are used for interpolation prediction. As reference pixels for the pixel 321 currently being processed, four neighboring pixels (◎) 323 arranged in the diagonally right direction and four neighboring pixels (● symbol) 324 arranged in the diagonally left direction are used. Using these reference pixels, interpolation prediction in the right diagonal direction and left diagonal direction is performed. These predicted values are compared, and a value with a smaller interpolation error is selected.

  The difference between the value of the processing target pixel 321 and the selected predicted value is taken, and this becomes the high frequency component data ΔP2 of the ¼ point difference image 305. The 1/4 difference image 305 has 1/4 of the number of pixels of the original image, and the high frequency component data ΔP2 has a smaller value as a result of selecting the smaller interpolation error. In the second step, the unprocessed pixel 322 becomes a quarter-point original image 306, the number of pixels of which is 1/4 of the original image, which remains the original pixel value P0.

  The data ΔP 1, ΔP 2, P 0 of the 1 / 2-point difference image 302, 1 / 4-point difference image 305, and 1 / 4-point original image 306 decomposed in this way are stored in the resolution decomposition memory 103, The transform / quantization unit 110 transforms / quantizes 307, 308, and 309 to obtain encoded data. Of these, the half-point difference image 302 and the quarter-point difference image 305 are obtained by encoding the high-frequency component data ΔP1 and ΔP2 having small values, so that the code amount is smaller. In this case, the ratio (difference ratio) of the number of pixels subjected to the differentiation process with respect to the entire pixels reaches 3/4. Subsequently, if the above resolution decomposition process is repeated for the pixels of the unprocessed quarter-point original image 306, the difference ratio can be further improved. This is called resolution decomposition divided into a plurality of layers. For example, the difference rate is improved to (3/4) + (1/4) × (3/4) = 15/16 by one iteration, and the code amount can be further reduced.

  FIG. 4 is a diagram illustrating an example of a prediction value selection method in the direction selection units 105 and 108. Here, an evaluation method using the total value of the L1 norm, which is the sum of absolute value errors of pixel values between adjacent pixels, is shown. As shown in FIG. 4, when the pixel values (for example, luminance components) of adjacent pixels P1 to P4 sequentially become values of 100, 200, 100, and 200, the total value T of the four L1 norms is 100 + 100 + 100 = 300. The total value T is obtained for each of a plurality of interpolation directions, and the direction showing the smallest value is selected. According to this evaluation method, it is possible to select a direction in which an interpolation error with an adjacent pixel becomes smaller, that is, a direction in which pixel values continue more smoothly. A similar selection can be made by evaluating the variance of pixel values.

  FIG. 5 is a diagram illustrating another example of setting reference pixels. In the reference pixels in the first step of FIG. 3, four points arranged in a line in the horizontal direction and in the vertical direction, respectively, were used. On the other hand, in this example, with respect to the processing target pixel 500, four points (reference numeral 501) arranged on the left side, four points arranged on the right side (reference numeral 502), and four points arranged on the upper side (reference numeral 503). Interpolation prediction using four points (reference numeral 504) arranged on the lower side. Alternatively, an average value of four points may be taken, and interpolation may be performed from two points in the horizontal direction and two points in the vertical direction. According to this method, by increasing the number of reference pixels, an error at the time of interpolation calculation is reduced, and the code amount can be further reduced.

  FIG. 6 is a diagram illustrating a flowchart of an image encoding method according to the present embodiment. For simplicity, a case will be described here where the original image is encoded without prediction (no difference processing).

In step 601, an image is input.
In step 602, an image range to be processed is set and multi-resolution decomposition is started. When encoding in a plurality of hierarchies, the first time starts from the highest hierarchy (entire image), and in the case of repeated processing, the next lower hierarchy (unprocessed image) is targeted.

In step 603, resolution resolution in the first step is started. First, a processing target pixel and its reference pixel are set. Here, interpolation prediction is performed in the horizontal direction and the vertical direction, with the position where one of x and y is an odd number as a processing target pixel and the other as a reference pixel.
In step 604, two values predicted by interpolation are compared, and the predicted value with the smaller interpolation error is selected.

In step 605, the difference between the predicted value selected in step 604 and the pixel value of the processing target pixel is taken and separated as a high frequency component ΔP1 of the target pixel.
In step 606, it is determined whether all the pixels to be processed in the first step have been completed. If not completed, the process returns to step 603 to process the next pixel. When completed, go to step 607.

In step 607, resolution resolution in the second step is started. First, a processing target pixel and its reference pixel are set. Here, among the pixels not targeted in the first step (step 603), the positions where both x and y are odd numbers are set as processing target pixels, and the other pixels are set as reference pixels, with respect to the right diagonal direction and the left diagonal direction. Interpolate prediction.
In step 608, two values predicted by interpolation are compared, and a predicted value having a smaller interpolation error is selected.

In step 609, the difference between the predicted value selected in step 608 and the pixel value of the processing target pixel is taken and separated as a high frequency component ΔP2 of the target pixel.
In step 610, it is determined whether or not the processing target pixels in the second step are all processed. If not completed, the process returns to step 607 to process the next pixel. When completed, go to step 611.

  In step 611, when encoding in a plurality of layers, it is determined whether or not the lowest layer has been reached. When the next layer is repeated, the process returns to step 602, the image range including unprocessed pixels is set as a processing target, and the resolution decomposition (first step and second step) of the next layer is repeated. If not, the process proceeds to step 612.

In step 612, pixel data (high-frequency components ΔP1, ΔP2 and unprocessed pixel value P0) subjected to multi-resolution decomposition is converted and quantized to generate encoded data.
In step 613, the generated encoded data is output.

  Thus, when performing multi-resolution decomposition, by performing interpolation prediction with different directions in two steps, encoding processing can be performed efficiently according to the pixel arrangement. Also, by repeating these steps, resolution can be decomposed to a desired hierarchy, and the code amount can be further reduced.

FIG. 2 is a block diagram showing an embodiment of an image decoding apparatus according to the present invention.
The image decoding apparatus 2 includes a stream analysis unit 201, an image prediction unit 202, an inverse quantization / inverse conversion unit 203, and a resolution synthesis memory 204. The resolution synthesis memory 204 includes two interpolation processing units 205 and 208, directions. The selection units 206 and 209 and the pixel synthesis units 207 and 210 are connected. The decoded image memory 211 and the image output unit 212 are connected to the output terminal of the resolution composition memory 204.

The operation of each component of the image decoding device 2 will be described in detail. Here, a case where encoded data created by the image encoding device 1 of the first embodiment is input will be described.
The stream analysis unit 201 analyzes the input encoded stream. Here, the stream analysis unit 201 also performs data extraction processing from packets and information acquisition processing of various headers and flags.

The image prediction unit 202 performs prediction using information such as a flag and a motion vector and a decoded image (past, future, or current frame) from the decoded image memory 211 to create a predicted image. Note that when the input stream is an original image, image prediction need not be performed.
The inverse quantization / inverse transform unit 203 performs inverse quantization and inverse transform on the encoded data, and returns it to pixel data and high frequency component data.

  The resolution synthesis memory 204 is a memory that holds original image data synthesized from multi-resolution decomposed image data. In this embodiment, the multi-resolution decomposed image is synthesized in two stages of resolution synthesis steps. The processing performed by the interpolation processing unit 205, the direction selection unit 206, and the pixel synthesis unit 207 is an inverse process of the first step of the first embodiment (FIG. 1) and is called an inverse first step. The processing performed by the selection unit 209 and the pixel synthesis unit 210 is an inverse process of the second step and is referred to as an inverse second step.

  Here, the order of the reverse first step and the reverse second step at the time of decoding is made to correspond to the order of the first step and the second step at the time of encoding. That is, when the first step is performed first at the time of encoding, the reverse second step is performed first at the time of decoding, and when the second step is performed first at the time of encoding, the first step is performed at the time of decoding. The reverse first step is performed. Thereby, the reference pixel data used for the interpolation prediction can be acquired without being insufficient.

In the present embodiment, processing is first performed from the reverse second step in correspondence with the coding conditions of the first embodiment.
The interpolation processing unit 208 selects a pixel at a position where both the coordinates x and y are odd numbers as a processing target, and performs interpolation prediction as in the second step at the time of encoding. The pixel at this position is a pixel to be processed at the time of encoding, and is provided as high frequency component data ΔP2 in the encoded data. As in the second step, the interpolation directions are two directions, a right diagonal direction and a left diagonal direction. By setting in this direction, reference pixel data can be acquired without shortage. The reference pixels in the interpolation use four points arranged in a line, and the interpolation method uses linear interpolation, quadratic or cubic polynomial interpolation, or the like.

  The direction selection unit 209 selects a prediction value in one direction from prediction values for a plurality of directions (right oblique direction and left oblique direction) predicted by interpolation by the interpolation processing unit 208. As in the second step, the prediction value selection method selects, for example, the prediction value in the direction in which this value is minimum using the total value of the L1 norm, which is the sum of absolute value errors between adjacent pixels.

  The pixel synthesis unit 210 calculates the sum of the high-frequency component ΔP2 of the processing target pixel and the predicted value selected by the direction selection unit 209, and stores this in the resolution synthesis memory 204 as the original pixel value. As a result, the original image is restored by synthesizing the high-frequency component ΔP2 of the processing target pixel.

Next, reverse first step processing is performed.
As in the first step, the interpolation processing unit 205 selects a pixel at a position where only one of the coordinates x and y is an odd number as a processing target, and performs interpolation prediction. The pixel at this position is also given as the high-frequency component data ΔP1, and has not been synthesized yet. The interpolation directions are two directions, the horizontal direction and the vertical direction, as in the first step. Pixels in this direction can be used as reference pixels by combining pixel values as a result of the reverse second step. The reference pixels in the interpolation use four points arranged in a line, and the interpolation method uses linear interpolation, quadratic or cubic polynomial interpolation, or the like.

  The direction selection unit 206 selects a prediction value in one direction from prediction values for a plurality of directions (horizontal direction and vertical direction) predicted by interpolation by the interpolation processing unit 205. As in the first step, the prediction value selection method uses, for example, the total value of the L1 norm, which is the sum of absolute value errors between adjacent pixels, and selects the prediction value in the direction in which this value is minimized.

  The pixel synthesis unit 207 calculates the sum of the high-frequency component ΔP1 of the processing target pixel and the predicted value selected by the direction selection unit 206, and stores this as the original pixel value in the resolution synthesis memory 204. As a result, the original image is restored by synthesizing the high-frequency component ΔP1 for the processing target pixel.

  As a result of the reverse second step and the reverse first step, pixel value data is synthesized for all the pixels given as the high frequency component. The combined decoded image is stored in the decoded image memory 211 and output from the image output unit 212. The decoded image is used for prediction in the image prediction unit 202.

  As described above, in this embodiment, in the resolution synthesis process, the interpolation processing units 205 and 208 that perform the interpolation prediction with respect to a plurality of directions, and the direction selection units 206 and 209 that select the prediction value in the direction with the smaller interpolation error are included. It is characterized by the provision. In order to perform efficient processing in consideration of the pixel arrangement, it is configured by two steps. As a result, it is possible to efficiently decode the encoded data (encoded stream) encoded by reducing the code amount with high accuracy, and to prevent degradation of the decoded image.

  FIG. 7 is a diagram illustrating a flowchart of the image decoding method according to the present embodiment. For simplicity, here, a case will be described in which an original image is decoded without prediction (without differentiation processing).

In step 701, an encoded stream is input.
In step 702, the encoded data is inversely quantized and inversely transformed to return to pixel value and high frequency component data.

In step 703, an image range to be processed is set and multi-resolution composition is started. When decoding in a plurality of layers, the first time starts from the lowest layer (partial image), and in the case of iterative processing, the next higher layer (unprocessed image) is targeted.
In step 704, the resolution composition of the reverse second step is started. First, a processing target pixel and its reference pixel are set. Here, interpolation prediction is performed for the right diagonal direction and the left diagonal direction, with the positions where both x and y are odd numbers as processing target pixels and the other positions as reference pixels.

In step 705, the two values predicted by interpolation are compared, and the predicted value with the smaller interpolation error is selected.
In step 706, the high-frequency component ΔP2 of the processing target pixel is added to the prediction value selected in step 705 to synthesize the pixel value of the target pixel.

In step 707, it is determined whether all the processing target pixels in the reverse second step have been completed. If not completed, the process returns to step 704 to process the next pixel. When completed, go to step 708.
In step 708, the resolution synthesis of the reverse first step is started. First, a processing target pixel and its reference pixel are set. Here, among the pixels not targeted in the second reverse step (step 704), the position where one of x and y is an odd number is the processing target pixel, and the other is the reference pixel, with respect to the horizontal and vertical directions. Perform interpolation prediction.

In step 709, two values predicted by interpolation are compared, and the predicted value with the smaller interpolation error is selected.
In step 710, the high-frequency component ΔP1 of the processing target pixel is added to the prediction value selected in step 709 to synthesize the pixel value of the target pixel.

  In step 711, it is determined whether or not all the processing target pixels in the reverse first step have been processed. If not completed, the process returns to step 708 to process the next pixel. When completed, go to Step 712.

In step 712, when decoding in a plurality of layers, it is determined whether or not the highest layer has been reached. If the next upper layer remains, the process returns to step 703 to set the image range composed of unprocessed pixels as a target, and repeat the resolution synthesis (reverse second step and reverse first step) of the next layer. . When not repeating, it progresses to step 713.
In step 713, the synthesized decoded image is output.

  As described above, when multi-resolution synthesis is performed, by performing interpolation prediction with different directions in two steps, decoding processing can be efficiently performed according to the pixel arrangement. In addition, by repeating these steps, it is possible to synthesize the resolution up to a predetermined layer and decode it with high accuracy.

  According to each of the embodiments described above, the optimum direction is selected as the pixel interpolation direction in resolution decomposition in each region of the image, and as a result, the value of each high-frequency component becomes small and the amount of encoded data is reduced. To do. In addition, as a result of selecting an optimal direction for the interpolation direction of pixels in the resolution synthesis, the encoded data can be suitably decoded to prevent deterioration of the decoded image. Therefore, it is possible to realize an encoding apparatus and method that perform encoding with high compression efficiency, and a decoding apparatus and method that restore a decoded image with good image quality.

  In the above embodiment, the interpolation direction selected at the time of encoding may be stored for each processing target pixel and described in the encoded data. For example, a flag for identifying whether the interpolation direction (interpolated prediction value) selected by the direction selection units 105 and 108 is the horizontal direction, the vertical direction, the right diagonal direction, or the left diagonal direction is added. By doing so, the interpolation direction can be uniquely determined according to the identification flag added to the encoded data at the time of decoding. Therefore, the processing of the interpolation processing units 205 and 208 is reduced, and the processing of the direction selection units 206 and 209 is not necessary.

  The encoding device and decoding device described above can be applied to a recording device, a playback device, a mobile phone, a digital camera, and the like that handle an encoded image signal.

  Furthermore, the resolution decomposition method according to the present embodiment can also be used for image retrieval. For example, resolution decomposition according to the present embodiment is performed up to a certain level, and the image stored in the database is compared with the search target image using the last reference pixel image and the high frequency component as a search key. At this time, since the image to be processed is reduced while retaining the feature of the image, a high-speed image search can be performed using a small image, so that an efficient search is possible.

  The resolution composition method according to the present embodiment can also be used for converting a low resolution image into a high resolution. For example, a low-resolution image is used as a reference image, and high-frequency components are predicted by an appropriate method, or all are input as zero. Images are restored from these, and a high-resolution image with four times the number of pixels having double the resolution in the horizontal and vertical directions is obtained. At that time, by selecting the interpolation direction as the optimum direction in each region of the image, it is possible to obtain a high-resolution image with less noise and a clear edge.

The block diagram which shows one Example of the image coding apparatus by this invention. The block diagram which shows one Example of the image decoding apparatus by this invention. The figure explaining the method of the multi-resolution decomposition | disassembly in a present Example in detail. The figure which shows an example of the selection method of the predicted value in a direction selection part. The figure which shows the other example of the setting of a reference pixel. The figure which shows the flowchart of an image coding method. The figure which shows the flowchart of an image decoding method.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 101 ... Image input part, 102 ... Image prediction part, 103 ... Resolution decomposition memory, 104, 107 ... Interpolation processing part, 105, 108 ... Direction selection part, 106, 109 ... Pixel separation part, 110 ... Conversion / quantization part, DESCRIPTION OF SYMBOLS 111 ... Code output part, 112 ... Inverse quantization / inverse conversion part, 113 ... Resolution composition memory, 114, 117 ... Interpolation processing part, 115, 118 ... Direction selection part, 116, 119 ... Pixel composition part, 120 ... Decoded image Memory 201, stream analysis unit 202, image prediction unit 203, inverse quantization / inverse conversion unit 204, resolution synthesis memory 205, 208 interpolation processing unit 206, 209, direction selection unit, 207, 210 ... Pixel synthesis unit 211... Decoded image memory 212.

Claims (16)

An image encoding device for encoding an image, comprising:
A first processing target pixel is selected from the input image at a predetermined interval, and interpolation prediction is performed for each of the processing target pixels with respect to the first plurality of interpolation directions using other reference pixels that are not processing targets. A first interpolation processing unit to perform;
A first direction selection unit that selects an interpolation prediction value in a direction with a small interpolation error among the interpolation prediction values for a plurality of directions predicted to be interpolated by the first interpolation processing unit;
A first pixel separation unit that separates a difference between the interpolation predicted value selected by the first direction selection unit and the pixel value of the processing target pixel as a high frequency component;
A conversion / quantization unit that converts and quantizes the high-frequency component of the first processing target pixel separated by the first pixel separation unit and the pixel values of other reference pixels that are not to be processed;
An image encoding apparatus comprising: a code output unit that outputs the coefficient transformed and quantized by the transform / quantization unit as encoded data. The image encoding device according to claim 1,
For each pixel that has not been processed by the first interpolation processing unit, a second processing target pixel is further selected at a predetermined interval, and each processing target pixel is used by using another reference pixel that is not to be processed. A second interpolation processing unit that performs interpolation prediction with respect to the second plurality of interpolation directions each time;
A second direction selection unit that selects an interpolation prediction value in a direction with a small interpolation error among the interpolation prediction values for a plurality of directions predicted to be interpolated by the second interpolation processing unit;
A second pixel separation unit that separates the difference between the interpolation predicted value selected by the second direction selection unit and the pixel value of the processing target pixel as a high-frequency component;
The transform / quantization unit includes the high-frequency component of the first processing target pixel, the high-frequency component of the second processing target pixel separated by the second pixel separation unit, and other non-processing targets. An image encoding apparatus characterized by converting and quantizing a pixel value of a reference pixel. The image encoding device according to claim 2,
The first plurality of interpolation directions in the first interpolation processing unit and the second plurality of interpolation directions in the second interpolation processing unit are a horizontal direction, a vertical direction, a 45 degree right direction, An image coding apparatus including a 45 degree direction on the left. The image encoding device according to claim 2,
The first direction selection unit and the second direction selection unit compare the total value of L1 norms, which is the sum of absolute value errors of pixel values between adjacent pixels in a plurality of directions, and the value becomes smaller. An image coding apparatus characterized by selecting an interpolation prediction value in a direction. An image encoding method for encoding an image, comprising:
As the first resolution decomposition step,
A first processing target pixel is selected from the input image at a predetermined interval, and interpolation prediction is performed for each of the processing target pixels with respect to the first plurality of interpolation directions using other reference pixels that are not processing targets. Done
Select an interpolation prediction value in a direction with a small interpolation error among interpolation prediction values for a plurality of directions predicted by interpolation,
Separating the difference between the selected interpolated prediction value and the pixel value of the processing target pixel as a high frequency component,
Transforming and quantizing the high-frequency component of the first processing target pixel separated by the first resolution decomposition step and the pixel values of other reference pixels that were not processed;
An image coding method characterized by outputting transformed and quantized coefficients as coded data. The image encoding method according to claim 5, comprising:
As a second resolution decomposition step,
For each pixel that has not been processed in the first resolution decomposition step, a second processing target pixel is further selected at a predetermined interval, and each processing target pixel is used using another reference pixel that is not to be processed. Interpolation prediction is performed for each of the second plurality of interpolation directions,
Select an interpolation prediction value in a direction with a small interpolation error among interpolation prediction values for a plurality of directions predicted by interpolation,
Separating the difference between the selected interpolated prediction value and the pixel value of the processing target pixel as a high frequency component,
The high-frequency component of the first processing target pixel separated in the first resolution decomposition step, the high-frequency component of the second processing target pixel separated in the second resolution decomposition step, and the processing target are not used. An image encoding method characterized by transforming and quantizing pixel values of other reference pixels. The image encoding method according to claim 6, comprising:
The first plurality of interpolation directions in the first resolution decomposition step and the second plurality of interpolation directions in the second resolution decomposition step are a horizontal direction, a vertical direction, a right 45 degree direction with respect to the pixel array, and An image encoding method characterized by including a left 45 degree direction. The image encoding method according to claim 6, comprising:
When selecting an interpolated prediction value in the first resolution decomposition step and the second resolution decomposition step, a total value of L1 norms, which is a sum of absolute value errors of pixel values between adjacent pixels in a plurality of directions, is compared. And selecting an interpolation prediction value in a direction in which the value decreases. An image decoding device for decoding an encoded image stream,
An inverse quantization / inverse transform unit that performs inverse quantization and inverse transform on the encoding coefficient of the encoded stream to obtain a pixel value or a high-frequency component;
A second plurality of interpolation directions are selected for each processing target pixel by selecting the second processing target pixel from the pixel from which the high-frequency component is obtained at a predetermined interval and using other reference pixels from which the pixel value is obtained. A second interpolation processing unit that performs interpolation prediction on
A second direction selection unit that selects an interpolation prediction value in a direction with a small interpolation error among the interpolation prediction values for a plurality of directions predicted to be interpolated by the second interpolation processing unit;
A second pixel synthesis unit that synthesizes a pixel value by adding a high-frequency component of the processing target pixel to the interpolation prediction value selected by the second direction selection unit;
An image decoding apparatus comprising: an image output unit that outputs an image synthesized by the second pixel synthesis unit. The image decoding device according to claim 9, wherein
Another reference pixel from which a pixel value is obtained by further selecting a first processing target pixel at a predetermined interval with respect to a pixel from which a high-frequency component that has not been processed by the second interpolation processing unit is obtained. A first interpolation processing unit that performs interpolation prediction with respect to the first plurality of interpolation directions for each processing target pixel;
A first direction selection unit that selects an interpolation prediction value in a direction with a small interpolation error among the interpolation prediction values for a plurality of directions predicted to be interpolated by the first interpolation processing unit;
A first pixel synthesis unit that synthesizes a pixel value by adding a high-frequency component of the processing target pixel to the interpolation prediction value selected by the first direction selection unit;
The image output device, wherein the image output unit outputs the image synthesized by the first pixel synthesis unit together with the image synthesized by the second pixel synthesis unit. The image decoding device according to claim 10, comprising:
The second plurality of interpolation directions in the second interpolation processing unit and the first plurality of interpolation directions in the first interpolation processing unit are a horizontal direction, a vertical direction, a 45 degree direction to the right with respect to the pixel array, and An image decoding apparatus including a 45-degree left direction. The image decoding device according to claim 10, comprising:
The second direction selection unit and the first direction selection unit compare the total value of L1 norms, which is the sum of absolute value errors of pixel values between adjacent pixels in a plurality of directions, and the value becomes smaller. An image decoding apparatus characterized by selecting a direction interpolation prediction value. An image decoding method for decoding an encoded image stream, comprising:
Perform inverse quantization and inverse transform on the coding coefficient of the coded stream to obtain the pixel value or high frequency component,
As the second resolution composition step,
A second plurality of interpolation directions are selected for each processing target pixel by selecting the second processing target pixel from the pixel from which the high-frequency component is obtained at a predetermined interval and using other reference pixels from which the pixel value is obtained. Interpolation prediction for
Select an interpolation prediction value in a direction with a small interpolation error among interpolation prediction values for a plurality of directions predicted by interpolation,
Adding a high-frequency component of the processing target pixel to the selected interpolation prediction value, and synthesizing the pixel value;
An image decoding method, comprising outputting the image synthesized in the second resolution synthesis step. The image decoding method according to claim 13, comprising:
Furthermore, as the first resolution composition step,
Another reference pixel from which a pixel value is obtained by selecting a first processing target pixel at a predetermined interval for a pixel from which a high-frequency component that has not been processed in the second resolution synthesis step is obtained. Is used to perform interpolation prediction for the first plurality of interpolation directions for each processing target pixel,
Select an interpolation prediction value in a direction with a small interpolation error among interpolation prediction values for a plurality of directions predicted by interpolation,
Adding a high-frequency component of the processing target pixel to the selected interpolation prediction value, and synthesizing the pixel value;
An image decoding method, wherein the image synthesized in the first resolution synthesis step is output together with the image synthesized in the second resolution synthesis step. The image decoding method according to claim 14, comprising:
The second plurality of interpolation directions in the second resolution composition step and the first plurality of interpolation directions in the first resolution composition step are a horizontal direction, a vertical direction, a right 45-degree direction with respect to the pixel array, and An image decoding method characterized by including a 45 ° left direction. The image decoding method according to claim 14, comprising:
When selecting an interpolated prediction value in the second resolution synthesis step and the first resolution synthesis step, the total value of L1 norms, which is the sum of absolute value errors of pixel values between adjacent pixels in a plurality of directions, is compared. An image decoding method characterized by selecting an interpolation prediction value in a direction in which the value decreases.

Images