JP5259632B2 - Image processing apparatus, encoding apparatus, decoding apparatus, and program - Google Patents

Description

  The present invention relates to an image processing technique, and more particularly, to an image processing apparatus, an encoding apparatus, a decoding apparatus, and a program for correcting an image signal that has been encoded by an irreversible encoding method and has undergone encoding degradation.

  As a signal interpolation technique using the correlation between component signals constituting an image signal, a technique for improving the accuracy of a pixel interpolation method in a single-plate primary color image sensor has been reported (for example, see Non-Patent Document 1). Since the image signal interpolation technique in such an image sensor is intended to interpolate RGB signals (R: red signal, G: green signal, B: blue signal) in the RGB color space, signal deterioration due to encoding is assumed. It has not been.

  Further, as a signal interpolation technique focusing on the difference in sampling frequency of YUV signals in the YUV color space, a color difference component interpolation technique for format color images has been reported (for example, see Non-Patent Document 2). In this technique, high-precision interpolation is performed by generating an interpolated signal of a color difference signal (U signal = BY, V signal = RY) using the sampling frequency of the luminance (Y) signal. . The signal interpolation technique that pays attention to such a difference in the sampling frequency of the YUV signal is also intended for the interpolation of the YUV signal, so that signal degradation due to encoding is not assumed.

  These signal interpolation techniques are suitable for interpolation on an image signal before encoding when encoding an image signal by an irreversible encoding method (for example, MPEG-2, H.264, etc.). It is not suitable for interpolation on the image signal after conversion. For example, when a YUV signal is encoded by irreversible encoding processing, the deterioration of the luminance signal propagates to the color difference signal based on the luminance signal as the luminance signal deteriorates. In addition, these signal interpolation techniques are not processes for reducing the degradation of the luminance signal itself, and therefore do not reduce the degradation of the luminance signal.

  In addition, there are various deblocking filters (for example, deblocking filters in H.264 etc.) in order to reduce the degradation of encoding, but these deblocking filters visually degrade each of the image signal components. Are processed independently so as not to stand out, and deterioration after encoding of the original image signal cannot be reduced.

Kuno, Sugiura, “Higher accuracy of pixel interpolation in single-plate primary color image sensor”, Journal of the Institute of Image Information and Television Engineers, Vol. 61, no. 7. July 1, 2007, pp. 1006 to 1016 Sugita, Taguchi, "YUV4: 2: 0 format color difference component interpolation method", IEICE Transactions, Vol. J88-A, no. 6, June 1, 2005, pp. 751-760

  The deterioration of the orthogonal transform coefficient due to the quantization is perceived as deterioration of the pixel, such as block distortion and mosquito noise, due to inverse quantization and inverse orthogonal transform. In addition, since the degree of deterioration differs for each pixel block, the difference may be a significant disturbance at the boundary between adjacent coding blocks, and block distortion may be detected. In such a case, there is room for further improving the block distortion by using the correlation between the component signals of the moving image.

  When an image is encoded in units of small areas using an irreversible encoding method (for example, MPEG-2, H.264, etc.), for example, in MPEG-2, if the 4: 2: 0 format is used, the luminance signal The color difference signal corresponding to the pixel block 16 × 16 pixels is 8 × 8 pixels, and the sampling frequency differs between the signals. For example, in an image encoding process typified by MPEG-2, signals having different sampling frequencies are processed in a common size 8 × 8 pixel block. That is, since the luminance signal is divided into four 8 × 8 pixel blocks and the color difference signal is also encoded with the 8 × 8 pixel block, the range occupied by the 8 × 8 pixel block to be encoded is the luminance signal and the color difference signal. It will be different.

  In view of the above-described problems, an object of the present invention is to provide an image processing apparatus, an encoding apparatus, a decoding apparatus, and an image processing apparatus capable of correcting an image signal even in the case of signal components in color spaces having different sampling frequencies. To provide a program.

  An image processing apparatus according to a first aspect of the present invention is an image processing apparatus for correcting a component signal (for example, YUV signal) of a moving image, and is an image signal having a predetermined image format obtained through an irreversible encoding method. Normalizing the second component signal (eg, U signal) and / or the third component signal (eg, V signal) to correct for the first component signal (eg, Y signal) at When a difference value obtained by comparing the first component signal and the teacher signal with a teacher signal generation unit that generates a teacher signal of the component signal is equal to or less than a predetermined threshold, based on the teacher signal Correction means for correcting the first component signal. The component signal may be in any color space such as RGB, YCbCr, LUV, Lab, XYZ.

  The image processing apparatus according to the second aspect of the present invention is an image processing apparatus for correcting a component signal (for example, YUV signal) of a moving image, and is an image signal of a predetermined image format obtained through an irreversible encoding method. To correct for the second component signal (e.g., U signal) and / or the third component signal (e.g., V signal), the first component signal is normalized and the second component signal and / or second A difference value obtained by comparing the teacher signal generating means for generating the teacher signal of the component signal 3 and the second component signal and / or the third component signal and the teacher signal is equal to or less than a predetermined threshold value; In some cases, modifying the second component signal and / or the third component signal based on the teacher signal A positive means, characterized in that it comprises a.

  In the image processing device according to the first aspect or the second aspect of the present invention, when the component signal of the moving image includes a component signal having a different sampling rate, the teacher signal generation unit is configured to output a luminance signal (for example, Luminance signal down-converting means for down-converting the component signal of the Y signal to a sampling rate of a color difference signal (for example, U signal, V signal) corresponding to the luminance signal to generate a teacher signal of the color difference signal; And a color difference signal up-conversion unit that up-converts a component signal of the color difference signal in the signal to a sampling rate of the luminance signal corresponding to the color difference signal to generate a teacher signal of the luminance signal.

  In the image processing apparatus according to the first aspect or the second aspect of the present invention, the component signal of the moving image includes each component signal in a predetermined color space, an orthogonal transform coefficient of the component signal, and a combination of these signals. It is characterized by including.

  The image processing apparatus according to the third aspect of the present invention is an image processing apparatus that mutually corrects image signals composed of signal components in color spaces having different sampling frequencies, and is a predetermined image obtained through an irreversible encoding method. Obtained from the luminance signal down-converting unit and the luminance signal down-converting unit that down-converts the luminance signal in the image signal to the sampling rate of the color difference signal corresponding to the luminance signal. A first normalization processing unit that normalizes each component of the orthogonal transformation coefficient of the luminance signal to be generated with reference to the value of the orthogonal transformation coefficient of each color difference signal corresponding to the luminance signal and generates a teacher signal of the color difference signal; For the orthogonal transform coefficient of the image signal, the color difference signal in the image signal is converted to a sample of the luminance signal corresponding to the color difference signal. The chrominance signal up-converter for up-converting to the rate, and each component of the orthogonal transform coefficient of each chrominance signal obtained from the chrominance signal up-converter are based on the value of the orthogonal transform coefficient of the luminance signal corresponding to each chrominance signal. The second normalization processing unit that normalizes and generates a luminance signal as a teacher signal, and compares each of the orthogonal transform coefficients of the color difference signal and the luminance signal with each of the teacher signals. A correction unit that retains the sign of the value of the original orthogonal transform coefficient when it is equal to or less than a predetermined threshold, and corrects the orthogonal transform coefficient of the color difference signal and the luminance signal based on the orthogonal transform coefficient of the teacher signal; An inverse orthogonal transform unit configured to perform inverse orthogonal transform on the corrected orthogonal transform coefficient of the luminance signal and the color difference signal supplied from the correction unit and generate a corrected image signal; Characterized in that it obtain.

  The image processing apparatus according to the fourth aspect of the present invention is an image processing apparatus that mutually corrects image signals composed of signal components in color spaces having different sampling frequencies, and is obtained by a predetermined method obtained through an irreversible encoding method. A first orthogonal transform unit that performs orthogonal transform on the image signal of the image format to generate respective orthogonal transform coefficients of the luminance signal and the color difference signal, and the luminance signal in the image signal is converted into a color difference corresponding to the luminance signal. Luminance signal down-conversion unit that down-converts to the signal sampling rate, and orthogonal conversion is performed on the down-converted luminance signal generated by the luminance signal down-conversion unit to generate an orthogonal transformation coefficient of the down-converted luminance signal Each component of the orthogonal transform coefficient of the luminance signal obtained from the second orthogonal transform unit and the second orthogonal transform unit Normalization is performed on the basis of the value of the orthogonal transformation coefficient of each color difference signal corresponding to the degree signal, and the orthogonal transformation coefficients of the two kinds of normalized luminance signals are output from the first orthogonal transformation unit. A first normalization processing unit that generates a color difference signal corresponding to each luminance signal as a teacher signal; and a color difference signal up-conversion that upconverts the color difference signal in the image signal to a sampling rate of the luminance signal corresponding to the color difference signal. A third orthogonal transformation unit that performs orthogonal transformation on the up-converted color difference signal obtained from the color difference signal up-conversion unit and generates an orthogonal transformation coefficient of the up-converted color difference signal, and the third orthogonal transformation Each component of the orthogonal transform coefficient of each color difference signal obtained from the image is normalized with reference to the value of the orthogonal transform coefficient of the luminance signal corresponding to each color difference signal A second normalization processing unit that generates a coefficient based on the orthogonal transform coefficient of each normalized color difference signal as a teacher signal of a luminance signal that is output from the first orthogonal transform unit and that corresponds to each of the color difference signals; For each orthogonal transform coefficient of the color difference signal and the luminance signal, when compared with the respective teacher signals, and the difference value obtained by comparison is equal to or less than a predetermined threshold value, the sign of the value of the original orthogonal transform coefficient is And a correction unit that corrects the orthogonal transform coefficient of the color difference signal and the luminance signal based on the orthogonal transformation coefficient of the teacher signal, and the corrected orthogonal transform coefficient of the luminance signal and the color difference signal supplied from the correction unit And an inverse orthogonal transform unit that performs inverse orthogonal transform on the image and generates a corrected image signal.

  In the image processing device according to the fourth aspect of the present invention, the first orthogonal transform unit, the second orthogonal transform unit, and the third orthogonal transform unit are composed of the same type of orthogonal transform processing.

  Further, in the image processing device according to the fourth aspect of the present invention, the correction unit, for the color difference signal, the absolute value of the teacher signal generated by the first normalization processing unit and the color difference from the first orthogonal transform unit. When the absolute value of the difference between the absolute values of the orthogonal transform coefficients of the signal is calculated and the absolute value of the difference is equal to or less than a predetermined threshold, the value of the original orthogonal transform coefficient from the first orthogonal transform unit is positive or negative The code is retained, the orthogonal transform coefficient of the color difference signal is corrected based on the orthogonal transform coefficient of the teacher signal, and the orthogonal transform coefficient of the corrected color difference signal is generated. An absolute value of a difference between the absolute value of the teacher signal generated by the processing unit and the absolute value of the orthogonal transform coefficient of the luminance signal from the first orthogonal transform unit is calculated, and the absolute value of the difference is equal to or less than a predetermined threshold value. In some cases, the first orthogonal The sign of the original orthogonal transform coefficient value from the conversion unit is retained, the orthogonal transform coefficient of the luminance signal is corrected based on the orthogonal transform coefficient of the teacher signal, and the corrected orthogonal transform coefficient of the luminance signal is generated It is characterized by doing.

  Further, in the image processing apparatus according to the fourth aspect of the present invention, the first normalization processing unit includes absolute components of the orthogonal transform coefficients of the color difference signals output from the first orthogonal transform unit, excluding a DC component. The maximum value of the value is detected, the component coordinate of the detected maximum value is specified, and the luminance signal obtained from the second orthogonal transform unit corresponding to the component coordinate of the orthogonal transform coefficient of each identified color difference signal The component coordinate of the orthogonal transformation coefficient is identified, and the ratio of the orthogonal transformation coefficient value of the identified color difference signal to the orthogonal transformation coefficient value of the identified luminance signal is orthogonally transformed of the luminance signal obtained from the second orthogonal transformation unit Normalize by multiplying each component of the coefficient, or from the second orthogonal transform unit corresponding to the average value or the mean square value of the absolute value of the orthogonal transform coefficient of each color difference signal output from the first orthogonal transform unit Output brightness signal Wherein the normalizing by multiplying the average value or the ratio of the mean square value of the absolute value of the orthogonal transform coefficients of the orthogonal transformation coefficients of the luminance signal output from the second orthogonal transformation unit.

  Further, in the image processing device according to the fourth aspect of the present invention, the second normalization processing unit includes absolute values of components other than a direct current component among orthogonal transform coefficients of a luminance signal output from the first orthogonal transform unit. , The component coordinates of the detected maximum value are identified, and the orthogonality of each color difference signal obtained from the third orthogonal transform unit corresponding to the component coordinate of the identified orthogonal transform coefficient of the luminance signal is detected. The component coordinates of the transformation coefficient are specified, and the ratio of the orthogonal transformation coefficient value of the specified luminance signal to the orthogonal transformation coefficient value of the specified color difference signal is orthogonally transformed for each color difference signal obtained from the third orthogonal transformation unit. Normalize by multiplying each component of the coefficient, or output from the third orthogonal transform unit corresponding to the average value or the mean square value of the absolute value of the orthogonal transform coefficient of the luminance signal output from the first orthogonal transform unit Each color difference Wherein the normalizing by multiplying the average value or the ratio of the mean square value of the absolute value of the orthogonal transform coefficients of the orthogonal transform coefficients of the color difference signal outputted from said third orthogonal transformation unit.

  An encoding apparatus according to the present invention includes the image processing apparatus according to any one of the first to fourth aspects of the present invention.

  A decoding device according to the present invention includes the image processing device according to any one of the first to fourth aspects of the present invention.

The present invention also provides a computer configured as an image processing device for correcting a component signal (for example, YUV signal) of a moving image, in a first image signal in a predetermined image format obtained through an irreversible encoding method. To correct for the component signal (eg, Y signal), the second component signal (eg, U signal) and / or the third component signal (eg, V signal) are normalized to correct the first component signal. When the difference value obtained by generating the teacher signal and comparing the first component signal and the teacher signal is equal to or smaller than a predetermined threshold value, the first component signal is generated based on the teacher signal. And a step for correcting the program.
Furthermore, the present invention provides a computer configured as an image processing apparatus for correcting a component signal of a moving image to a second component signal and / or a second component signal in an image signal of a predetermined image format obtained through an irreversible encoding method. Normalizing the first component signal to generate a second component signal and / or a teacher signal of the third component signal to modify the third component signal; and the second component signal and / or When a difference value obtained by comparing the third component signal and the teacher signal is equal to or less than a predetermined threshold value, the second component signal and / or the third component signal is determined based on the teacher signal. It is also characterized as a program to execute

  In addition, the present invention provides a computer configured as an image processing apparatus that mutually corrects image signals composed of signal components in color spaces having different sampling frequencies, and (a) a predetermined image format obtained through an irreversible encoding method. Performing orthogonal transform on the image signal to generate respective orthogonal transform coefficients of the luminance signal and the color difference signal; and (b) sampling the luminance signal in the image signal for the color difference signal corresponding to the luminance signal. (C) performing orthogonal transformation on the down-converted luminance signal generated in the step (b) to generate an orthogonal transformation coefficient of the down-converted luminance signal; and (d) ) Each component of the orthogonal transformation coefficient of the luminance signal obtained through the step (c) is converted into each color corresponding to the luminance signal. Normalization is performed with reference to the value of the orthogonal transform coefficient of the signal, and the normalized orthogonal transform coefficients of the two types of luminance signals are output from the first orthogonal transform unit, and correspond to the two types of luminance signals, respectively. A step of generating a color difference signal as a teacher signal, (e) a step of upconverting the color difference signal in the image signal to a sampling rate of a luminance signal corresponding to the color difference signal, and (f) through the step (e). Performing orthogonal transformation on the obtained up-converted color difference signal to generate an orthogonal transformation coefficient of the up-converted color difference signal; and (g) an orthogonal transformation coefficient of each color difference signal obtained through the step (f). Are normalized on the basis of the value of the orthogonal transform coefficient of the luminance signal corresponding to each color difference signal, and based on the normalized orthogonal transform coefficient of each color difference signal. Generating a coefficient as a teacher signal of a luminance signal corresponding to each color difference signal output from the first orthogonal transform unit; and (h) each orthogonal transform coefficient of the color difference signal and the luminance signal for each of the orthogonal transform coefficients. When the difference value obtained by comparison with the teacher signal is equal to or less than a predetermined threshold, the sign of the original orthogonal transform coefficient value is retained, and the color difference based on the orthogonal transform coefficient of the teacher signal is retained. Correcting each orthogonal transform coefficient of the signal and the luminance signal; and (i) performing an inverse orthogonal transform on the corrected orthogonal transform coefficient of the luminance signal and the color difference signal obtained through the step (h), and correcting And a step of generating a processed image signal.

  According to the present invention, even an image signal encoded by an irreversible encoding method can be corrected as an image with little image deterioration. When the corrected image is displayed on a display device, for example, the present invention Image degradation can be further reduced as compared with the case where the method is not applied.

1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. It is a figure which shows operation | movement of the image processing apparatus of Example 1 by this invention. As an operation in the image processing apparatus according to the first embodiment of the present invention, (a) shows a state in which the U signal and the V signal of the color difference signal in the 4: 2: 0 format are corrected based on the down-converted luminance signal Y. (B) is a diagram showing how the luminance signal Y in the 4: 2: 0 format is corrected based on the U signal or V signal of the up-converted color difference signal. As an operation of the image processing apparatus according to the first embodiment of the present invention, (a) shows a state in which the U signal and the V signal of the color difference signal in the 4: 2: 2 format are corrected based on the down-converted luminance signal Y. (B) is a figure which shows a mode that the luminance signal Y in 4: 2: 2 format is corrected based on the U signal or V signal of the up-converted color difference signal. It is a figure which shows the example at the time of applying the image processing apparatus of Example 1 by this invention to the decoding apparatus (for example, H.264 decoding apparatus) for moving images. 1 is a detailed block diagram of an image processing apparatus according to a first embodiment of the present invention. It is a figure which shows the encoding apparatus (for example, H.264 decoding apparatus) for the moving images in Example 2 which concerns on this invention. It is a figure which shows the decoding apparatus (for example, H.264 decoding apparatus) for the moving images in Example 2 which concerns on this invention. It is a block diagram which shows the modification of the image processing apparatus of Example 1 by this invention. It is a figure which shows the structural example of the encoding apparatus (for example, H.264 encoding apparatus) for the moving image from the past. (A) is a figure which shows the example of a signal of 4: 2: 2 format in a frame image, (b) is a figure which shows the example of a signal of 4: 2: 0 format in a frame image.

  The image processing apparatus according to the first embodiment of the present invention will be described below.

  The image processing apparatus 1 according to the first embodiment is an apparatus that corrects a decoded image signal after decoding an image signal encoded by a conventional moving image encoding apparatus. First, in order to understand the present invention, FIG. 10 shows a configuration example of a conventional moving picture coding apparatus (for example, an H.264 coding apparatus).

[Encoding device]
In general, a moving image is encoded by motion compensation prediction, orthogonal transform, quantization, and variable length encoding. In the case of an encoding method using motion compensated prediction, a decoding device is included in an encoding device in order to use a decoded image for prediction.

  A conventional encoding device 111 includes a rearrangement unit 12, a subtraction unit 13, an orthogonal transformation unit 14, a quantization unit 15, a variable length coding unit 16, an inverse quantization unit 17, and an inverse orthogonal transform. Unit 18, changeover switch 19, intra-frame prediction unit 20, frame memory 21, motion compensation prediction unit 22, and addition unit 23. Note that the intra-frame prediction unit 20 has a function of performing so-called intra prediction, but is not the subject of the present application, so the description thereof is omitted. Hereinafter, an example in which motion compensation prediction is performed will be described.

  The rearrangement unit 12 rearranges image signals (for example, image sensor output) configured as an array of pixel values for each pixel as a frame image, and subtracts the input image signal constituting the frame image from the subtraction unit 13 and motion compensation. Send to the prediction unit 22.

  The motion compensation prediction unit 22 performs motion vector detection on the input image signal supplied from the rearrangement unit 12 using a reference image acquired from the frame memory 21, and performs motion compensation using the obtained motion vector. The prediction image obtained as a result is output to the subtraction unit 13 and the addition unit 23 via the changeover switch 19. The motion vector information is sent to the variable length encoding unit 16.

  The subtraction unit 13 generates a difference signal between the input image signal from the rearrangement unit 12 and the predicted image from the motion compensation prediction unit 22 and sends the difference signal to the orthogonal transformation unit 14.

  The orthogonal transform unit 14 performs orthogonal transform (for example, DCT) on the differential signal supplied from the subtraction unit 13 for each pixel block in the small area, and sends the result to the quantization unit 15.

  The quantization unit 15 selects a quantization table corresponding to the pixel block in the small region supplied from the orthogonal transform unit 14, performs quantization processing, and sends the quantization table to the variable length coding unit 16 and the inverse quantization unit 17. .

  The variable length encoding unit 16 performs scanning on the quantized signal supplied from the quantization unit 15 to perform variable length encoding processing to generate a bit stream, and also generates a motion vector supplied from the motion compensation prediction unit 22. Information is also subjected to variable length coding and output.

  The inverse quantization unit 17 performs an inverse quantization process on the quantized signal supplied from the quantization unit 15 and outputs the result to the inverse orthogonal transform unit 18.

  The inverse orthogonal transform unit 18 performs inverse orthogonal transform (for example, IDCT) on the orthogonal transform coefficient supplied from the inverse quantization unit 17 and outputs the result to the addition unit 23.

  The adding unit 23 adds the signal obtained by the inverse orthogonal transform obtained from the inverse orthogonal transform unit 18 and the prediction image obtained from the image processing apparatus 1 via the motion compensation prediction unit 22 to generate a decoded image, and generates a frame memory. 21.

  The changeover switch 19 is used for switching between intra prediction and inter prediction.

  Next, a general image format used in image encoding will be described. In general, in image encoding, one image frame is composed of a combination of signals of different pixel sizes because of differences in sensitivity to luminance and color differences in human perception. As such an image format, as shown in FIG. 11A, for the pixel S1 in the frame image F, the 4: 2: 2 format (in the horizontal direction, the number U of pixels of the luminance signal (Y) is 1). As shown in FIG. 11B, the number of pixels of the signal and the V signal is 1/2), and the pixel S2 in the frame image F has a 4: 2: 0 format (in the horizontal and vertical directions, the luminance signal). The number of pixels of each of the U signal and the V signal is ½) with respect to the number of pixels 1.

  In a typical block coding such as MPEG, such an image signal is coded for every certain number of pixels without distinguishing between a luminance signal and a color difference signal. For this reason, the image ranges occupied by the encoding blocks of the luminance signal and the color difference signal are different, and the range in which the encoding deterioration occurs is also different.

  The difference in the number of samplings expressed by the luminance signal and the color difference signal means that the sampling frequency of the luminance signal and the color difference signal is different. For example, in the case of 4: 2: 2 format, two color difference signals (U signal and V signal) corresponding to an image range formed by luminance signals of 16 × 16 pixels are each configured by 8 × 16 pixels. Therefore, in the case of the 4: 2: 2 format, the sampling frequency of the color difference signal is halved with respect to the luminance signal in the horizontal direction. Similarly, in the 4: 2: 0 format, the sampling frequency of the color difference signal is halved with respect to the luminance signal in the horizontal and vertical directions.

  An image processing apparatus according to an embodiment of the present invention is an apparatus that uses the difference in sampling frequency to reduce signal degradation caused by the difference in encoding degradation characteristics of the luminance signal and the color difference signal.

  FIG. 1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 1 is a YUV signal of a predetermined image format (for example, a luminance signal having a different sampling frequency and a signal output from a decoder (not shown) that decodes an image signal encoded by an irreversible encoding method, for example. A new YUV signal (modified luminance signal and corrected luminance signal) and the predetermined image format are suppressed while suppressing the encoding deterioration of the luminance signal and the color difference signal by using the difference in sampling frequency. Color difference signal).

  As described above, the pixel block (8 × 8 pixels) of one luminance signal and the pixel block (8 × 8 pixels) of one color difference signal, which are the minimum units of the encoding process, are sampled for the luminance signal and the color difference signal. The image range expressed from the difference is different. Deterioration due to quantization differs for each pixel block of 8 × 8 pixels, and the tendency of deterioration of each block is uncorrelated, so that visually noticeable deterioration may occur particularly at the boundary between pixel blocks.

  However, considering that the area of the color difference signal after encoding is twice as large as the area of the luminance signal after encoding (or horizontal and vertical), the corresponding luminance is calculated from the color difference signal. It is possible to know the correlation of signals between adjacent pixel blocks of the signal. Also, since the number of luminance signal samplings is higher than that of chrominance signals and has a higher resolution, the luminance signal including coding degradation is down-converted to the sampling rate of the chrominance signal to reduce the degradation, and this luminance is reduced. The color difference signal can be restored by using the signal as a teacher signal for the color difference signal. Therefore, in order to compare the correlation between the luminance signal and the color difference signal, it is preferable that the sampling rates of the luminance signal and the color difference signal are the same.

  Hereinafter, the image processing apparatus 1 of the present embodiment will be described in detail with reference to FIG.

[Configuration of image processing apparatus]
The image processing apparatus 1 includes a first orthogonal transform unit 2, a luminance signal down-conversion unit 3, a second orthogonal transform unit 4, a first normalization processing unit 5, a color difference signal up-conversion unit 6, and a third orthogonal A conversion unit 7, a second normalization processing unit 8, a correction unit 9, and an inverse orthogonal transform unit 10 are provided.

  The first orthogonal transform unit 2 performs orthogonal transform on each of the luminance signal and the color difference signal in an image signal (decoder output described above) of a predetermined image format obtained through an irreversible encoding method. The orthogonal transformation coefficients of the luminance signal and the color difference signal are sent to the correction unit 9, the orthogonal transformation coefficient of the color difference signal is sent to the first normalization processing unit 5, and the orthogonal transformation coefficient of the luminance signal is sent to the second normalization coefficient. To the processing unit 8. This orthogonal transform process can be an orthogonal transform process in the encoding procedure of the lossy encoding method. For example, this orthogonal transform processing is performed in MPEG-2, H.264. H.264 or other encoding scheme DCT (Discrete Cosine Transform) or integer precision DCT can be used, but it may be used in common in the image processing apparatus 1, and the same type of any predetermined encoding scheme. Can be used.

  The luminance signal down-conversion unit 3 down-converts the luminance signal in the image signal (decoder output described above) of a predetermined image format obtained through the irreversible encoding method to the sampling rate of the color difference signal corresponding to the luminance signal. And sent to the second orthogonal transform unit 4. For example, when the input image signal is a 4: 2: 2 format YUV signal, the luminance signal down-conversion unit 3 down-converts the input luminance signal by a factor of 1/2 in the horizontal direction. The converted luminance signal is sent to the second orthogonal transform unit 4. Alternatively, when the input image signal is a 4: 2: 0 format YUV signal, the luminance signal down-conversion unit 3 reduces the input luminance signal by half in the horizontal and vertical directions. The converted and down-converted luminance signal is sent to the second orthogonal transform unit 4. As for the filter processing used for down-conversion, a known down-converter filter can be used as long as the input luminance signal is down-converted to the sampling rate of the corresponding color difference signal. A filter with higher order filter coefficients that only change is desirable. A delay adjustment function may be provided in the luminance signal down-conversion unit 3.

  The second orthogonal transform unit 4 performs orthogonal transform similar to the orthogonal transform process used in the first orthogonal transform unit 2 on the down-converted luminance signal input from the luminance signal down-conversion unit 3, and the obtained down-converted luminance signal The orthogonal transformation coefficient of the converted luminance signal is sent to the first normalization processing unit 5.

  The first normalization processing unit 5 normalizes each component of the orthogonal transformation coefficient of the luminance signal obtained from the second orthogonal transformation unit 4 with reference to the value of the orthogonal transformation coefficient of each color difference signal corresponding to the luminance signal. Then, the orthogonal transform coefficients of the two types of normalized luminance signals are generated as teacher signals for the respective color difference signals respectively output from the first orthogonal transform unit 2 and corresponding to the two types of luminance signals. More specifically, the first normalization processing unit 5 calculates the absolute value of each component excluding the (0, 0) component that is a direct current component among the orthogonal transform coefficients of each color difference signal output from the first orthogonal transform unit 2. The maximum value of the value is detected, the component coordinate of the detected maximum value is specified, and the luminance signal obtained from the second orthogonal transform unit 4 corresponding to the component coordinate of the orthogonal transform coefficient of each identified color difference signal The component coordinate of the orthogonal transformation coefficient is identified, and the ratio of the orthogonal transformation coefficient value of the identified color difference signal to the orthogonal transformation coefficient value of the identified luminance signal is orthogonally transformed of the luminance signal obtained from the second orthogonal transformation unit 4 Normalize by multiplying each component of the coefficient, or from the second orthogonal transform unit 4 corresponding to the average value or the mean square value of the absolute value of the orthogonal transform coefficient of each color difference signal output from the first orthogonal transform unit 2 Absolute orthogonal transformation coefficient of output luminance signal Is normalized by multiplying the ratio of the average value or the square mean value of the orthogonal transformation coefficient of the luminance signal output from the second orthogonal transformation unit 4, and the orthogonal transformation coefficients of the two kinds of normalized luminance signals are converted into the first orthogonal transformation. The signal is output from the unit 2 to the correction unit 9 as a teacher signal of each color difference signal corresponding to each of the two types of luminance signals. A specific example of the normalization process will be described later.

  The chrominance signal up-conversion unit 6 up-converts a chrominance signal in an image signal (decoder output described above) having a predetermined image format obtained through an irreversible encoding method up to a luminance signal sampling rate corresponding to the chrominance signal. And sent to the third orthogonal transform unit 7. For example, when the input image signal is a 4: 2: 2 format YUV signal, the color difference signal up-conversion unit 6 up-converts the input color difference signal twice in the horizontal direction and up-converts it. The color difference signal is sent to the third orthogonal transform unit 7. When the input image signal is a 4: 2: 0 format YUV signal, the color difference signal up-conversion unit 6 up-converts the input color difference signal twice in the horizontal and vertical directions, The up-converted color difference signal is sent to the third orthogonal transform unit 7. As the filter processing used for up-conversion, a known up-converter filter can be used as long as the input color difference signal is up-converted to the corresponding luminance signal sampling rate. A filter with higher order filter coefficients that only change is desirable. A delay adjustment function may be provided in the color difference signal up-conversion unit 6.

  The third orthogonal transform unit 7 divides the up-converted color difference signal input from the color difference signal up-conversion unit 6 into pixel blocks of corresponding luminance signals, and sequentially converts each pixel block to the first orthogonal transform unit 2. Orthogonal transformation similar to the orthogonal transformation processing used in is performed, and the obtained orthogonal transformation coefficient of the color difference signal after up-conversion is sent to the second normalization processing unit 8. As a result, orthogonal transform blocks of the same size can be processed between the luminance signal and the color difference signal.

  The second normalization processing unit 8 normalizes each component of the orthogonal transformation coefficient of each color difference signal obtained from the third orthogonal transformation unit 7 with reference to the value of the orthogonal transformation coefficient of the luminance signal corresponding to each color difference signal. Then, a coefficient based on the orthogonal transform coefficient of each normalized color difference signal is generated as a teacher signal of a luminance signal output from the first orthogonal transform unit 2 and corresponding to the color difference signal. More specifically, the second normalization processing unit 8 determines the absolute value of each component excluding the (0, 0) component which is a direct current component among the orthogonal transform coefficients of the luminance signal output from the first orthogonal transform unit 2. Is detected, the component coordinates of the detected maximum value are specified, and the orthogonality of each color difference signal obtained from the third orthogonal transform unit 7 corresponding to the component coordinate of the identified orthogonal transformation coefficient of the luminance signal is detected. The component coordinates of the transformation coefficient are specified, and the ratio of the orthogonal transformation coefficient value of the identified luminance signal to the orthogonal transformation coefficient value of the identified color difference signal is orthogonally transformed for each color difference signal obtained from the third orthogonal transformation unit 7 Normalize by multiplying each component of the coefficient, or output from the third orthogonal transform unit 7 corresponding to the average value or the mean square value of the absolute value of the orthogonal transform coefficient of the luminance signal output from the first orthogonal transform unit 2 Of the orthogonal transform coefficient of each color difference signal Is multiplied by the orthogonal transform coefficient of each color difference signal output from the third orthogonal transform unit 7 to normalize, and a coefficient based on the normalized orthogonal transform coefficient of each color difference signal is calculated as the first coefficient. The luminance signal corresponding to the color difference signal output from the orthogonal transform unit 2 is transmitted to the correction unit 9 as a teacher signal. A specific example of the normalization process will be described later. The coefficient based on the orthogonal transformation coefficient of each chrominance signal is selected from the larger one of the orthogonal transformation coefficients of each chrominance signal, or the average or square of the absolute values of the orthogonal transformation coefficients of each chrominance signal. It means that it can be an orthogonal transform coefficient generated so as to correlate with the orthogonal transform coefficient of each color difference signal obtained by normalization, such as an average value.

The correction unit 9 compares each orthogonal transform coefficient of the color difference signal and the luminance signal with each of the teacher signals, and when the difference value obtained by the comparison is equal to or less than a predetermined threshold, the original orthogonal transform coefficient A correction process is performed in which the sign of the value is held and the correction is performed based on the orthogonal transform coefficient of the teacher signal. More specifically, for the color difference signal, the correction unit 9 determines the absolute value of the teacher signal (orthogonal transformation coefficient down-converted for the luminance signal) generated by the first normalization processing unit 5 and the first orthogonal transformation unit 2. The absolute value of the difference between the absolute values of the orthogonal transform coefficients of the color difference signal from the signal is calculated, and the absolute value of this difference is a predetermined threshold value α _U [v] [u] for U signal or threshold value α _V for V signal. [v] [u] or less, the sign of the original orthogonal transform coefficient value from the first orthogonal transform unit 2 is retained, and the orthogonality of these color difference signals is based on the orthogonal transform coefficient of the teacher signal. The transformation coefficient is corrected, and the orthogonal transformation coefficient of the corrected color difference signal is sent to the inverse orthogonal transformation unit 10, and the luminance signal is up-converted with respect to the teacher signal generated by the second normalization processing unit 8 (the color difference signal is up-converted). Orthogonal transform coefficient) and the absolute value The absolute value of the difference between the absolute values of the orthogonal transform coefficients of the luminance signal from the 1-orthogonal transform unit 2 is calculated, and the absolute value of the difference is equal to or less than a predetermined threshold value α _Y [v] [u] for the Y signal. In this case, the sign of the original orthogonal transform coefficient value from the first orthogonal transform unit 2 is retained, the orthogonal transform coefficient of the luminance signal is corrected based on the orthogonal transform coefficient of the teacher signal, and the corrected luminance The orthogonal transform coefficient of the signal is sent to the inverse orthogonal transform unit 10. The correction process based on the orthogonal transform coefficient of the teacher signal is not performed on the (0, 0) component that is the DC component of the orthogonal transform coefficient. The correction processing based on the orthogonal transformation coefficient of the teacher signal includes replacement of the teacher signal with the orthogonal transformation coefficient, amplitude adjustment to a weighted average value of the orthogonal transformation coefficient of the teacher signal, and the like.

  The inverse orthogonal transform unit 10 performs inverse orthogonal transform on the corrected orthogonal transform coefficient of the luminance signal and the color difference signal after the correction process supplied from the correction unit 9 to generate a corrected image signal (YUV signal). . By displaying the corrected image signal (YUV signal) on a display device (not shown), even an image signal encoded by an irreversible encoding method is corrected as an image with little image deterioration. I can confirm that.

  The operation of the image processing apparatus 1 according to the present embodiment will be described below with reference to FIGS.

[Operation of image processing apparatus]
FIG. 2 shows the operation of the image processing apparatus according to an embodiment of the present invention. Hereinafter, a case where a YUV signal having a predetermined image format (a luminance signal and a color difference signal component signal having different sampling frequencies) is input will be described.

  In step S1, the image processing apparatus 1 sends a YUV signal (with a different sampling frequency) of a predetermined image format output from a decoder (not shown) that decodes an image signal encoded by an irreversible encoding method. The luminance signal and the color difference signal component signal) are input.

  First, the luminance signal down-conversion unit 3 down-converts the luminance signal in the image signal of a predetermined image format obtained through the irreversible encoding method to the sampling rate of the color difference signal corresponding to this luminance signal (step S2) The chrominance signal up-conversion unit 6 up-converts the chrominance signal in the image signal of a predetermined image format obtained through the irreversible encoding method to the sampling rate of the luminance signal corresponding to the chrominance signal (step S2). S3).

  Next, the first orthogonal transform unit 2 performs orthogonal transform on each of the original luminance signal and the color difference signal in the image signal of a predetermined image format obtained through the irreversible encoding method, and the luminance signal and the color difference Each orthogonal transform coefficient of the signal is generated (step S4), and the first orthogonal transform unit 2 applies the luminance signal down-converted by the second orthogonal transform unit 4 from the luminance signal down-conversion unit 3. Orthogonal transformation similar to the orthogonal transformation processing to be used is performed, an orthogonal transformation coefficient of the luminance signal after down-conversion is generated (step S5), and the up-conversion input from the color difference signal up-conversion unit 6 by the third orthogonal transformation unit 7 The obtained color difference signal is subjected to orthogonal transformation similar to the orthogonal transformation processing used in the first orthogonal transformation unit 2, and the color difference after up-conversion Generating an orthogonal transform coefficient of No. (step S6).

  That is, by the second orthogonal transformation unit 4 and the third orthogonal transformation unit 7, for example, in the case of a 4: 2: 2 format YUV signal, the DCT coefficient of one 8 × 8 pixel block for the luminance signal, A set of four 8 × 8 pixel block DCT coefficients is obtained for the color difference signal.

  Next, each component of the orthogonal transformation coefficient of the luminance signal obtained from the second orthogonal transformation unit 4 by the first normalization processing unit 5 is based on the value of the orthogonal transformation coefficient of each color difference signal corresponding to this luminance signal. The normalization processing is performed, and the orthogonal transformation coefficients of the two types of normalized luminance signals are generated as teacher signals of the respective color difference signals respectively output from the first orthogonal transformation unit 2 and corresponding to the two types of luminance signals. At the same time (step S7), the second normalization processing unit 8 converts each component of the orthogonal transformation coefficient of each color difference signal obtained from the third orthogonal transformation unit 7 into the orthogonal transformation coefficient of the luminance signal corresponding to each color difference signal. A normalization process is performed on the basis of the value, and a coefficient based on the orthogonal transform coefficient of each normalized color difference signal is output from the first orthogonal transform unit 2 as a teacher signal of a luminance signal corresponding to the color difference signal. Generate -Up S8).

  Here, for example, two procedures will be described as a procedure for generating a teacher signal from a luminance signal for a color difference signal.

(First example)
As shown in the following equation, the absolute value is the maximum value among the values in the component coordinates (u, v) of the orthogonal transform coefficient of each color difference signal (excluding the (0, 0) component which is a direct current component). When the component coordinates are given by (a, b), the teacher signal T_U [v] [u] for the U signal and the teacher signal T_V [v] [u] for the V signal are orthogonal transform coefficients U [ b] [a] (or V [b] [a]) The luminance signal orthogonal transformation coefficient of the down-converted luminance signal by the ratio based on the orthogonal transformation coefficient Y [b] [a] of the luminance signal of the component coordinates It can be obtained by normalizing each component of Y [v] [u].
T_U [v] [u] = Y [v] [u] * U [b] [a] / Y [b] [a]
T_V [v] [u] = Y [v] [u] * V [b] [a] / Y [b] [a]
Here, u and v are 0 to 7 for an 8 × 8 pixel block. The normalization coefficient here is given by U [b] [a] / Y [b] [a] or V [b] [a] / Y [b] [a].

The teacher signal T_Y [v] [u] for the luminance signal can be obtained in the same manner. For example, as shown in the following equation, the maximum absolute value of the values in the component coordinates (u, v) of the orthogonal transformation coefficient of the luminance signal (excluding the (0, 0) component which is a direct current component) Is given by (a, b), the teacher signal T_Y [v] [u] for the luminance signal has a component coordinate corresponding to the orthogonal transformation coefficient Y [b] [a] of the luminance signal. The orthogonal transform coefficient U [v] [u] (or V [[V] [upconverted] of the up-converted color difference signal based on the ratio based on the orthogonal transform coefficient U [b] [a] (or V [b] [a]) of the color difference signal. v] [u]) can be obtained by normalizing each component.
T_Y [v] [u] = U [v] [u] * Y [b] [a] / U [b] [a], or
= V [v] [u] * Y [b] [a] / V [b] [a]
The normalization coefficient here is given by Y [b] [a] / U [b] [a] or Y [b] [a] / V [b] [a].
Since the luminance signal is composed of a plurality of orthogonal transform blocks with respect to the color difference signal, for example, in the case of 4: 2: 0 format, the up-converted color difference signal is divided into four and the luminance signal is orthogonally transformed block. Normalize each block with respect to.

  Since two teacher signals T_Y [v] [u] for a certain luminance signal are obtained, the larger one of these two teacher signals is used as the teacher signal or the average of the absolute values A teacher signal having a larger value (or mean square value) is used as a teacher signal, a teacher signal having a higher correlation of two teacher signals is selected, or a teacher selected based on a predetermined selection criterion The teacher signal can be calculated in a manner suitable for the purpose of use of the image processing apparatus 1.

(Second example)
In the first example, the example in which the maximum value of the absolute value of the orthogonal transform coefficient is detected and normalized in order to generate the teacher signal has been described, but the average value (or the mean square value) of the absolute value of the orthogonal transform coefficient is described. An example of normalization based on) is as follows.

For example, when a teacher signal for a chrominance signal is generated by the root mean square value of the orthogonal transformation coefficient of the luminance signal, the orthogonal transformation coefficient Y [v] [u] of the luminance signal and the orthogonal transformation coefficient of the chrominance signal U [v ] [u], V [v] [u], a normalization coefficient can be obtained and a teacher signal can be generated as in the following description example in C language.
<Example description in C>
Tmp [Y] = 0; // Buffer for saving AC energy of Y signal
Tmp [U] = 0; // Buffer for saving U signal AC energy
Tmp [V] = 0; // V signal AC energy storage buffer
for (v = 0; v <8; v ++) {
for (u = 0; u <8; u ++) {
tmp [U] + = (U [v] [u] * U [v] [u]);
tmp [V] + = (V [v] [u] * V [v] [u]);
tmp [Y] + = (Y [v] [u] * Y [v] [u]);
}
}
N _U = pow (tmp [U] / tmp [Y], 0.5); // Normalization factor N _U for U signal
N _V = pow (tmp [V] / tmp [Y], 0.5); // Normalization factor N _V for V signal

  Since the luminance signal is composed of a plurality of orthogonal transform blocks, the teacher signal also corresponds to a block (for example, in the case of 4: 2: 0 format, four teacher signals generated by the up-converted color difference signal are used. 4 orthogonal transform blocks).

Referring again to FIG. 2, the absolute value of the teacher signal (orthogonal transform coefficient obtained by up-converting the chrominance signal) generated by the second normalization processing unit 8 and the first orthogonal transform for the luminance signal by the correction unit 9. If the absolute value of the difference between the absolute values of the orthogonal transformation coefficients of the luminance signal from the unit 2 is calculated and the absolute value of this difference is less than or equal to a predetermined threshold value α _Y [v] [u] for the Y signal, The sign of the value of the original orthogonal transform coefficient from the 1 orthogonal transform unit 2 is held and correction processing based on the orthogonal transform coefficient of the teacher signal is performed (steps S9 and S10). The absolute value of the difference between the absolute value of the teacher signal (orthogonal transform coefficient down-converted for the luminance signal) generated by the normalization processing unit 5 and the absolute value of the orthogonal transform coefficient of the color difference signal from the first orthogonal transform unit 2 is calculated. And this difference If the absolute value is a threshold value in advance for predetermined U-signal alpha _U [v] threshold value for [u] or V signal alpha _V [v] [u] or less, the original orthogonal transform from the first orthogonal transformation unit 2 The sign of the coefficient value is retained and correction processing based on the orthogonal transform coefficient of the teacher signal is performed (steps S11 and S12). However, the correction process based on the orthogonal transform coefficient of the teacher signal is not performed on the (0, 0) component that is the DC component of the orthogonal transform coefficient. The correction processing based on the orthogonal transformation coefficient of the teacher signal includes replacement of the teacher signal with the orthogonal transformation coefficient, amplitude adjustment to a weighted average value of the orthogonal transformation coefficient of the teacher signal, and the like.

  For example, a description example in C language of the correction process for the U signal in the correction unit 9 is as follows. The same description can be applied to the Y signal and the V signal.

<Example of correction processing written in C>
for (v = 0; v <8; v ++) {
for (u = 0; u <8; u ++) {
if (! (v == 0 && u == 0)) {// v = 0 and u = 0 do not process.
m = (U [v] [u] <0? -1: 1); // m = 1 if U [v] [u] is positive, m = -1 if negative.
tmp = fabs (T_U [v] [u]); // Save the absolute value of the teacher signal to tmp.
U [v] [u] = (fabs (U [v] [u] * m-tmp)> α _U [v] [u]? U [v] [u]: m * tmp) // U [v If the difference between the absolute value of] [u] and tmp is greater than α _U [v] [u], U [v] [u] is not changed. If the difference between the absolute value of U [v] [u] and tmp is less than or equal to α _U [v] [u], the sign is retained and corrected (replaced in this example).
}
}
}

  Next, the inverse orthogonal transform unit 10 performs inverse orthogonal transform on the corrected orthogonal transform coefficient of the luminance signal and the color difference signal supplied from the modification unit 9 to generate a corrected image signal (YUV signal). (Step S13). Note that the luminance signal corrected in this way is corrected by a color difference signal having a low sampling rate, and thus may be a totally blurred signal. Therefore, instead of rewriting all the corrected luminance signals, deblocking is performed by replacing the outermost peripheral pixel (for example, one or two pixels at a predetermined pixel position) of each pixel block in which deterioration is likely to appear significantly. It can also be configured to operate as a filter.

  3 and 4 are diagrams schematically showing a series of processing operations in the image processing apparatus 1. FIG. 3A shows a state in which the U signal and V signal of the color difference signal in the 4: 2: 0 format are corrected based on the down-converted luminance signal Y, and FIG. It shows how the luminance signal Y in the 2: 0 format is corrected based on the U signal or V signal of the up-converted color difference signal. FIG. 4A shows how the U and V signals of the color difference signal in the 4: 2: 2 format are corrected based on the down-converted luminance signal Y, and FIG. It shows how the luminance signal Y in the 2: 2 format is corrected based on the U signal or V signal of the up-converted color difference signal.

  As a result, according to the image processing apparatus 1 of the present embodiment, even an image signal encoded by an irreversible encoding method can be corrected as an image with little image degradation, resulting from encoding. Thus, it is possible to further reduce image degradation that has occurred.

  Note that MPEG-2 and MPEG-4 AVC / H. Although DCT is used as orthogonal transform in many encoding systems such as H.264, according to the image processing apparatus of the present embodiment, the difference in sampling frequency between the luminance signal of the image signal and the color difference signal that are generally used Can be applied to any coding scheme. In particular, when an image deteriorated by an irreversible encoding method is input and the purpose is to recover the deterioration due to the encoding method, the same orthogonal transformation as the encoding method used for input is used. Thus, the effect of reducing deterioration will be exhibited.

  Hereinafter, a case where the image processing apparatus 1 of the present embodiment is applied to a moving picture decoding apparatus will be described.

  FIG. 5 shows an example in which the image processing apparatus 1 of the present embodiment is applied to a moving picture decoding apparatus 31a. In the following description, the same components will be described with the same reference numerals.

  The decoding device 31a according to the present embodiment includes a variable length decoding unit 32, an inverse quantization unit 33, an inverse orthogonal transform unit 34, an addition unit 35, the image processing device 1, a frame memory 36, and motion compensation prediction. Unit 37, rearrangement unit 38, and changeover switch 39 used for switching between intra prediction and inter prediction. The decoding apparatus 31a according to the present embodiment is obtained by adding the image processing apparatus 1 according to the present embodiment to a known decoding apparatus (for example, a decoding apparatus for MPEG-2, H.264). .

  The variable length decoding unit 32 receives the encoded bitstream, performs variable length decoding processing, and sends it to the inverse quantization unit 33. At the same time, the variable length decoding unit 32 decodes motion vector information and sends it to the motion compensation prediction unit 37. . In addition, the variable length decoding unit 32 sends this quantization information to the correction unit 9 in the image processing apparatus 1.

  The inverse quantization unit 33 acquires the orthogonal transform coefficient of the differential signal obtained by performing the inverse quantization process on the quantized signal supplied from the variable length decoding unit 32 and performing motion compensation, and sends the orthogonal transform coefficient to the inverse orthogonal transform unit 34. .

  The inverse orthogonal transform unit 34 performs inverse orthogonal transform (for example, IDCT) on the orthogonal transform coefficient of the difference signal supplied from the inverse quantization unit 33, and sends the obtained difference signal to the addition unit 35.

  The motion compensation prediction unit 37 generates a prediction image using the reference image obtained from the frame memory 36 and the motion vector obtained from the variable length decoding unit 32, and outputs the prediction image to the addition unit 35 via the changeover switch 39.

  The adding unit 35 adds the difference signal obtained from the inverse orthogonal transform unit 34 and the predicted image supplied from the motion compensation prediction unit 37 to restore the image signal, and the restored image signal is related to the present embodiment. The image is sent to the image processing device 1 in the decoding device 31a.

  The image processing device 1 in the decoding device 31a generates a modified image signal in which the image signal obtained from the adding unit 35 is suppressed from encoding deterioration as described above, stores the image signal in the frame memory 36, and rearranges the unit 38. To send.

  The rearrangement unit 38 rearranges the corrected image signal obtained from the image processing apparatus 1 as a display signal.

Here, FIG. 6 shows a configuration example of the image processing apparatus 1 in FIG. The configuration of the image processing apparatus 1 shown in FIG. 6 is the same as that of the image processing apparatus 1 shown in FIG. 1, and further detailed description is omitted, but the correction unit 9 uses the threshold value α for the Y signal used for the correction process. _Y [v] [u] (or U signal threshold value α _U [v] [u] or V signal threshold value α _V [v] [u]) within the quantization range obtained from the quantization unit 15 This is an example using the half value (= q [v] [u] / 2) of the quantized value q [v] [u], which is the value of the quantized value q [v] [u] or less. Any value of can be set.

  That is, in the correction unit 9 in the image processing device 1 applied to the decoding device 31a, for example, in the case of a chrominance signal, the generated teacher signal is compared with the absolute value of the original orthogonal transform coefficient of the chrominance signal, and the encoding method is used. When the difference amount falls within the dependent quantization range, the sign of the original orthogonal transform coefficient value of the color difference signal is retained, and the original orthogonal transform coefficient is corrected based on the orthogonal transform coefficient of the teacher signal. . Similarly, in the case of the luminance signal, the correction unit 9 compares the generated teacher signal with the absolute value of the original orthogonal transform coefficient of the luminance signal, and the teacher signal falls within the quantization range depending on the encoding method. In this case, the sign of the value of the original orthogonal transform coefficient of the luminance signal is retained, and the original orthogonal transform coefficient is corrected based on the orthogonal transform coefficient of the teacher signal.

  The correction process for the U signal in the correction unit 9 in this example can be shown as a description example in C language as follows. The same applies to the Y signal and the V signal.

<Example of correction processing written in C>
for (v = 0; v <8; v ++) {
for (u = 0; u <8; u ++) {
if (! (v == 0 && u == 0)) {// v = 0 and u = 0 do not process.
m = (U [v] [u] <0? -1: 1); // m = 1 if U [v] [u] is positive, m = −1 if negative.
tmp = fabs (T_U [v] [u]); // Save the absolute value of the teacher signal to tmp.
U [v] [u] = (fabs (U [v] [u] * m-tmp)> q [v] [u] / 2? U [v] [u]: m * tmp) // U [ If the difference between the absolute value of v] [u] and tmp is greater than q [v] [u] / 2, U [v] [u] is not changed. If the difference between the absolute value of U [v] [u] and tmp is equal to or less than q [v] [u] / 2, the sign is retained and corrected (for example, replaced).
}
}
}

  As described above, when the image processing apparatus 1 of the present embodiment is applied to a decoding apparatus, an image signal that is encoded by an existing encoding apparatus, for example, composed of signal components of color spaces having different sampling frequencies is decoded. Even if it exists, it can correct so that degradation of the image signal may be reduced.

  Next, a case where the image processing apparatus 1 of the first embodiment is applied to a moving image encoding apparatus will be described as a second embodiment with reference to FIG. In the following description, the same components will be described with the same reference numerals.

[Encoding device]
The encoding device 11 according to the second embodiment includes an image processing device 1 illustrated in FIG. 1, a rearrangement unit 12, a subtraction unit 13, an orthogonal transformation unit 14, a quantization unit 15, and a variable length encoding unit 16. , An inverse quantization unit 17, an inverse orthogonal transform unit 18, a changeover switch 19, an intra-frame prediction unit 20, a frame memory 21, a motion compensation prediction unit 22, and an addition unit 23. Note that the encoding apparatus 11 according to the present embodiment has the image processing apparatus 1 shown in FIG. 1 added to a known encoding apparatus (for example, an encoding apparatus for MPEG-2, H.264). Is. The intra-frame prediction unit 20 has a function of performing so-called intra prediction, but is not the subject of the present application, and therefore its description is omitted. Hereinafter, an example in which motion compensation prediction is performed will be described.

  The rearrangement unit 12 rearranges image signals (for example, image sensor output) configured as an array of pixel values for each pixel as a frame image, and subtracts the input image signal constituting the frame image from the subtraction unit 13 and motion compensation. Send to the prediction unit 22.

  The motion compensation prediction unit 22 performs motion vector detection on the input image signal supplied from the rearrangement unit 12 using a reference image acquired from the frame memory 21 via the image processing device 1, and the obtained motion vector Is used to perform motion compensation, and a predicted image obtained as a result is output to the image processing apparatus 1. The motion vector information is sent to the variable length encoding unit 16.

  The subtraction unit 13 generates a difference signal between the input image signal from the rearrangement unit 12 and the prediction image from the image processing device 1 obtained through the motion compensation prediction unit 22 and sends the difference signal to the orthogonal transformation unit 14.

  The orthogonal transform unit 14 performs orthogonal transform (for example, DCT) on the differential signal supplied from the subtraction unit 13 for each pixel block in the small area, and sends the result to the quantization unit 15.

  The quantization unit 15 selects a quantization table corresponding to the pixel block in the small region supplied from the orthogonal transform unit 14, performs quantization processing, and sends the quantization table to the variable length coding unit 16 and the inverse quantization unit 17. .

  The variable length encoding unit 16 performs scanning on the quantized signal supplied from the quantization unit 15 to perform variable length encoding processing to generate a bit stream, and also generates a motion vector supplied from the motion compensation prediction unit 22. Information is also subjected to variable length coding and output.

  The inverse quantization unit 17 performs an inverse quantization process on the quantized signal supplied from the quantization unit 15 and outputs the result to the inverse orthogonal transform unit 18.

  The inverse orthogonal transform unit 18 performs inverse orthogonal transform (for example, IDCT) on the orthogonal transform coefficient supplied from the inverse quantization unit 17 and outputs the result to the addition unit 23.

  The adding unit 23 adds the signal obtained by the inverse orthogonal transform obtained from the inverse orthogonal transform unit 18 and the prediction image obtained from the image processing apparatus 1 via the motion compensation prediction unit 22 to generate a decoded image, and generates a frame memory. 21.

  The changeover switch 19 is used for switching between intra prediction and inter prediction.

  Next, the case where the image processing apparatus 1 shown in FIG. 1 is applied to a moving picture decoding apparatus will be described with reference to FIG.

[Decoding device]
The decoding device 31b according to the second embodiment includes a variable length decoding unit 32, an inverse quantization unit 33, an inverse orthogonal transform unit 34, an addition unit 35, the image processing device 1, a frame memory 36, and motion compensation prediction. Unit 37, rearrangement unit 38, and changeover switch 39 used for switching between intra prediction and inter prediction. Note that the decoding apparatus 31b according to the present embodiment adds the image processing apparatus 1 according to the present embodiment to the known decoding apparatus (for example, a decoding apparatus for MPEG-2, H.264) in the feedback loop. It has been done.

  The variable length decoding unit 32 receives the encoded bitstream, performs variable length decoding processing, and sends it to the inverse quantization unit 33. At the same time, the variable length decoding unit 32 decodes motion vector information and sends it to the motion compensation prediction unit 37. . Further, the variable length decoding unit 32 sends the quantization information to the correction unit 9 in the image processing apparatus 1.

  The inverse quantization unit 33 acquires the orthogonal transform coefficient of the differential signal obtained by performing the inverse quantization process on the quantized signal supplied from the variable length decoding unit 32 and performing motion compensation, and sends the orthogonal transform coefficient to the inverse orthogonal transform unit 34. .

  The inverse orthogonal transform unit 34 performs inverse orthogonal transform (for example, IDCT) on the orthogonal transform coefficient of the difference signal supplied from the inverse quantization unit 33, and sends the obtained difference signal to the addition unit 35.

  The motion compensation prediction unit 37 generates a prediction image using the reference image obtained from the frame memory 36 and the motion vector obtained from the variable length decoding unit 32 and outputs the prediction image to the image processing apparatus 1.

  The image processing device 1 in the decoding device 31 b generates a corrected image signal in which the encoding deterioration is suppressed as described above from the predicted image signal obtained from the motion compensation prediction unit 37, and adds it via the changeover switch 39. Send to unit 35.

  The adder 35 adds the difference signal obtained from the inverse orthogonal transform unit 34 and the predicted image signal supplied from the image processing device 1 to restore the image signal, and rearranges the restored image signal. To send.

  The rearrangement unit 38 rearranges the image signal obtained from the addition unit 35 as a display signal.

  In this way, as shown in the encoding device 11 and the decoding device 31b of the second embodiment, the image processing device 1 shown in FIG. 1 can be applied to the MPEG encoding method. In the encoding device 11 and the decoding device 31b of the second embodiment, for example, even when an image signal composed of signal components in color spaces having different sampling frequencies is decoded, degradation of the image signal is reduced. It can be corrected.

  As described above, the image processing device 1 according to the present invention is a device for correcting a component signal of a moving image, and the first component signal (in the image signal of a predetermined image format obtained through an irreversible encoding method) For example, to correct for the Y signal), the second component signal (eg, U signal) and / or the third component signal (eg, V signal) is normalized to provide the first component signal (eg, Y signal). When the difference value obtained by comparing the teacher signal generation function for generating the teacher signal of (1) and the first component signal (for example, Y signal) and the generated teacher signal is equal to or less than a predetermined threshold value, And a correction processing function for correcting the first component signal (for example, Y signal) based on the teacher signal.

  Furthermore, the image processing apparatus 1 according to the present invention is a device for correcting a component signal of a moving image, and a second component signal (for example, U) in an image signal of a predetermined image format obtained through an irreversible encoding method. Signal) and / or a third component signal (e.g., V signal), the first component signal (e.g., Y signal) is normalized to obtain a second component signal (e.g., U signal) and / or Or a teacher signal generation function for generating a teacher signal of a third component signal (for example, V signal), and a second component signal (for example, U signal) and / or a third component signal (for example, V signal) When the difference value obtained by comparing with the generated teacher signal is equal to or less than a predetermined threshold value, the second value is determined based on the teacher signal. Component signal (e.g., U signal) and a correction processing function to correct and / or third component signals (eg, V signal).

  Furthermore, as one aspect of the present invention, the image processing apparatus 1 of each embodiment can be configured as a computer. A program for causing a computer to realize each component of the image processing apparatus 1 described above is stored in a storage unit provided inside or outside the computer. Such a storage unit can be realized by an external storage device such as an external hard disk or an internal storage device such as ROM or RAM. The control unit provided in the computer can be realized by controlling a central processing unit (CPU) or the like. In other words, the CPU can appropriately read from the storage unit a program in which the processing content for realizing the function of each component is described, and realize the function of each component on the computer. Here, the function of each component may be realized by a part of hardware.

  In addition, the program describing the processing contents can be distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM, and such a program can be distributed on a server on a network, for example. Can be distributed by transferring the program from the server to another computer via the network.

  In addition, a computer that executes such a program can temporarily store, for example, a program recorded on a portable recording medium or a program transferred from a server in its own storage unit. As another embodiment of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and each time the program is transferred from the server to the computer. In addition, the processing according to the received program may be executed sequentially.

  While the embodiments of the present invention have been described in detail with specific examples, it will be apparent to those skilled in the art that various modifications and changes can be made without departing from the scope of the claims of the present invention. In particular, the normalization process in the normalization unit only needs to perform level adjustment for correlation between signals in the correction process using the teacher signal.

  For example, if the sampling rate of the signal input to the image processing apparatus is the same, the luminance signal down-conversion unit and the color difference signal up-conversion unit can be bypassed.

  Further, as shown in FIG. 9, the image processing apparatus of the present invention can be configured without using orthogonal transform processing. Furthermore, the image processing apparatus of the present invention is not limited to the YUV signal for each of the first to third component signals, and may be configured to input orthogonal transform coefficients and omit the orthogonal transform processing. . The component signal may be in any color space such as RGB, YCbCr, LUV, Lab, XYZ. Therefore, the component signal of the moving image input to the image processing apparatus includes each component signal in a predetermined color space, an orthogonal transform coefficient of the component signal, and a combination obtained by merging these signals.

  Furthermore, the correction unit 9 may be configured as a correction unit that compares and corrects each component signal in the image signal and / or signals of these orthogonal transform coefficients, and converts the sampling rate when each component signal is input. Correction processing is performed on the processed signal, correction processing is performed on the orthogonal transform coefficient of the signal obtained by converting the sampling rate by inputting each component signal, or the signal of the orthogonal transform coefficient of each component signal is input and the sampling rate Various modifications are possible, such as performing a modification process after quantizing a signal obtained by converting the signal, and using a combination of these.

  According to the present invention, for example, even if an image signal is composed of signal components in color spaces having different sampling frequencies, the image signal can be corrected so as to reduce the deterioration of the image signal. This is useful for any application that uses an image signal that handles encoding processing by a method.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 1st orthogonal transformation part 3 Luminance signal down-conversion part 4 2nd orthogonal transformation part 5 1st normalization process part 6 Color difference signal up-conversion part 7 3rd orthogonal transformation part 8 2nd normalization process part 9 Correction Unit 10 inverse orthogonal transform unit 11 encoding device 12 rearrangement unit 13 subtraction unit 14 orthogonal transform unit 15 quantization unit 16 variable length coding unit 17 inverse quantization unit 18 inverse orthogonal transform unit 19 changeover switch 20 intra-frame prediction unit 21 Frame memory 22 Motion compensation prediction unit 23 Addition unit 31a, 31b Decoding device 32 Variable length decoding unit 33 Inverse quantization unit 34 Inverse orthogonal transformation unit 35 Addition unit 36 Frame memory 37 Motion compensation prediction unit 38 Rearrangement unit 39 Changeover switch 111 Code Device 131 Luminance signal down-conversion unit 161 Color difference signal up-conversion unit

Claims (9)

An image processing apparatus for mutually correcting image signals composed of signal components of color spaces having different sampling frequencies,
Luminance signal down-conversion for down-converting the luminance signal in the image signal to the sampling rate of the chrominance signal corresponding to the luminance signal with respect to the orthogonal transform coefficient of the image signal of a predetermined image format obtained through an irreversible encoding method And
Each component of the orthogonal transformation coefficient of the luminance signal obtained from the luminance signal down-conversion unit is normalized based on the value of the orthogonal transformation coefficient of each color difference signal corresponding to this luminance signal, and generated as a teacher signal of the color difference signal. A first normalization processing unit;
A color difference signal up-conversion unit that up-converts the color difference signal in the image signal to the sampling rate of the luminance signal corresponding to the color difference signal with respect to the orthogonal transform coefficient of the image signal;
Each component of the orthogonal transform coefficient of each color difference signal obtained from the color difference signal up-conversion unit is normalized with reference to the value of the orthogonal transform coefficient of the luminance signal corresponding to each color difference signal, and generated as a teacher signal of the luminance signal. A second normalization processing unit;
For each orthogonal transform coefficient of the color difference signal and the luminance signal, when compared with the respective teacher signals, and the difference value obtained by comparison is equal to or less than a predetermined threshold value, the sign of the value of the original orthogonal transform coefficient is Holding a correction unit for correcting the orthogonal transform coefficient of the color difference signal and the luminance signal based on the orthogonal transform coefficient of the teacher signal;
An inverse orthogonal transform unit configured to perform inverse orthogonal transform on the orthogonal transform coefficients of the luminance signal and the color difference signal supplied from the correction unit and generate a corrected image signal. Processing equipment. An image processing apparatus for mutually correcting image signals composed of signal components of color spaces having different sampling frequencies,
A first orthogonal transform unit that performs orthogonal transform on an image signal of a predetermined image format obtained through an irreversible encoding method, and generates respective orthogonal transform coefficients of a luminance signal and a color difference signal;
A luminance signal down-converting unit that down-converts the luminance signal in the image signal to a sampling rate of a color difference signal corresponding to the luminance signal;
A second orthogonal transform unit that performs orthogonal transform on the down-converted luminance signal generated by the luminance signal down-conversion unit and generates an orthogonal transform coefficient of the down-converted luminance signal;
Two kinds of luminance signals normalized by normalizing each component of the orthogonal transformation coefficient of the luminance signal obtained from the second orthogonal transformation unit with reference to the value of the orthogonal transformation coefficient of each color difference signal corresponding to the luminance signal. A first normalization processing unit that generates the orthogonal transformation coefficients of the color difference signals corresponding to the two kinds of luminance signals output from the first orthogonal transformation unit;
A color difference signal up-converting unit that up-converts the color difference signal in the image signal to a sampling rate of a luminance signal corresponding to the color difference signal;
A third orthogonal transform unit that performs orthogonal transform on the up-converted color difference signal obtained from the color difference signal up-conversion unit and generates an orthogonal transform coefficient of the color difference signal after up-conversion;
Each component of the orthogonal transformation coefficient of each color difference signal obtained from the third orthogonal transformation unit is normalized with reference to the value of the orthogonal transformation coefficient of the luminance signal corresponding to each color difference signal, and the orthogonality of each normalized color difference signal is obtained. A second normalization processing unit that generates a coefficient based on a transform coefficient as a teacher signal of a luminance signal output from the first orthogonal transform unit and corresponding to the color difference signal;
For each orthogonal transform coefficient of the color difference signal and the luminance signal, when compared with the respective teacher signals, and the difference value obtained by comparison is equal to or less than a predetermined threshold value, the sign of the value of the original orthogonal transform coefficient is Holding a correction unit for correcting the orthogonal transform coefficient of the color difference signal and the luminance signal based on the orthogonal transform coefficient of the teacher signal;
An inverse orthogonal transform unit configured to perform inverse orthogonal transform on the orthogonal transform coefficients of the luminance signal and the color difference signal supplied from the correction unit and generate a corrected image signal. Processing equipment. The image processing apparatus according to claim 2 , wherein the first orthogonal transform unit, the second orthogonal transform unit, and the third orthogonal transform unit include the same type of orthogonal transform processing. For the color difference signal, the correction unit calculates the absolute value of the difference between the absolute value of the teacher signal generated by the first normalization processing unit and the absolute value of the orthogonal transform coefficient of the color difference signal from the first orthogonal transform unit. When the absolute value of this difference is less than or equal to a predetermined threshold value, the sign of the original orthogonal transform coefficient value from the first orthogonal transform unit is retained, and the orthogonal transform coefficient of this teacher signal is stored. Based on this, the orthogonal transform coefficient of the color difference signal is corrected to generate an orthogonal transform coefficient of the corrected color difference signal, and for the luminance signal, the absolute value of the teacher signal generated by the second normalization processing unit and the first Calculating the absolute value of the difference between the absolute values of the orthogonal transform coefficients of the luminance signal from the 1 orthogonal transform unit;
When the absolute value of the difference is equal to or less than a predetermined threshold, the sign of the original orthogonal transform coefficient value from the first orthogonal transform unit is retained, and the luminance is determined based on the orthogonal transform coefficient of the teacher signal. Fixed orthogonal transform coefficients of the signal, and generating an orthogonal transform coefficients of the modified luminance signal, the image processing apparatus according to claim 2 or 3. The first normalization processing unit detects the maximum value of the absolute value of each component excluding the DC component among the orthogonal transform coefficients of the color difference signals output from the first orthogonal transform unit, and the detected maximum value The component coordinates of the orthogonal transform coefficient of the luminance signal obtained from the second orthogonal transform unit corresponding to the component coordinates of the orthogonal transform coefficient of each specified color difference signal are identified, and the identified brightness Normalizing by multiplying each component of the orthogonal transform coefficient of the luminance signal obtained from the second orthogonal transform unit by the ratio of the value of the orthogonal transform coefficient of the specified color difference signal to the value of the orthogonal transform coefficient of the signal, or The absolute value of the orthogonal transform coefficient of the luminance signal output from the second orthogonal transform unit corresponding to the average value or the mean square value of the orthogonal transform coefficient of each color difference signal output from the first orthogonal transform unit Value or root mean square Wherein the normalizing by multiplying the ratio to the orthogonal transform coefficients of the luminance signal output from the second orthogonal transformation unit, an image processing apparatus according to any one of claims 2-4. The second normalization processing unit detects the maximum value of the absolute value of each component excluding the DC component among the orthogonal transform coefficients of the luminance signal output from the first orthogonal transform unit, and the detected maximum value The component coordinates are specified, the component coordinates of the orthogonal transformation coefficient of each color difference signal obtained from the third orthogonal transformation unit corresponding to the component coordinates of the orthogonal transformation coefficient of the identified luminance signal are identified, and the identified color difference signal Normalizing by multiplying each component of the orthogonal transform coefficient of each color difference signal obtained from the third orthogonal transform unit by the ratio of the value of the orthogonal transform coefficient of the specified luminance signal to the value of the orthogonal transform coefficient of The absolute value of the orthogonal transform coefficient of each color difference signal output from the third orthogonal transform unit corresponding to the average value or the square mean value of the orthogonal transform coefficient of the luminance signal output from the first orthogonal transform unit Value or root mean square Characterized by normalizing the ratio by multiplying the orthogonal transform coefficients of the color difference signal outputted from said third orthogonal transformation unit, an image processing apparatus according to any one of claims 2-4. An encoding device comprising the image processing device according to any one of claims 1 to 6 . Characterized in that it comprises an image processing apparatus according to any one of claims 1 to 6, the decoding apparatus. In a computer configured as an image processing apparatus for mutually correcting image signals composed of signal components of color spaces having different sampling frequencies,
(A) performing orthogonal transformation on an image signal of a predetermined image format obtained through an irreversible encoding method to generate respective orthogonal transformation coefficients of a luminance signal and a color difference signal;
(B) down-converting a luminance signal in the image signal to a sampling rate of a color difference signal corresponding to the luminance signal;
(C) performing orthogonal transform on the down-converted luminance signal generated in step (b) to generate an orthogonal transform coefficient of the down-converted luminance signal;
(D) Two types of components obtained by normalizing each component of the orthogonal transformation coefficient of the luminance signal obtained through the step (c) on the basis of the value of the orthogonal transformation coefficient of each color difference signal corresponding to the luminance signal. Generating an orthogonal transformation coefficient of the luminance signal as a teacher signal of each color difference signal corresponding to each of the two types of luminance signals output from the first orthogonal transformation unit;
(E) Up-converting a color difference signal in the image signal to a sampling rate of a luminance signal corresponding to the color difference signal;
(F) performing orthogonal transform on the up-converted color difference signal obtained through the step (e) to generate an orthogonal transform coefficient of the up-converted color difference signal;
(G) Each component of the orthogonal transform coefficient of each color difference signal obtained through the step (f) is normalized based on the value of the orthogonal transform coefficient of the luminance signal corresponding to each color difference signal, and each normalized color difference Generating a coefficient based on an orthogonal transform coefficient of a signal as a teacher signal of a luminance signal output from the first orthogonal transform unit and corresponding to the color difference signal;
(H) Each orthogonal transform coefficient of the color difference signal and the luminance signal is compared with each of the teacher signals, and when the difference value obtained by comparison is equal to or less than a predetermined threshold value, the value of the original orthogonal transform coefficient Maintaining the positive / negative sign and correcting each orthogonal transform coefficient of the color difference signal and the luminance signal based on the orthogonal transform coefficient of the teacher signal;
(I) A program for performing an inverse orthogonal transform on the corrected orthogonal transform coefficient of the luminance signal and the color difference signal obtained through the step (h) and generating a corrected image signal. .

Images