JP2007006390A - Apparatus and method for image processing, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus and a method for image processing, and a program, for improving a subjective image quality in an area in which deterioration is apt to be conspicuous from a human visual characteristic. <P>SOLUTION: The image processing apparatus includes a protection level set means for setting a protection level to each of the plurality of areas in a moving image, based on the characteristic of the area concerned, and an image quality improvement means for performing image quality improvement processing to improve the image quality of the decoded image in the protection area, according to the protection level set in the protection level set means. A protection area identification circuit 22 sets the protection area and the protection level to image data to be processed, a DCT coefficient control circuit 27 eliminates a DCT coefficient to the extent not to influence the image quality, an MB type control circuit 38 and an intra-prediction mode control circuit 39 decide a prediction mode of the intra-prediction to be performed in an intra-prediction circuit 36, and a quantization scale control circuit 41 generates a quantization parameter QP to be used in a quantization circuit 28. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for improving the subjective image quality of a flat region that is prone to deterioration from human visual characteristics.

  In recent years, MPEG (Moving) which handles image data as digital data and compresses it by DCT transform such as discrete cosine transform and motion compensation using redundancy unique to image information for the purpose of efficient transmission and storage of information. A device conforming to a scheme such as Picture Experts Group) has been widely used for information distribution, reception, storage, and the like.

However, an image compression technique such as MPEG is a technique that performs irreversible compression, and is accompanied by deterioration in image quality. In recent years, H.C. A new standard such as H.264 / MPEG4AVC has appeared, and both high image quality and high compression have been achieved. However, since it is irreversible compression, the reproduced image is slightly deteriorated.
In particular, when a compressed image is reproduced, a slight brightness or color difference is perceived in a region having high brightness and color with high human resolution, and there is a disadvantage that image quality deterioration is easily recognized.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and a program for improving the subjective image quality of an area where deterioration is conspicuous from human visual characteristics in order to eliminate the disadvantages described above.

  In order to eliminate the disadvantages described above, an image processing apparatus according to a first aspect of the present invention is an image processing apparatus that compresses and encodes a moving image having regions on a plurality of images, and the image processing device includes: A protection level setting means for setting a protection level for each of the plurality of areas based on characteristics of the area; and for the protection area, the protection area of the protection area according to the protection level set by the protection level setting means Image quality improvement means for performing image quality improvement processing for improving the image quality of the decoded image.

  Preferably, the image processing apparatus has the protection level set by the protection level setting unit for the protection area in order to reduce the bit rate increased with the image quality improvement operation performed by the image quality improvement unit. And a rate reduction means for performing a rate reduction operation in response to.

  Preferably, the characteristic of the protection area is at least one of a luminance signal value, a color difference signal value, a luminance signal variance value, and a color difference signal variance value.

  Preferably, the image quality improving means includes at least one of a quantization scale control means, an intra prediction mode control means, and a prefilter.

  More preferably, the quantization scale control means generates a new quantization parameter for the protection area according to the protection level set by the protection level setting means, and the intra prediction mode control means The macroblock type is restricted and the prediction direction is determined for the protection area according to the protection level set by the protection level setting means, and the prefilter cuts high-frequency components for the protection area. .

The quantization scale control means reduces the quantization step and traces a slight difference in pixel value, thereby improving the reproduction quality of the protected area during decoding.
Further, the intra prediction mode control means minimizes distortion with respect to the flat protected area and reduces block noise, so that the image quality of the protected area is improved.
Further, the prefilter can remove block noise caused by a slight pixel difference in the protection region.

  Preferably, the rate reduction means includes at least one of orthogonal transform coefficient control means and macroblock type control means.

  More preferably, the orthogonal transform coefficient control means removes a predetermined number of orthogonal transform coefficients having a small absolute value after orthogonal transform from the protection area, and the macroblock type control means controls the protection area. When selecting a macroblock type in intra prediction, a macroblock type that generates a smaller rate is preferentially selected.

The DCT transform coefficient control means can compress the protected area without subjective image quality degradation, and reduce the rate.
The macroblock type control means can reduce the rate at which the protection area is compressed.

  An image processing method according to a second invention is an image processing method for compressing and encoding a moving image having regions on a plurality of images, and for each of the plurality of regions in the moving image, the region A first step of setting a protection level based on the characteristics of the image quality, and an image quality improvement process for improving the image quality of the decoded image of the protection region according to the protection level with respect to the protection region set in the first step And a second step of performing.

  A program of a third invention is a program executed by an image processing apparatus that compresses and encodes a moving image having regions on a plurality of images, and for each of the plurality of regions in the moving image, A first procedure for setting a protection level based on the characteristics of the region, and an image quality for improving the image quality of a decoded image in the protection region according to the protection level with respect to the protection region set in the first procedure The image processing apparatus is caused to execute a second procedure for performing the improvement process.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and a program that improve the subjective image quality of a region where deterioration is conspicuous due to human visual characteristics.

  Hereinafter, an image processing apparatus, an image processing method, and a program according to an embodiment of the present invention will be described.

The encoding device 2 of the present embodiment is the image processing device of the present invention, the protection region of the present embodiment is the protection region of the present invention, the protection level of the present embodiment is the protection level of the present invention, and the quantum of the present embodiment. Each quantization parameter is an example of the quantization parameter of the present invention.
Further, the protection area specifying circuit of the present embodiment is the protection level setting means of the present invention, the quantization scale control circuit and the intra prediction mode control circuit of the present embodiment are the image quality improvement means of the present invention, and the MB type of the present embodiment. The control circuit and the orthogonal transform coefficient control circuit are the rate reduction means of the present invention, the quantization scale control circuit of the present embodiment is the quantization scale control means of the present invention, and the intra prediction mode control circuit of the present embodiment is the present invention. The MB type control circuit of the present embodiment of the intra prediction mode control means is an example of the MB type control means of the present invention, and the DCT coefficient control circuit of the present embodiment is an example of the orthogonal transform coefficient control means of the present invention.

FIG. 1 is an overall configuration diagram of a communication system 1 according to an embodiment of the present invention.
As shown in FIG. 1, the communication system 1 includes an encoding device 2 provided on the transmission side and a decoding device 3 provided on the reception side.
In the communication system 1, in the encoding device 2 on the transmission side, frame image data (bit stream) compressed by orthogonal transform and motion compensation such as discrete cosine transform (DCT) and Karhunen-Loeve transform is generated, After the frame image data is modulated, the frame image data is transmitted via a transmission medium such as a satellite broadcast wave, a cable TV network, a telephone line network, and a cellular phone line network.
On the receiving side, after demodulating the received image signal, frame image data expanded by inverse transformation of orthogonal transformation and motion compensation at the time of modulation is generated and used.
The transmission medium may be a recording medium such as an optical disk, a magnetic disk, and a semiconductor memory.
The decoding device 3 performs decoding corresponding to the encoding of the encoding device 2.

FIG. 2 is a block diagram showing an example of the encoding device 2 shown in FIG.
As shown in FIG. 2, the encoding device 2 includes, for example, an A / D conversion circuit 21, a protection area specifying circuit 22, a prefilter 23, a screen rearrangement circuit 24, an arithmetic circuit 25, an orthogonal transformation circuit 26, and a DCT coefficient control. Circuit 27, quantization circuit 28, lossless encoding circuit 29, buffer 30, inverse quantization circuit 31, inverse orthogonal transform circuit 32, addition circuit 33, deblock filter 34, frame memory 35, intra prediction circuit 36, motion prediction / A compensation circuit 37, an MB type control circuit 38, an intra prediction mode control circuit 39, a rate control circuit 40, and a quantization scale control circuit 41 are provided.

Hereinafter, each component of the encoding circuit 2 will be described.
The A / D conversion circuit 21 converts the input original image signal composed of the analog luminance signal Y and the color difference signals Cb and Cr into a digital image signal, which is converted into a protection area specifying circuit 22 and a prefilter 23. Output.
The protection area specifying circuit 22 detects and specifies an area where noise in the digital image signal input from the A / D conversion circuit 21 is easily perceived from the statistical data of the visual characteristics, and makes the noise conspicuous. Set the protection level accordingly.

Human vision has a characteristic that it is easy to find noise and deterioration when an image is compressed in a region where the complexity of the image is small, that is, a flat region.
Furthermore, human vision has the characteristic that a slight difference in hue can be perceived at brightness and color with high visual resolution.
Based on the above points, it is determined whether or not the corresponding area is an area where deterioration is conspicuous based on the flatness, hue, and luminance elements of each area in the image data.

Hereinafter, the protection area specifying process performed by the protection area specifying circuit 22 will be described.
First, the protection area specifying circuit 22 extracts the following parameters for each block / macroblock in the image data.
・ 8 × 8 block, macro block, variance value of luminance signal in the screen ・ 8 × 8 block, macro block, variance value of color difference signal in the screen ・ 8 × 8 block, macro block, luminance signal value in the screen 8 × 8 block, macroblock, value of color difference signal in screen The variance value of the luminance signal is a value indicating the variation of the luminance signal Y in the target region.
The variance value of the color difference signal is a value indicating variations in the color difference signals Cb and Cr in the target region.
The value of the luminance signal is a value taken by the luminance signal Y of the target area.
The value of the color difference signal is a value taken by the color difference signals Cb and Cr in the target area.

Based on these parameters, the protection area specifying circuit 22 sets a protection level (weak, medium, strong, boundary, normal) for each area in the processing target image data.
FIG. 3 is a flowchart illustrating an example of the protection area specifying process.

Step ST1:
The protection area specifying circuit 22 has the value of the luminance signal Y (brightness signal value) of the MB (macroblock) and the values of the color difference signals Cb and Cr (color difference signal value) in the image data to be processed within the specific range A. It is determined whether or not there is. If there is, the process proceeds to step ST2, and if not, the process proceeds to step ST3.
Step ST2:
The protection area specifying circuit 22 determines whether or not the luminance signal dispersion value and color difference signal dispersion value of the MB in the image data to be processed are within the specific range B. If there is, the process proceeds to step ST4. Proceed to step ST6.
Step ST3:
The protection area specifying circuit 22 determines whether the luminance signal variance value and the color difference signal variance value of the 8 × 8 block in the image data to be processed are within the specific range C, and if so, the process proceeds to step ST5. If not, the process proceeds to step ST15.

Step ST4:
The protection area specifying circuit 22 sets a protection level (weak, medium, strong) based on the following criteria for the processing target area. If the protection level is weak, the process proceeds to step ST6. If the protection level is high, the process proceeds to ST7.
When the macro block variance value is small to some extent, or the macro block variance value is quite small, but the area has a pixel value that is not so noticeable (hereinafter, the luminance signal value and the color difference signal value are collectively referred to as a pixel value). , Set the protection level to weak • Set the protection level to medium when the macroblock variance value is very small and has a pixel value that tends to be noticeable in some degree of degradation. If the area has a value, the protection level is set to strong Step ST5:
The protection area specifying circuit 22 sets a protection level (weak, medium, strong) for the processing target image data based on the above-described determination criteria. If the protection level is medium or strong, the process proceeds to step ST9, and otherwise. Advances to step ST15.

Step ST10:
The protection area specifying circuit 22 proceeds to step ST15 when the area of the processing target area is small and the deterioration is not conspicuous or the processing target area is a linear area having a width of 1 macroblock, and otherwise, the protection area described above. The process proceeds to one of steps ST11, ST12, ST13, and ST14 according to the level.

The ranges A, B, and C in the above steps ST1, ST2, and ST3 are values determined in advance based on statistical data.
The determination criterion in step ST4 is based on range data determined in advance based on statistical data based on the relationship between the average value of pixel values (Y, Cb, Cr) in the region and the degree of conspicuous deterioration.

As described above, there are five types of protection levels set by the protection area specifying circuit 22 of the present embodiment: weak, medium, strong, boundary, and normal, and accordingly, each area is classified into a weak protection area and a medium protection area according to the protection level. They are called strong protection area, boundary protection area, and normal area.
Each area will be described below.

-Weak protected area An area having a pixel value whose macroblock variance value is small to some extent, or whose macroblock variance value is quite small but whose deterioration is not so noticeable.
Medium protection area An area having a pixel value that has a very small macroblock variance and is easily noticeable in deterioration.
-Strongly protected area An area with a pixel value that has a very small macroblock variance and is prone to deterioration.
-Boundary protection area An area that is in contact with the protection area, has an 8 × 8 block variance value that is very small, and deterioration is more noticeable to a certain extent.
• Normal area Area that is not protected. An area where deterioration is not conspicuous or has a pixel value / pixel dispersion value to be set as a protection area, but the area is small or elongated and the deterioration is not conspicuous.

  As described above, according to the protection area specifying circuit 22, it is determined whether or not a plurality of areas of the processing target image data are to be protected areas, and the protection level is determined for the areas that are determined to be protection areas. Can be set.

  The protection area specifying circuit 22 outputs the protection level information set as described above to the prefilter 23, the DCT coefficient control circuit 27, the MB type control circuit 38, the intra prediction mode control circuit 39, and the quantization scale control circuit 41. .

Based on the protection level information input from the protection area specifying circuit 22, the pre-filter 23 cuts high-frequency components in areas other than the normal area of the processing target image data, and outputs them to the screen rearrangement circuit 24.
Thereby, block-like noise caused by a slight pixel difference can be eliminated.
Further, the normal area of the processing target image data is output to the screen rearrangement circuit 24 without doing anything.
The screen rearrangement circuit 24 encodes the frame image signal of the processing target image data input from the pre-filter 23 in the order of encoding according to the GOP (Group of Pictures) structure including the picture types I, P, and B. The rearranged original image data (frame image data) is output to the arithmetic circuit 25 and the motion prediction / compensation circuit 37.

The arithmetic circuit 25 generates image data indicating a difference between the original image data and the predicted image data PI input from the intra prediction circuit 36 or the motion prediction / compensation circuit 37, and outputs this to the orthogonal transform circuit 26.
The orthogonal transform circuit 26 performs image data (for example, a DCT coefficient signal) by performing orthogonal transform such as discrete cosine transform and Karhunen-Labe transform on the image data, and outputs this to the DCT coefficient control circuit 27.

The DCT coefficient control circuit 27 manipulates the value of the DCT coefficient signal input from the orthogonal transform circuit 26 based on the protection level information input from the protection area specifying circuit 22.
Specifically, when the protection level is medium or strong, the DCT coefficient control circuit 27 removes the DC component and M (M is an arbitrary natural number) low frequency components if the image is an intra macroblock, for example. , Remove all DCT coefficients. If it is an inter macro block, all DCT coefficients are removed.
When the protection level is a boundary, the DCT coefficient control circuit 27, for example, if the image is an intra macroblock, all DCT coefficients except for the DC component and N (N is an arbitrary natural number) low frequency components. Remove. If it is an inter macro block, all DCT coefficients are removed.
The natural numbers M and N described above are numbers that are determined so that even if M or N low frequency components are removed, the influence on the subjective image quality of the protected area is small.
When the protection level is normal or weak, the DCT coefficient control circuit 27 does not perform coefficient operation.

As a result, the encoding apparatus 2 of the present embodiment can remove the DCT coefficients that have little influence on the subjective image quality, and as a result, can reduce the rate within a range in which the image quality can be maintained.
The DCT coefficient control circuit 27 outputs the image data (DCT coefficient signal) for which the above-described operation has been completed to the quantization circuit 28.

  The quantization circuit 28 quantizes the image data using the quantization parameter input from the quantization scale control circuit 41 to generate a transform coefficient, which is converted into the lossless encoding circuit 29 and the inverse quantization circuit 31. Output.

  The lossless encoding circuit 29 performs variable length encoding or arithmetic encoding on the transform coefficient, and stores it in the buffer 30 as encoded image data.

The encoded image data stored in the buffer 30 is transmitted after being modulated or the like.
The inverse quantization circuit 31 generates a signal obtained by inversely quantizing the transform coefficient and outputs the signal to the inverse orthogonal transform circuit 32.
The inverse orthogonal transform circuit 32 outputs image data generated by subjecting the signal input from the inverse quantization circuit 31 to inverse transform of the orthogonal transform in the orthogonal transform circuit 26 to the adder circuit 33.
The addition circuit 33 adds the image data input from the inverse orthogonal transform circuit 32 and the predicted image data PI input from the intra prediction circuit 36 or the motion prediction / compensation circuit 37 to generate reconstructed image data, This is output to the deblock filter 34.

The deblock filter 34 writes the image data from which the block distortion of the reconstructed image data input from the adder circuit 33 is removed to the frame memory 35 as reference image data REF.
In the frame memory 35, for example, the reconstructed image data of the original image data that is the target of the motion prediction / compensation processing by the motion prediction / compensation circuit 37 and the intra prediction processing in the intra prediction circuit 36 has been processed. Written in order in units of macroblocks.

  The intra prediction circuit 36 uses the block data (macroblock type) of a plurality of sizes determined by the MB type control circuit 38 as a unit, performs intra prediction encoding according to the intra prediction mode determined by the intra prediction mode control circuit 39, The predicted image data PI and the difference DIF between the original image data and the predicted image data PI are output to the arithmetic circuit 25.

  The motion prediction / compensation circuit 37 generates a motion vector MV based on the macroblock address in the original image data and the macroblock address in the reference image data, and further generates predicted image data PI based on the motion vector MV. A difference DIF between the original image data and the predicted image data PI is generated and output to the arithmetic circuit 25.

The MB type control circuit 38 preferentially selects and determines a macroblock type with a low generation rate during intra prediction in the intra prediction circuit 36, and outputs the determined macroblock type to the intra prediction circuit 36.
Specifically, the skip mode, 16 × 16 mode, etc. are preferentially selected within a range where the image quality does not fail.
Thereby, the rate of the protection area can be reduced without degrading the image quality.

Based on the protection level information set by the protection area specifying circuit 22, the intra prediction mode control circuit 39 restricts or prioritizes macroblock types when the intra prediction circuit 36 performs intra prediction encoding, and in a specific macroblock mode. The prediction direction derivation method is changed, the macroblock type used in the intra prediction circuit 36 is determined, and the determined macroblock type information is output to the intra prediction circuit 36.
Specifically, for example, the following processing is performed.
・ Intra 16 × 16 mode is prohibited.
In the intra 4 × 4 mode, the prediction direction is determined so that the distortion is minimized.
Thereby, block noise can be eliminated or inconspicuous, and distortion can be further reduced. That is, the image quality of the protection area can be improved.

The rate control circuit 40 determines a basic QP (Quantization parameter) in the quantization circuit 28 based on the original image data and the encoded image data read from the buffer 30, and a quantization scale control circuit 41. Output to.
The quantization scale control circuit 41 performs a quantization scale control operation for newly determining the QP 40 based on the basic QP input from the rate control circuit 40 and the protection level information input from the protection area specifying circuit 22. Thereafter, the QP 40 is output to the quantization circuit 28.

Specifically, for example, when the protection level is normal or weak, the basic quantization parameter is left as it is, and when the protection level is medium, strong, or boundary, the newly determined protection QP 40 is input to the quantization circuit 28. Output.
By performing the above-described quantization scale control operation, the encoding apparatus 2 of the present embodiment reduces the quantization step and traces a slight pixel difference, so that the image quality of the area to be protected can be improved.

  FIG. 4 is a diagram summarizing operations performed by the DCT coefficient control circuit 27, the MB type control circuit 38, the intra prediction mode control circuit 39, and the quantization scale control circuit 41 on each region.

  As described above, according to the encoding device 2 of the present embodiment, by performing the operation shown in FIG. 4, statistical data is obtained from the viewpoint of luminance / hue in an area that is easily deteriorated in view of human visual characteristics. Based on the classification, the area can be classified into a plurality of protection levels, and the image quality for the area can be improved and the rate can be reduced to suppress the rate increase associated with the image quality improvement.

That is, according to the encoding device 2 of the present embodiment described above, an area where deterioration is conspicuous is identified, and an image quality improvement operation such as quantization scale control, intra prediction mode control, and prefilter is performed on the area, Further, rate reduction operations such as DCT coefficient control and MB type control can be performed in order to suppress the rate increase associated with the image quality improvement operation.
As a result, it is possible to suppress deterioration in a region where deterioration is conspicuous in normal encoding, and to prevent an increase in rate due to the suppression of deterioration, that is, a reduction in compression rate.

The present invention is not limited to the embodiment described above.
That is, those skilled in the art may make various modifications, combinations, subcombinations, and alternatives regarding the components of the above-described embodiments within the technical scope of the present invention or an equivalent scope thereof.
In the encoding device 2 of the present embodiment, the luminance signal dispersion value and the color difference signal dispersion value are used as parameters for specifying flatness. In addition to this, for example, the block degree of the local image, the original image, Parameters such as the difference in pixel values between the data and the local image data and the number of non-zero coefficients after DCT can also be used.

FIG. 1 is an overall configuration diagram of a communication system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the encoding device 2 shown in FIG. FIG. 3 is a flowchart illustrating an example of the protection area specifying process. FIG. 4 is a diagram showing operations performed on each area by the DCT coefficient control circuit 27, the MB type control circuit 38, the intra prediction mode control circuit 39, and the quantization scale control circuit 41.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Communication system, 2 ... Encoding apparatus, 3 ... Decoding apparatus, 21 ... A / D conversion circuit, 22 ... Protection area specific circuit, 23 ... Pre filter, 24 ... Screen rearrangement circuit, 25 ... Arithmetic circuit, 26 ... Orthogonal transformation circuit, 27 ... DCT coefficient control circuit, 28 ... Quantization circuit, 29 ... Lossless encoding circuit, 30 ... Buffer, 31 ... Inverse quantization circuit, 32 ... Inverse orthogonal transformation circuit, 33 ... Adder circuit, 34 ... Deblocking filter, 35 ... Frame memory, 36 ... Intra prediction circuit, 37 ... Motion prediction / compensation circuit, 38 ... MB type control circuit, 39 ... Intra prediction mode control circuit, 40 ... Rate control circuit, 41 ... Quantization scale control circuit

Claims (9)

An image processing apparatus that compresses and encodes a moving image having regions on a plurality of images,
Protection level setting means for setting a protection level for each of the plurality of regions in the moving image based on characteristics of the region;
Image quality improvement means for performing image quality improvement processing for improving the image quality of the decoded image of the protection area in accordance with the protection level set by the protection level setting means for the protection area;
An image processing apparatus. Rate reduction in which a rate reduction operation is performed on the protection area in accordance with the protection level set by the protection level setting unit in order to reduce the bit rate increased with the image quality improvement operation performed by the image quality improvement unit. The image processing apparatus according to claim 1, further comprising: means. The image processing apparatus according to claim 1, wherein the characteristic of the protection area is at least one of a luminance signal value, a color difference signal value, a luminance signal variance value, and a color difference signal variance value. The image processing apparatus according to claim 1, wherein the image quality improvement unit further includes at least one of a quantization scale control unit, an intra prediction mode control unit, and a prefilter. The quantization scale control means generates a new quantization parameter for the protection area according to the protection level set by the protection level setting means,
The intra prediction mode control means performs macroblock type restriction and prediction direction determination for the protection area according to the protection level set by the protection level setting means,
The image processing apparatus according to claim 4, wherein the pre-filter cuts a high-frequency component with respect to the protection area. The image processing apparatus according to claim 2, wherein the rate reduction unit includes at least one of an orthogonal transform coefficient control unit and a macroblock type control unit. The orthogonal transform coefficient control means removes a predetermined number of orthogonal transform coefficients having a small absolute value after orthogonal transform for the protection area,
The image processing apparatus according to claim 6, wherein the macroblock type control means preferentially selects a macroblock type that generates a smaller rate when the macroblock type is selected in intra prediction for the protection area. An image processing method for compressing and encoding a moving image having regions on a plurality of images,
A first step of setting a protection level for each of the plurality of regions in the moving image based on characteristics of the region;
An image processing method comprising: a second step of performing an image quality improvement process for improving the image quality of a decoded image in the protection area in accordance with the protection level with respect to the protection area set in the first step. A program executed by an image processing apparatus that compresses and encodes a moving image having regions on a plurality of images,
A first procedure for setting a protection level for each of the plurality of regions in the moving image based on characteristics of the region;
A program for causing the image processing apparatus to execute a second procedure for performing an image quality improvement process for improving an image quality of a decoded image of the protection area according to the protection level for the protection area set in the first procedure. .

Images