JP2007110618A - Image improvement apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve subjective image quality by complementing an orthogonal transformation coefficient of a frequency component which is lost in the case of quantization, for an image signal orthogonally transformed and quantized by an existent encoder. <P>SOLUTION: A coefficient completing means 201 provided in an image quality enhancing apparatus 200 applies complementary operation to an orthogonal transformation coefficient c (m, n) using a predetermined function f<SB>m, n</SB>to generate a new orthogonal transformation coefficient c'(m, n). Said function f<SB>m, n</SB>of the complementary operation includes a non-zero orthogonal transformation coefficient that is included in code data, in an argument and complements a zero orthogonal transformation coefficient to be non-zero. Thus, an orthogonal transformation coefficient lost in processing of an encoder 100 can be complemented by the coefficient complementing means 201. Subjective image quality is thereby improved at the time of decoding with a decoder 300. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a decoding process of an image coding system that performs orthogonal transformation and quantization, and more particularly to an image improvement device that improves the subjective image quality of an image that has deteriorated due to lossy coding.

  2. Description of the Related Art Conventionally, there has been an image improving apparatus that enhances image quality so as to prevent unnatural contour correction and emphasis on a viewer by emphasizing a high frequency component constituting an image ( (See Patent Document 1). Further, in EDTV (Enhanced Definition TeleVision) -II (wide clear vision), there is a technique that improves image quality by using a reinforcement signal added to an image signal on the transmission side (see Patent Document 2). Specifically, the image quality is improved by detecting the state of the received image signal and controlling the information amount of the reinforcement signal according to the detection result.

JP-A-5-75900 JP-A-5-3004766

  FIG. 5 is a block diagram showing a configuration of a conventional image processing system. This image processing system includes an encoder 100 and a decoder 300, which are connected by a transmission path. When an input image is input, the encoder 100 divides the image into predetermined blocks, performs orthogonal transform for each block, and performs quantization. Then, the orthogonal transform coefficient that has been subjected to the orthogonal transform and quantization is transmitted to the decoder 300 via the transmission path. When receiving the orthogonal transform coefficient, the decoder 300 performs inverse quantization, performs inverse orthogonal transform, and reconstructs a block. Then, the decoder 300 outputs a decoded image. The image processing system shown in FIG. 5 is a system applied to still images. This image processing system can be applied to a moving image by further providing a motion compensation function. In FIG. 5, the functions of transmission path encoding and transmission path decoding are omitted. The same applies to the following description.

  When such an image processing system is used to realize encoding for image compression by orthogonal transformation and quantization of JPEG, MPEG, MPEG2, MPEG4, etc. (hereinafter referred to as JPEG), the compression rate (bit rate) or Depending on the type of input image, particularly high frequency components of the frequency components constituting the image may be completely lost during the quantization process. In this case, when the decoder 300 performs the decoding process using the orthogonal transform coefficient from which the high frequency components are lost, unnatural distortion and noise are generated in the decoded image.

  In order to solve this problem, even if an attempt is made to improve the image quality by emphasizing the high frequency component by the technique of Patent Document 1 described above, the high frequency component is completely lost in the input code in the first place. Cannot be improved.

  Further, even if an attempt is made to improve the image quality using the reinforcement signal added to the image signal on the transmission side by the technique of Patent Document 2 described above, a relatively large number of bits need to be allocated. Efficiency is lost. This is a signal that is added to the upper and lower black areas generated when the 16: 9 aspect ratio video is contained within the 4: 3 aspect ratio video, for the purpose of high definition. This is because it is a necessary signal.

  Therefore, the present invention has been made in order to solve such a problem, and an object of the present invention is to reduce frequency components lost during quantization of an image signal orthogonally transformed and quantized by an existing encoder. It is an object of the present invention to provide an image quality improving apparatus capable of improving subjective image quality by complementing the above.

  Here, subjective image quality is evaluated experimentally or statistically, unlike objective image quality that is evaluated by a physically clear definition formula such as an S / N ratio without depending on human senses. Is. Specifically, subjective image quality is evaluated by subjective image quality evaluation methods such as the double stimulus continuous quality scale method, ratio scale method, and multidimensional scale construction method recommended in ITU-R Recommendation 500-7 and the like. . Therefore, the case where the subjective image quality is improved includes, for example, a case where blur, mosquito noise, block distortion or the like in the vicinity of the edge in the image becomes inconspicuous and the image looks better.

  In order to solve the above-described problems and achieve the object, an image quality improvement apparatus according to the present invention is an apparatus for improving the image quality of an image encoded by orthogonal transform and quantization, wherein the code of the encoded image is encoded. Based on a function defined in advance using the orthogonal transform coefficient as an argument for the orthogonal transform coefficient included in the code data, zero orthogonal transform coefficient is obtained using the argument of the nonzero orthogonal transform coefficient. Coefficient complementing means for performing a complementary operation so as to be non-zero and newly generating an orthogonal transform coefficient lost by the quantization is provided. Thereby, since a decoded image can be obtained using the complemented orthogonal transform coefficient, subjective image quality can be improved.

  The image quality improvement apparatus according to the present invention is characterized in that the orthogonal transform coefficient as an argument of the function is a part of all the orthogonal transform coefficients included in the code data. Further, the predetermined function performs an operation by multiplying an orthogonal transformation coefficient by a weighting coefficient. In addition, when the orthogonal transformation coefficient included in the code data is non-zero, the predetermined function is used as the new orthogonal transformation coefficient as it is, and the orthogonal transformation coefficient included in the code data is zero. In addition, the orthogonal transform coefficient is multiplied by a weight coefficient to perform the calculation. Thereby, since the process of the coefficient complementing means can be simplified, the processing load can be reduced as the whole image quality improvement apparatus.

  The image quality improvement apparatus according to the present invention further includes feature extraction means for inputting the code data and extracting an image feature based on information included in the code data, and based on the image feature extracted by the feature extraction means. And weight coefficient determining means for determining a weight coefficient used in the coefficient complementing means. Thereby, the function can be adaptively changed according to the image feature. In addition, since the encoder that transmits the code data does not need to newly generate information for determining the weighting factor, a processing load does not occur. Therefore, it is possible to suppress a decrease in encoding efficiency and to improve the image quality efficiently according to the image.

  The image quality improvement apparatus according to the present invention further receives an identifier added to the code data, which is information different from the code data, and determines a weighting coefficient used by the coefficient complementing means based on the identifier. There is provided a weight coefficient determining means for performing the above. Thereby, the function can be adaptively changed according to the intention on the encoder side. In addition, since the encoder that transmits the code data does not need to newly generate information for determining the weighting factor, a processing load does not occur. Therefore, it is possible to suppress a decrease in encoding efficiency and improve image quality according to the intention on the encoder side.

  According to the present invention, since code data that has been orthogonally transformed and quantized by an existing encoder is supplemented with frequency components lost during quantization, the subjective image quality can be improved. Is possible. For example, blur, mosquito noise, block distortion, etc. near the edges in the image become inconspicuous and the image looks better.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The embodiment of the present invention is an image quality improvement apparatus according to the first to third embodiments. The image quality improvement apparatus according to the first embodiment performs a complementary operation using a predetermined function on an orthogonal transform coefficient that is an input coefficient, and obtains an orthogonal transform coefficient after complement that is an output coefficient. The image quality improvement apparatus according to the second embodiment determines a weighting coefficient based on information included in the code data, weights the orthogonal transform coefficient with the weighting coefficient, performs a complementary operation using a predetermined function, An orthogonal transform coefficient is obtained. Further, the image improvement apparatus according to the third embodiment determines a weighting factor based on information added to the code data, weights the orthogonal transformation coefficient with the weighting factor, and performs a complementary operation with a predetermined function, The orthogonal transform coefficient after complement is obtained. In this case, the orthogonal transform coefficient of the frequency component lost at the time of quantization is estimated based on the orthogonal transform coefficient that has not been lost, and complemented as an orthogonal transform coefficient that tends to appear empirically. By performing a decoding process using orthogonal transform coefficients, subjective image quality can be improved.

  First, Example 1 will be described. FIG. 1 is a block diagram showing Example 1 of an image quality improvement apparatus according to an embodiment of the present invention. The image processing system including the image quality improvement apparatus includes an encoder 100, an image quality improvement apparatus 200, and a decoder 300. The encoder 100 and the image quality improvement apparatus 200 are connected by a transmission path. The image quality improvement apparatus 200 is inserted between the encoder 100 and the decoder 300.

  The encoder 100 is an existing standard encoder similar to that shown in FIG. 5, and divides an image into one or more blocks and performs orthogonal transform for each block to obtain orthogonal transform coefficients. calculate. Then, quantization is performed, and code data including the quantized orthogonal transform coefficient is transmitted to the image quality improvement apparatus 200 via the transmission path.

  The image quality improving apparatus 200 includes coefficient complementing means 201. When the image quality improvement apparatus 200 receives the code data transmitted from the encoder 100 via the transmission path, the coefficient complementing means 201 inputs the code data and applies the orthogonal transform coefficients of each block included in the code data. Then, a complementary operation is performed using a predetermined function, and the orthogonal transformation coefficient after the complement is output. That is, the orthogonal transformation coefficient (input coefficient) of each block constituting the input code is mapped by a predetermined function, and the orthogonal transformation coefficient (output coefficient) is output. An orthogonal transform coefficient complementing method (mapping method) will be described later.

  Similarly to the one shown in FIG. 5, the decoder 300 receives the orthogonal transform coefficient supplemented by the image quality improvement apparatus 200, performs a decoding process using the complemented orthogonal transform coefficient, and converts the decoded image into a decoded image. obtain.

  Next, the processing of the coefficient complementing unit 201 provided in the image quality improving apparatus 200, that is, the orthogonal transform coefficient complementing method will be described in detail. In the following description, attention is paid to one block constituting the image, and the orthogonal transform coefficient in the block is c, and the complemented orthogonal transform coefficient is c ′. The orthogonal transform coefficient c is a coefficient calculated at the time of quantization by the encoder 100, and is included in the code data transmitted by the encoder 100. The complemented orthogonal transform coefficient c ′ is a coefficient output by the coefficient complementing means 201 of the image quality improving apparatus 200. Further, these orthogonal transform coefficients c and c ′ are calculated for each block constituting the image, and M in the horizontal (horizontal frequency) direction and N in the vertical (vertical frequency) direction in one image. , The m-th component in the horizontal direction (m = 0, 1,..., M−1) and the n-th component in the vertical direction (n = 0, 1,. m, n), c ′ (m, n).

この関数ｆ_ｍ，ｎの引数は、係数補完手段２０１に入力される符号データに含まれる全ての直交変換係数ｃ（ｍ，ｎ）である。すなわち、係数補完手段２０１は、全ての直交変換係数ｃ（ｍ，ｎ）を用いて関数ｆ_ｍ，ｎによる補完演算を施し、補完後の直交変換係数ｃ’（ｍ，ｎ）を算出する。特定の種類の画像に特化して補完演算を施すことにより、主観的な画質の向上を一層図ることができる。 As described above, the coefficient complementing unit 201 inputs code data, complements the orthogonal transform coefficient c (m, n) of each block included in the code data, and completes the orthogonal transform coefficient c ′ (m, n) after the complement. ) Is output. Specifically, the coefficient complementing unit 201 uses the orthogonal transform coefficient c (m, n) based on one or more of the M × N orthogonal transform coefficients c (m, n) that are input coefficients. On the other hand, a complementation operation is performed by the function fm _{, n} shown in the equation (1) to calculate the orthogonal transform coefficient c ′ (m, n) after the complementation.

The arguments of this function fm _{, n} are all the orthogonal transform coefficients c (m, n) included in the code data input to the coefficient complementing means 201. That is, the coefficient complementing means 201 performs a complementation operation with the function fm _{, n} using all the orthogonal transform coefficients c (m, n), and calculates a complemented orthogonal transform coefficient c ′ (m, n). By subjecting a specific type of image to a complementary operation, subjective image quality can be further improved.

The coefficient complementing unit 201 uses a part of all the orthogonal transform coefficients c (m, n) as an argument, performs a complementation operation with the function fm _{, n} using the part of the coefficients, and performs orthogonality. The conversion coefficient c ′ (m, n) may be calculated. For example, interpolation is performed using the low-frequency component (low-frequency component) orthogonal transform coefficient of all the orthogonal transform coefficients c (m, n) as an argument. FIG. 2 is a diagram illustrating an example of a complementing method when an orthogonal transform coefficient of a low frequency component is used as an argument. The left diagram in FIG. 2 is a diagram in which all orthogonal transform coefficients c (m, n) are graphically associated with images. In the figure, a black square indicates a component whose orthogonal transform coefficient is not zero (non-zero), and a white square indicates a component whose orthogonal transform coefficient is zero (zero). A component surrounded by a dotted line indicates an orthogonal transform coefficient group used for complementation. As described above, the coefficient complementing unit 201 uses the low-frequency component orthogonal transformation coefficient group surrounded by the dotted line as an argument, performs a complementation operation with the function fm _{, n} using the coefficient, and obtains the orthogonal transformation coefficient c ′ (m , N).

  In this case, the low-frequency component area (dotted line area) that is the target of the argument is set in advance. For example, by performing a zigzag scan from the lowest DC component to the higher frequency component side with respect to the orthogonal transformation coefficient c (m, n) represented in two dimensions, the two-dimensional orthogonal transformation coefficient is converted into a one-dimensional Rearrange to orthogonal transform coefficients. In this one-dimensional orthogonal transform coefficient, up to L orthogonal transform coefficients from the beginning are set as low-frequency component regions. For example, when discrete cosine transform (DCT) is used as orthogonal transform, signal power concentrates on low-frequency components of the orthogonal transform coefficients. Therefore, the coefficient complementing unit 201 can perform complementation using, as an argument, an orthogonal transform coefficient of a low frequency component where signal power is concentrated in the case of such orthogonal transform.

  As described above, the coefficient complementing unit 201 performs a complementation operation using a part of the orthogonal transform coefficients c (m, n) as an argument, so that various images (for example, landscapes) can be obtained without specializing in a specific type of image. , Portrait, sports, CG, CM) can improve the subjective image quality. Further, the processing load can be reduced, and efficient complementary processing can be realized.

ここで、ｋ（ｍ，ｎ；ｉ，ｊ）は重み係数、Ｑは量子化関数を示す。 In addition, the coefficient complementing unit 201 performs a complementation operation using a linear transformation and a quantization function represented by the formula (2) instead of the function fm _{, n} represented by the formula (1), and performs the orthogonal transform coefficient c after the complement. '(M, n) is calculated. The function fm _{, n in the} equation (2) performs linear transformation and quantization operations on the orthogonal transformation coefficient c (m, n).

Here, k (m, n; i, j) is a weighting coefficient, and Q is a quantization function.

  The quantization function Q performs a quantization operation compatible with that used in the standard encoder 100, and is quantized in finer steps than the quantization step in the standard encoder 100. It is preferable to perform the above calculation.

The weight coefficient k is a predetermined constant for improving the subjective image quality so that the decoded image becomes a natural image. The weight coefficient k is a coefficient that is arbitrarily determined in advance. For example, a weight coefficient k suitable for the image may be determined in advance by subjective evaluation of the image. Specifically, the weighting coefficient k is determined by the following subjective evaluation experiment procedure.
(1) The weight coefficient k is determined by a random number, for example.
(2) Decoded images are obtained for various images using the image quality improvement apparatus 200.
(3) The subject is presented with the decoded image group obtained in (2) and subjectively evaluated whether or not the image looks the most beautiful.
By repeating such a procedure, the weight coefficient k when a decoded image that looks most beautiful is obtained by subject's subjective evaluation is adopted, and the weight coefficient k is used as k in the equation (2).

On the other hand, the weight coefficient k may be determined in advance by machine learning. Specifically, in the machine learning mode, the encoder 100 takes out the orthogonal transform coefficient before quantization and the orthogonal transform coefficient after quantization for a plurality of images, and sets the learning data as a pair to the learning data. Get for each image.
I (i) = (orthogonal transform coefficient before quantization, orthogonal transform coefficient after quantization)
Here, i represents an image number, and I (i) represents learning data. In addition, the image quality improvement apparatus 200 obtains a complemented orthogonal transform coefficient for each image in the machine learning mode. A learning device (not shown) receives learning data from the encoder 100 and inputs a complemented orthogonal transform coefficient from the image quality improvement device 200. Further, for each image (i), an error (square error, etc.) e (i) between the orthogonal transform coefficient before quantization and the orthogonal transform coefficient after complement is calculated, and E = Σe (i) is calculated. Then, the optimum value of the weighting factor k is determined while changing the weighting factor k using the steepest descent method or the like so that E becomes small. The weighting factor k determined as the optimum value is adopted, and the weighting factor k is used as k in the equation (2).

  Further, a weighting factor k suitable for the position or the like may be determined in advance according to the position of the divided block with respect to the image, the size of the block, or the like. Specifically, by the aforementioned subjective evaluation experiment or machine learning, for example, when a predetermined position counted in the order of raster scan from the upper left of the image is used as a reference, 4 × 4, 4 × 8, 8 × 4 in the image , 8 × 8 block size, etc., and the optimum weighting coefficient k is determined. For example, in the case of an image in which the upper side is sky and the lower side is land like a landscape image, there is an image in which blocks having different spatial frequency properties exist above and below the image. In this case, subjective image quality can be improved by using a weighting factor k suitable for the position of the block with respect to the image. Also, for example, H. An image by an encoding method standardized by H.264 or the like may have various block divisions such as 4 × 4, 4 × 8, 8 × 4, and 8 × 8. In this case, the subjective image quality can be improved by using the weighting coefficient k suitable for the block size.

The coefficient complementing means 201 performs the complement operation using the functions fm _{, n} shown in the expressions (1) and (2), but the zero component is added to the orthogonal transform coefficient used as an argument. May be included. In this case, a complementary operation is performed using zero orthogonal transform coefficients. Further, it is not always necessary to complement all the orthogonal transform coefficients by the complement calculation, and a part of them may be complemented. For example, the coefficient complementing unit 201 may output an orthogonal transform coefficient c ′ (m, n) in which all orthogonal transform coefficients that are sufficiently high frequency components are set to zero. Here, it is assumed that a sufficiently high frequency component region is set in advance. For example, as described above, a two-dimensional orthogonal transform coefficient is zigzag scanned to be rearranged into a one-dimensional orthogonal transform coefficient. In this one-dimensional orthogonal transform coefficient, the last M orthogonal transform coefficients are sufficient. Is set as a high-frequency component region. Further, the coefficient complementing means 201 may perform a complementing operation according to the equation (2) using a part of the orthogonal transform coefficients c (m, n) as an argument. In this case, it is possible to reduce the number of images necessary for the subjective evaluation experiment for determining the weighting coefficient k and learning data necessary for machine learning.

図２を参照して、係数補完手段２０１は、（３）式により、図２の左図に示した黒塗りの四角成分の非零の係数を、図２の右図に示すようにそのまま値を変化させないで、補完後の直交変換係数として出力する。また、左図に示した白塗りの四角成分（零の係数）に対して、左図の点線で囲まれた低域成分の直交変換係数群を引数とし、当該係数を用いて（３）式の関数ｆ_ｍ，ｎによる補完演算を施し、右図に示す網掛けの四角成分のように補完する。この場合、全ての直交変換係数を引数として補完演算を施すようにしてもよい。 In addition, the coefficient complementing unit 201 performs a complementation operation using the function shown in the equation (3) instead of the function fm _{, n} shown in the equations (1) and (2), and the orthogonal transformation coefficient c ′ after the complementation is performed. Calculate (m, n). The function fm _{, n} of the expression (3) uses the output coefficient c ′ (m, n) as the input coefficient c (m, n) with respect to the component whose input coefficient c (m, n) is non-zero. The output is performed, and the component having the input coefficient c (m, n) of zero is subjected to a complementary operation using the function fm _{, n} .

Referring to FIG. 2, the coefficient complementing means 201 uses the equation (3) to directly calculate the non-zero coefficient of the black square component shown in the left diagram of FIG. 2 as shown in the right diagram of FIG. Is output as a complemented orthogonal transform coefficient without changing. In addition, for the white square component (zero coefficient) shown in the left figure, the low-frequency component orthogonal transformation coefficient group surrounded by the dotted line in the left figure is used as an argument, and using the coefficient, the expression (3) function f _m, performs complementary operations by _n, complements as square components shaded as shown in the right figure. In this case, a complementary operation may be performed using all orthogonal transform coefficients as arguments.

As described above, according to the image quality improving apparatus 200 of the first embodiment, the coefficient complementing unit 201 performs a complementing operation on the orthogonal transform coefficient c (m, n) using the predetermined function fm _{, n} . . With this complementary calculation function fm _{, n} , the existing orthogonal transform coefficient (non-zero orthogonal transform coefficient) is used as an argument, and the non-existing orthogonal transform coefficient (zero orthogonal transform coefficient) is complemented with the non-zero orthogonal transform coefficient. be able to. As a result, the orthogonal transform coefficient lost (becomes zero) in the processing of the encoder 100 can be supplemented in a pseudo manner by the coefficient complementing means 201 of the image quality improvement apparatus 200. Since this interpolation is processed based on a function that can obtain a result of an image evaluated statistically or experimentally, the decoder 300 is a decoded image that is less likely to be unnatural as it looks. And subjective image quality can be improved.

Next, Example 2 will be described. FIG. 3 is a block diagram showing Example 2 of the image quality improvement apparatus according to the embodiment of the present invention. Comparing the second embodiment and the first embodiment, both the embodiments use a predetermined function fm, with the orthogonal transform coefficient c (m, n) included in the code data transmitted by the encoder 100 as an argument _{. n} is the same in that the orthogonal transform coefficient c (m, n) is complemented using _n , but in the first embodiment _, the weight coefficient of the function fm _{, n} is a predetermined value. The second embodiment is different in that the weight coefficient is determined based on information included in the code data transmitted from the encoder 100. That is, in the second embodiment, the image quality improvement apparatus 200 determines a mapping based on information included in the code data transmitted by the encoder 100.

  Referring to FIG. 3, the image processing system including the image quality improvement apparatus includes an encoder 100, an image quality improvement apparatus 200, and a decoder 300. The encoder 100 and the image quality improvement apparatus 200 are connected by a transmission path. The image quality improvement apparatus 200 is inserted between the encoder 100 and the decoder 300.

  The encoder 100 and the decoder 300 have functions equivalent to those shown in FIG. That is, the encoder 100 performs orthogonal transform and quantization, and transmits code data including the quantized orthogonal transform coefficient to the image quality improvement apparatus 200 via the transmission path. In addition, the decoder 300 receives the orthogonal transform coefficient c ′ (m, n) supplemented by the image quality improving apparatus 200, performs a decoding process using the complemented orthogonal transform coefficient, and obtains a decoded image. .

  The image quality improving apparatus 200 includes a coefficient complementing unit 201, a feature extracting unit 202, and a weighting factor determining unit 203. When the image quality improvement apparatus 200 receives code data transmitted from the encoder 100 via a transmission path, the feature extraction unit 202 extracts image features from information included in the code data. For example, the absolute value is extracted as an image feature from the motion vector information included in the code data. In addition, feature amounts are extracted as image features from luminance value information and color information included in the code data. Furthermore, the average luminance of the block is extracted as an image feature from the DC component of the orthogonal transform coefficient included in the code data.

  The weighting factor determination unit 203 inputs the image feature extracted by the feature extraction unit 202, and determines the weighting factor k group of the above-described equation (2) according to the image feature. For example, when an absolute value of a motion vector is input as an image feature, a weighting factor k corresponding to the magnitude of the absolute value of the motion vector is prepared in advance as a table, and the absolute value of the motion vector is used using the table. A weighting factor k corresponding to the value is determined.

Specifically, as in the first embodiment, a table of weighting factors k (for example, a set of k0 and k1) is determined by the following subjective evaluation experiment procedure.
(1) The weighting factors k0 and k1 are determined by random numbers, for example.
(2) Decoded images are obtained for various images using the image quality improvement apparatus 200. In this case, the absolute value of the motion vector included in the code data is compared with a predetermined threshold value, k1 is assigned to a block whose absolute value exceeds the threshold value, and k2 is assigned to other blocks. To obtain a decoded image.
(3) The subject is presented with the decoded image group obtained in (2) and subjectively evaluated whether or not the image looks the most beautiful.
By repeating such a procedure, the weighting coefficients k0 and k1 obtained when the decoded image that looks most beautiful is obtained by subject's subjective evaluation is adopted and used in the table of the weighting coefficient k.

  On the other hand, as in the first embodiment, the weight coefficient k may be determined in advance by machine learning. That is, the learning apparatus inputs learning data I (i) = (orthogonal transform coefficient before quantization, orthogonal transform coefficient after quantization) from the encoder 100, and the complemented orthogonal transform coefficient from the image quality improvement apparatus 200. Enter. Then, for each image (i), an error (such as a square error) e (i) between the orthogonal transform coefficient before quantization and the orthogonal transform coefficient after complement is calculated, and E = Σe (i) is calculated, and the optimum values of the weighting factors k0 and k1 are determined while changing the weighting factors k0 and k1 using the steepest descent method or the like so that E becomes small. The weighting factors k0 and k1 determined as the optimum values are adopted and used for the table of the weighting factor k.

  When the average luminance of a block is input as an image feature, a weighting factor k corresponding to the average luminance is prepared in advance as a table, and the weighting factor k corresponding to the average luminance is determined using the table. . Again, the weighting coefficient is determined by the aforementioned subjective evaluation experiment or machine learning.

The coefficient complementing means 201 receives the code data transmitted from the encoder 100 via the transmission path, and receives the weight coefficient k group determined by the weight coefficient determining means 203. Then, the orthogonal transform coefficient c (m, n) included in the code data is subjected to a complementing operation by the function fm _{, n} shown in the formula (2) or (3) of the weighting coefficient k, and after completion The orthogonal transform coefficient c ′ (m, n) is calculated.

As described above, according to the image quality improvement apparatus 200 of the second embodiment, the coefficient complementing unit 201 uses the function fm _{, n} of the weighting coefficient k determined from the information included in the code data, and uses the orthogonal transform coefficient c. A complementary operation is performed on (m, n). Thereby, there exists an effect similar to Example 1. Further, the weighting factor determination means 203 determines the weighting factor k of the function fm _{, n} for performing the complementary operation based on the image feature in each block constituting the image. As a result, a complementary calculation according to the image scene becomes possible, so that the subjective image quality can be further improved. Further, since the feature extraction unit 202 extracts the image feature based on the information included in the code data, the encoder 100 newly generates information necessary for the complementary operation according to the image scene. There is no need to do so, and the processing burden can be reduced.

Next, Example 3 will be described. FIG. 4 is a block diagram showing Example 3 of the image quality improvement apparatus according to the embodiment of the present invention. Comparing the third embodiment and the first embodiment, both the embodiments use a predetermined function fm, with the orthogonal transform coefficient c (m, n) included in the code data transmitted by the encoder 100 as an argument _{. n} is the same in that the orthogonal transform coefficient c (m, n) is complemented using _n , but in the first embodiment _, the weight coefficient of the function fm _{, n} is a predetermined value. The third embodiment is different in that the weight coefficient is determined based on information added to the code data transmitted from the encoder 100. Further, when the third embodiment and the second embodiment are compared, both the embodiments are the same in that the weighting factor is determined based on the information transmitted from the encoder 100. The weighting factor is determined based on information included in the code data, whereas in the third embodiment, the weighting factor is determined based on information added to the code data (information different from the code data). It is different in point. That is, in the third embodiment, the image quality improving apparatus 200 determines a mapping based on additional information transmitted from the encoder 100 together with the code data.

  Referring to FIG. 4, the image processing system including the image quality improvement apparatus includes an encoder 100, an image quality improvement apparatus 200, and a decoder 300. The encoder 100 and the image quality improvement apparatus 200 are connected by a transmission path. The image quality improvement apparatus 200 is inserted between the encoder 100 and the decoder 300.

  Similar to the first and second embodiments, the encoder 100 is an existing standard encoder, and divides an image into one or more blocks and performs orthogonal transform for each block to calculate orthogonal transform coefficients. To do. Then, quantization is performed, and the code data including the quantized orthogonal transform coefficient is transmitted to the image quality improvement apparatus 200 via the transmission path, and information to be added to the code data is also transmitted. This additional information is information for determining the weighting factor k. For example, an identifier (switching information) for determining a weighting factor.

  The image quality improving apparatus 200 includes coefficient complementing means 201 and weighting coefficient determining means 203. When the image quality improvement apparatus 200 receives code data and additional information transmitted from the encoder 100 via a transmission path, the coefficient complementing means 201 inputs the code data, and the weighting coefficient determination means 203 inputs additional information. . Then, the weight coefficient determination unit 203 determines the weight coefficient k group based on the additional information. For example, when an identifier is input as additional information, a weighting factor k group corresponding to the identifier is prepared in advance as a table, and the weighting factor k group corresponding to the identifier is determined using the table. In this case, the identifier is used to select a weighting factor k group corresponding to the identifier from a plurality of weighting factor k groups stored in the table. Therefore, the identifier is also switching information for switching from the currently selected weighting factor k group to another weighting factor k group corresponding to the identifier.

The coefficient complementing unit 201 has the same function as in the second embodiment. The coefficient complementing means 201 receives the code data transmitted from the encoder 100 via the transmission path, and receives the weight coefficient k group determined by the weight coefficient determining means 203. Then, the orthogonal transform coefficient c (m, n) included in the code data is subjected to a complementing operation by the function fm _{, n} shown in the formula (2) or (3) of the weighting coefficient k, and after completion The orthogonal transform coefficient c ′ (m, n) is calculated.

  The decoder 300 has a function equivalent to that shown in FIG. That is, the decoder 300 receives the orthogonal transform coefficient c ′ (m, n) supplemented by the image quality improving apparatus 200, performs a decoding process using the complemented orthogonal transform coefficient, and obtains a decoded image. .

As described above, according to the image quality improvement apparatus 200 of the third embodiment, the coefficient complementing unit 201 uses the function fm _{, n} of the weighting coefficient k determined from the additional information _, and uses the orthogonal transform coefficient c (m, n ) Complement operation. Thereby, there exists an effect similar to Example 1. Also, the weighting factor determination unit 203 determines the weighting factor k of the function fm _{, n} for performing the complementary operation based on additional information different from the code data transmitted by the encoder 100. As a result, a complementary calculation according to the intention on the encoder 100 side becomes possible, so that the subjective image quality can be further improved.

  The present invention has been described with reference to the embodiments. However, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the technical idea thereof. For example, the orthogonal transform by the encoder 100 may be performed directly on the luminance value and the color component of the original image, or may be performed on the image obtained as a result of the preprocessing. For example, in the case of a moving image, motion compensation may be performed as preprocessing, and orthogonal transformation may be performed on the residual of the motion compensation. Alternatively, the original image may be subjected to filter processing such as smoothing, and orthogonal transformation may be performed on the value after the processing. In this case, discrete cosine transform (DCT), discrete Hadamard transform (DHT), Karoonen-Loeve transform (KLT), wavelet transform (WT), etc. can be used as orthogonal transform.

  In the encoder 100, the orthogonal transform coefficient c (m, n) is distributed in two dimensions in the horizontal direction and the vertical direction as shown in FIG. You may make it do.

  In the first to third embodiments, the image quality improvement apparatus 200 is independent as an apparatus different from the decoder 300 and is inserted between the encoder 100 and the decoder 300. It may be configured as a part of 300. In this case, the decoder 300 has the function of the image quality improvement apparatus 200 and receives the code data transmitted from the encoder 100 via the transmission path. In the case of the third embodiment, the decoder 300 receives additional information together with code data.

It is a block diagram which shows Example 1 of the image quality improvement apparatus which concerns on embodiment of this invention. It is a figure explaining an example of complementation of an orthogonal transformation coefficient. It is a block diagram which shows Example 2 of the image quality improvement apparatus which concerns on embodiment of this invention. It is a block diagram which shows Example 3 of the image quality improvement apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the conventional image processing system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Encoder 200 Image quality improvement apparatus 201 Coefficient complementing means 202 Feature extracting means 203 Weight coefficient determining means 300 Decoder

Claims (6)

An apparatus for improving the image quality of an image encoded by orthogonal transform and quantization,
The code data of the encoded image is input, and an argument of a non-zero orthogonal transform coefficient is set based on a function defined in advance using the orthogonal transform coefficient as an argument for the orthogonal transform coefficient included in the code data. An image quality improving apparatus comprising coefficient complementing means for performing a complementary operation so that a zero orthogonal transform coefficient is non-zero, and newly generating an orthogonal transform coefficient of a component lost by the quantization. The image quality improving apparatus according to claim 1,
The image quality improvement apparatus according to claim 1, wherein the orthogonal transform coefficient as an argument of the function is a part of all the orthogonal transform coefficients included in the code data. The image quality improving apparatus according to claim 1 or 2,
The image quality improvement apparatus according to claim 1, wherein the predetermined function performs an operation by multiplying an orthogonal transform coefficient by a weighting coefficient. In the image quality improvement device according to claim 3,
When the orthogonal transform coefficient included in the code data is non-zero, the predetermined function is used as the new orthogonal transform coefficient as it is, and when the orthogonal transform coefficient included in the code data is zero, An image quality improvement apparatus for performing an operation by multiplying an orthogonal transformation coefficient by a weighting coefficient. In the image quality improvement device according to claim 3 or 4,
Further, feature extraction means for inputting the code data and extracting image features based on information included in the code data;
An image quality improving apparatus comprising: a weighting factor determining unit that determines a weighting factor used in the coefficient complementing unit based on the image feature extracted by the feature extracting unit. In the image quality improvement device according to claim 3 or 4,
Furthermore, it is information different from the code data, and includes an identifier added to the code data, and based on the identifier, weight coefficient determination means for determining a weight coefficient used in the coefficient complementing means is provided. An image quality improving apparatus characterized by that.

Images