JP4743604B2 - Image processing apparatus, image processing method, program, and information recording medium - Google Patents

Description

  The present invention relates to the field of moving image compression / decompression, and more specifically, when a code subjected to frequency conversion in the time direction is expanded for each group of moving image frames (GOF: Group of Frames). The present invention relates to a technique for suppressing the occurrence of boundary distortion.

  Japanese Patent Application Laid-Open No. 2004-133826 describes a method that uses a discrete wavelet transform to generate a frame to be interpolated in order to smooth the motion of a moving image. In this method, a discrete wavelet transform in the time direction (frame arrangement direction) is performed every N consecutive frames of a moving image, and a frame k having a coefficient value of the lowest frequency component in the generated wavelet coefficients as a pixel value is obtained. In addition, a threshold value for removing the high-frequency component in the next stage is determined. Next, discrete wavelet transform in the time direction is performed on the N + 1 frame obtained by interpolating the frame k into the original N frame, and high frequency components exceeding the threshold value in the generated wavelet coefficients are removed. Discrete wavelet inverse transform is performed on the wavelet coefficients after removal of the high-frequency components, and an interpolation frame composed of the pixel values of the obtained frame k is generated.

  In a method such as JPEG2000 in which an image is divided into tiles and compressed using wavelet transform for each tile, when the image is expanded from the compressed data, distortion may occur at the tile boundary. In Patent Document 2, in such decompression processing of compressed data, tile boundary distortion is effectively removed by correcting high-pass coefficients in the vicinity of tile boundaries using low-pass coefficients and high-pass coefficients of adjacent tiles. A method is described.

  As a high-definition moving image compression method, there is Motion-JPEG2000 which encodes each frame with JPEG2000. This compression method has the advantage that individual frames can be processed independently. However, since the difference between frames is not used during compression, it is not always advantageous in terms of image quality compared to a compression method such as MPEG that uses the difference between frames. It may not be. As a countermeasure, for a moving image in which each frame such as a Motion-JPEG2000 moving image is two-dimensionally frequency-converted, a frequency such as wavelet transform in the time direction for each group of consecutive frames (GOF: Group of Flames). Attempts have been made to three-dimensionally encode moving images by performing conversion. However, it has been pointed out that a GOF boundary distortion (jerk) that fluctuates like a GOF boundary occurs when the code of a moving image frequency-converted in the time direction is expanded in such a GOF unit ( For example, refer nonpatent literature 1).

JP 2002-199400 A JP 2001-257596 A C. Parisot, M. Antonini, M.M. Barlaud, “Motion-compensated scan based wavelet transform for video coding,” Proceedings of the International Workshop on Digital Communications, Capri, Italy, September 2002, pp. 121-127

  An object of the present invention is to provide an image processing apparatus and method for effectively suppressing GOF boundary distortion, which is a problem when decompressing a code of a moving image frequency-converted in the time direction for each GOF.

According to a first aspect of the present invention, there is provided an inverse frequency conversion means for performing an inverse frequency conversion in the time direction on a frequency coefficient generated by performing frequency conversion in the time direction for each group of moving image frames (hereinafter referred to as GOF). A low frequency coefficient when a high frequency coefficient adjacent to a GOF boundary in a frequency coefficient subjected to inverse frequency conversion by the frequency conversion means is not subjected to GOF division prior to the inverse frequency conversion. An image processing apparatus comprising coefficient correction means for correcting to be equal to an inverse frequency conversion value using only

According to the second aspect of the present invention, an inverse frequency transform that performs an inverse frequency transform in the time direction on a frequency coefficient generated by performing a frequency transform hierarchically in the time direction for each group of moving image frames (hereinafter referred to as GOF). A high frequency coefficient adjacent to a GOF boundary in a frequency coefficient of a specific layer or lower that is reverse frequency converted by the reverse frequency conversion means, prior to the reverse frequency conversion, an inverse frequency conversion value centered on the high frequency coefficient is GOF. An image processing apparatus comprising coefficient correction means for correcting to be equal to an inverse frequency conversion value using only a low frequency coefficient when no division is performed.

The invention of claim 3 is the image processing apparatus according to claim 1 or 2, wherein the frequency transform is a 5 × 3 wavelet transform, and the inverse frequency transform is a 5 × 3 wavelet inverse transform, The coefficient correction means is a series of frequency coefficients in the vicinity of the interleaved GOF boundary before the inverse frequency conversion by the inverse frequency conversion means.
L1 H1 L2 H2 L3 H3 L4 H4
(However, L1, L2, L3, and L4 are low frequency coefficients, H1, H2, H3, and H4 are high frequency coefficients, and there is a boundary between two GOFs temporally continuous between H2 and L3, and L1 , H1, L2, and H2 belong to the GOF on the earlier side in time, and L3, H3, L4, and H4 belong to the GOF on the later side in time), the correction value of the high-frequency coefficient H2 adjacent to the GOF boundary
H2 = (1/3) H1- (2/3) L2 + (2/3) L3
The image processing apparatus is characterized by being calculated by the following.

The invention of claim 4 is the image processing apparatus according to claim 1 or 2, wherein the frequency transform is a 9 × 7 wavelet transform, the inverse frequency transform is a 9 × 7 wavelet inverse transform, and The coefficient correction means is a series of frequency coefficients in the vicinity of the interleaved GOF boundary before the inverse frequency conversion by the inverse frequency conversion means.
H0 L1 H1 L2 H2 L3 H3 L4 H4
(However, L1, L2, L3, and L4 are low frequency coefficients, H0, H1, H2, H3, and H4 are high frequency coefficients, and there are two temporally continuous GOF boundaries between H2 and L3. H0, L1, H1, L2, and H2 belong to the GOF on the earlier side in time, and L3, H3, L4, and H4 belong to the GOF on the later side in time), and the high-frequency coefficient H2 adjacent to the GOF boundary Correction value
H2 = (-0.0535H0 + 0.15644H1 + 0.09127L1-0.59127L2 + 0.59127L3-0.09127L4) /0.60295
The image processing apparatus is characterized by being calculated by the following.

A fifth aspect of the present invention is the image processing apparatus according to any one of the first to fourth aspects of the present invention, wherein the correction of the high frequency coefficient adjacent to the GOF boundary by the coefficient correction means has a predetermined number of quantization steps. An image processing apparatus that is applied only when the value is smaller than the value.

A sixth aspect of the present invention is the image processing apparatus according to any one of the first to fifth aspects of the present invention, wherein the correction of the high frequency coefficient adjacent to the GOF boundary by the coefficient correcting means is performed by a user by a GOF boundary distortion. An image processing apparatus that is applied only when suppression is designated.

A seventh aspect of the present invention is the image processing apparatus according to any one of the first to sixth aspects of the present invention, wherein the coefficient correction means is configured such that a correction value of a high frequency coefficient adjacent to a GOF boundary is a correction value of the high frequency coefficient. Clipping processing to correct the correction value of the high-frequency coefficient to the value at the end of the quantization section closer to the correction value of the high-frequency coefficient when the previous value belongs to the quantization section An image processing apparatus characterized by the above.

The invention according to claim 8 is a reverse frequency conversion step of performing reverse frequency conversion in the time direction on a frequency coefficient generated by performing frequency conversion in the time direction for each group of moving image frames (hereinafter referred to as GOF), The high frequency coefficient adjacent to the GOF in the frequency coefficient subjected to the reverse frequency conversion in the reverse frequency conversion process is the low frequency coefficient when the reverse frequency conversion value centered on the high frequency coefficient is not subjected to the GOF division prior to the reverse frequency conversion. An image processing method characterized by having a coefficient correction step of correcting so as to be equal to the inverse frequency conversion value using only .

The invention according to claim 9 is an inverse frequency transform that performs inverse frequency transformation in the time direction on a frequency coefficient generated by hierarchical frequency transformation in the time direction for each group of moving image frames (hereinafter referred to as GOF). The reverse frequency conversion value centered on the high frequency coefficient is set to GOF prior to the reverse frequency conversion of the high frequency coefficient adjacent to the GOF boundary in the frequency coefficient below the specific layer that is reverse frequency converted by the reverse frequency conversion step. It is an image processing method characterized by having a coefficient correction process which corrects so that it may become equal to the inverse frequency conversion value using only the low frequency coefficient when not dividing.

The invention of claim 10 is an image processing method according to the invention of claim 8 or 9, wherein
The frequency transform is a 5 × 3 wavelet transform, and the inverse frequency transform is a 5 × 3 wavelet inverse transform;
The coefficient correction step includes a sequence of frequency coefficients in the vicinity of the interleaved GOF boundary before the reverse frequency conversion is performed by the reverse frequency conversion step.
L1 H1 L2 H2 L3 H3 L4 H4
(However, L1, L2, L3, and L4 are low frequency coefficients, H1, H2, H3, and H4 are high frequency coefficients, and there is a boundary between two GOFs temporally continuous between H2 and L3, and L1 , H1, L2, and H2 belong to the GOF on the earlier side in time, and L3, H3, L4, and H4 belong to the GOF on the later side in time), the correction value of the high-frequency coefficient H2 adjacent to the GOF boundary
H2 = (1/3) H1- (2/3) L2 + (2/3) L3
It is an image processing method characterized by calculating by the following.

The invention of claim 11 is an image processing method according to the invention of claim 8 or 9,
The frequency transform is a 9 × 7 wavelet transform, and the inverse frequency transform is a 9 × 7 wavelet inverse transform;
The coefficient correction step includes a sequence of frequency coefficients in the vicinity of the interleaved GOF boundary before the reverse frequency conversion is performed by the reverse frequency conversion step.
H0 L1 H1 L2 H2 L3 H3 L4 H4
(However, L1, L2, L3, and L4 are low frequency coefficients, H0, H1, H2, H3, and H4 are high frequency coefficients, and there are two temporally continuous GOF boundaries between H2 and L3. H0, L1, H1, L2, and H2 belong to the GOF on the earlier side in time, and L3, H3, L4, and H4 belong to the GOF on the later side in time), and the high-frequency coefficient H2 adjacent to the GOF boundary Correction value
H2 = (-0.0535H0 + 0.15644H1 + 0.09127L1-0.59127L2 + 0.59127L3-0.09127L4) /0.60295
It is an image processing method characterized by calculating by the following.

The invention of claim 12 is an image processing method according to any one of claims 8 to 11,
In the image processing method, the high-frequency coefficient adjacent to the GOF boundary in the coefficient correction step is corrected only when the number of quantization steps is smaller than a predetermined value.

The invention of claim 13 is an image processing method according to any one of claims 8 to 12,
The image processing method is characterized in that the correction for the high-frequency coefficient adjacent to the GOF boundary in the coefficient correction step is performed only when the user specifies GOF boundary distortion suppression.

The invention of claim 14 is an image processing method according to any one of claims 8 to 12,
In the coefficient correction step, when the correction value of the high-frequency coefficient adjacent to the GOF boundary protrudes from the quantization section to which the value before the correction of the high-frequency coefficient belonged, the coefficient on the side close to the correction value of the high-frequency coefficient An image processing method characterized by performing a clipping process for correcting the correction value of the high-frequency coefficient to the end value of the conversion interval.

A fifteenth aspect of the invention is a program that causes a computer to function as an inverse frequency conversion unit and a coefficient correction unit of the image processing apparatus according to any one of the first to seventh aspects.

According to a sixteenth aspect of the present invention, there is provided a computer-readable information recording medium on which a program for causing a computer to function as an inverse frequency converting unit and a coefficient correcting unit of the image processing apparatus according to any one of the first to seventh aspects is recorded. .

According to the first to fourteenth aspects of the present invention, the correction for suppressing the GOF boundary distortion is performed on the high-frequency coefficient in the vicinity of the GOF boundary prior to the inverse frequency conversion in the time direction. Is possible.

  Note that the high-frequency coefficient corrected to suppress the GOF boundary distortion is specifically a high-frequency coefficient adjacent to the GOF boundary as described in an embodiment below, and a vicinity of the GOF boundary but not adjacent to the GOF boundary. It is a high frequency coefficient. Since the correction effect is high for the high frequency coefficient adjacent to the GOF boundary, the correction may be performed only for the high frequency coefficient adjacent to the GOF boundary.

  Further, the correction for suppressing the GOF boundary distortion with respect to the high-frequency coefficient near the GOF boundary is the reverse frequency using only the low frequency coefficient when the inverse frequency transform value centered on the coefficient adjacent to the GOF boundary is not GOF divided. The correction is approximately equal to the conversion value, and the correction formula depends on the type of frequency conversion to be applied. In a later-described embodiment, correction equations in the case of two types of wavelet transforms defined in JPEG2000 are shown.

In addition, according to the inventions of claims 2, 5 , 9 , and 12 , it is possible to reduce the amount of processing for suppressing GOF boundary distortion and achieve GOF boundary distortion suppression by efficient processing. According to the sixth and thirteenth aspects of the present invention, when the user does not want to suppress the GOF boundary distortion, the coefficient correction process for suppressing the GOF boundary distortion is not performed, thereby speeding up the moving image reproduction process. There is an effect that can be done.

According to the invention of claim 15, 16 can be an image processing apparatus using a computer to easily realized according to the invention of claims 1 to 7.

  When a code obtained by dividing and compressing an image by JPEG2000 is expanded, tile boundary distortion may be conspicuous when the compression rate is high. The tile boundary distortion removal (detiling) method described in Patent Document 2 is intended to remove tile boundary distortion in the state of wavelet coefficients in the decompression process, and when the mechanism is understood, implementation of the present invention described below will be described. It will be easier to understand the form.

  First, a specific example of the tile boundary distortion removal method will be described together with the JPEG2000 wavelet transform / inverse transform, the cause of tile boundary distortion, and the like.

The tile boundary distortion removal method is a method similar to image restoration in a wavelet coefficient space. In image restoration, in order to restore lost information, a certain empirical constraint condition is adopted, and a solution to be restored is calculated based on the condition. In the method of Patent Document 2,
Approximation 1: A tile boundary occurs when the compression ratio is high, and all high-pass coefficients are quantized to 0 regardless of tiling.

Approximation 2: The low-pass coefficient when tiling is equal to the low-pass coefficient without tiling.
Adopting the approximation
{Tile boundary pixel value when inverse wavelet transform is performed using corrected high-pass coefficient (= quantized to 0)} = {Inverse wavelet transform without tiling when high-pass coefficient is 0 Pixel value at the same position when
From this equation, the correction value of the high pass coefficient is calculated (the process of calculating the correction value other than 0 for the high pass coefficient quantized to 0 is image restoration).

  Hereinafter, the calculation of the correction equation will be described by taking 5 × 3 wavelet transform defined in JPEG2000 as an example.

  FIGS. 1 to 4 show an example of a process of performing wavelet transformation (forward transformation) called 5 × 3 transformation in two dimensions (vertical direction and horizontal direction) on a 16 × 16 monochrome image.

As shown in FIG. 1, an xy coordinate is taken, and a pixel value of a pixel whose y coordinate is y is expressed as P (y) (0 ≦ y ≦ 15) for a certain x. In JPEG 2000, first, a coefficient C (2i + 1) is obtained by applying a high-pass filter in the vertical direction (Y-coordinate direction) around a pixel having an odd y-coordinate (y = 2i + 1). Next, a coefficient C (2i) is obtained by applying a low-pass filter around a pixel having an even y coordinate (y = 2i) (this is performed for all x coordinates).

Here, the high-pass filter is expressed by equation (1), and the low-pass filter is expressed by equation (2). The symbol | _x_ | in the equation indicates a floor function of x (a function that replaces the real number x with an integer that does not exceed x and is closest to x).
Formula (1) C (2i + 1) = P (2i + 1) − | _ (P (2i) + P (2i + 2)) / 2_ |
Formula (2) C (2i) = P (2i) + | _ (C (2i-1) + C (2i + 1) +2) / 4_ |

  Note that there may be no adjacent pixel group at the edge of the image with respect to the central pixel. In this case, the insufficient pixel value is compensated by a technique called “mirroring” shown in FIG. . The mirroring is a process that literally folds back the pixel value with the boundary as the center, and regards the folded value as the value of the adjacent pixel group.

  For the sake of simplicity, if the coefficient obtained by the high-pass filter is denoted by H and the coefficient obtained by the low-pass filter is denoted by L, the image in FIG. 1 is converted into an array of coefficients as shown in FIG. The

Subsequently, a high-pass filter is applied to the coefficient array of FIG. 2 in the horizontal direction around the coefficient whose x coordinate is odd (y = 2i + 1), and then the coefficient whose x coordinate is even (x = 2i). A low pass filter is applied centering on (this is done for all y). In this case, P (2i) and the like in the expressions (1) and (2) are read as those representing coefficient values. For simplicity,
The coefficient obtained by applying a low pass filter around the L coefficient is LL,
The coefficient obtained by applying a high-pass filter around the L coefficient is HL,
The coefficient obtained by applying a low-pass filter around the H coefficient is LH,
HH is a coefficient obtained by applying a high-pass filter around the H coefficient,
2 is converted into a coefficient array as shown in FIG. Here, the coefficient group which attached | subjected the same symbol is called a subband. That is, FIG. 3 is composed of four subbands.

  When one wavelet transform (one decomposition (decomposition)) is completed and only LL coefficients are collected (collecting coefficients for each subband as shown in FIG. 4 and extracting only the LL subbands), An “image” having a resolution of 1/2 that of the original image can be obtained (in this way, classification for each subband is called deinterleaving, and arrangement in the state shown in FIG. 3 is expressed as interleaving). .

  In the second wavelet transform, the above LL subband may be regarded as an original image and the same transformation as described above may be performed. In this case, when rearrangement is performed, a schematic FIG. 5 is obtained. The coefficient prefixes 1 and 2 in FIGS. 4 and 5 indicate how many times the coefficient has been obtained by the wavelet transform, and are called decomposition levels. In the above discussion, when only one-dimensional wavelet transform is desired, processing in only one direction may be performed.

On the other hand, in the wavelet inverse transform, an inverse low-pass filter is first applied to the array of interleaved coefficients as shown in FIG. 3 in the horizontal direction, centering on the coefficient whose x coordinate is an even number (x = 2i). An inverse high-pass filter is applied around a coefficient whose x coordinate is an odd number (x = 2i + 1) (this is performed for all y). Here, the inverse low-pass filter is represented by Expression (3), and the inverse high-pass filter is represented by Expression (4). As in the case of the wavelet forward transformation, there may be no adjacent coefficient group for the central coefficient at the edge of the image. In this case as well, the coefficient value is appropriately compensated by the mirroring in FIG.
Formula (3) P (2i) = C (2i) − | _ (C (2i−1) + C (2i + 1) +2) / 4_ |
Formula (4) P (2i + 1) = C (2i + 1) + | _ (P (2i) + P (2i + 2)) / 2_ |

  As a result, the coefficient array in FIG. 3 is converted (inversely converted) into a coefficient array as shown in FIG. Subsequently, in the vertical direction, an inverse low-pass filter is applied centering on a coefficient whose y coordinate is an even number (y = 2i), and then an inverse high pass filter is centered on a coefficient whose y coordinate is an odd number (y = 2i + 1). If this is performed (this is performed for all x), one wavelet inverse transformation is completed, and the image of FIG. 1 is returned (reconstructed). When wavelet transformation is performed a plurality of times, FIG. 1 is regarded as an LL subband, and similar inverse transformation may be repeated using other coefficients such as HL.

  As described above, in the inverse 5 × 3 wavelet transform, the inverse low-pass filter is applied to the center of the even position and the inverse high-pass filter is applied to the center of the odd position for the coefficient sequence interleaved in the order of L, H, L, and H.

  As apparent from the expansion of the filter equations (1) to (4) by omitting the floor function, the number of taps is reversed in the forward transformation and the inverse transformation, and the filter coefficients are also interleaved.

For this reason, the forward conversion filter coefficient is high-pass (HIGH): -0.5, 1, -0.5
Low pass (LOW): -0.125, 0.25, 0.75, 0.25, -0.125
In case of, the inverse transform filter coefficient is inverse low pass (LOW): -0.25, 1, -0.25
Reverse high pass (HIGH): -0.125, 0.5, 0.75, 0.5, -0.125
(Here, the filter coefficient is expressed without considering the floor function part).

“When tiling is not performed and all the high-pass coefficients are 0 (approximate 1)”, an interleaved coefficient sequence
L1 H1 L2 H2 L3 H3
The value obtained by applying an inverse high-pass filter to the center of the H2 position is
-0.125H1 + 0.5L2 + 0.75H2 + 0.5L3-0.125H3 = 0.5L2 + 0.5L3 (i)
It becomes.

On the other hand, it is assumed that the coefficient is calculated after being divided into two tiles, and the H2 position is a tile boundary. In this case, L1, H1, L2, and H2 are coefficients of the left tile, and L3 and H3 are coefficients of the right tile. Here, H2 is a coefficient that is affected by mirroring during forward conversion (a 3-tap high-pass filter) and is a correction target. Tiling is a process that performs wavelet transform using only the pixels in each tile, or a process that performs inverse wavelet transform using only the coefficients in each tile.
L1 H1 L2 H2
Is supplemented to the right
L1 H1 L2 H2 L2 H1
It becomes.

Therefore, the value obtained by applying an inverse high-pass filter around the H2 position is
-0.125H1 + 0.5L2 + 0.75H2 + 0.5L2-0.125H1 = -0.25H1 + L2 + 0.75H2 (ii)
It becomes.

In order to detil, in order to aim (i) = (ii)
Formula (iii) H2 = 1 / 3H1-2 / 3L2 + 2 / 3L3
Get. This is a “correction formula” for a high-pass coefficient that is adjacent to the tile boundary and affected by mirroring during forward conversion. However, since only the tiled coefficients exist when calculating the correction value, L2 uses the left tile coefficient and L3 uses the right tile coefficient (approximate 2).

The same equation can be used for the low-pass coefficient (L3) adjacent to the tile boundary. As a result, the equation (iv) H3 = 0 is used as a correction equation for H3.
Can be obtained. However, in the case of decomposition level 1, H3 does not include a mirroring error, so it is not corrected. To correct it).

For the irreversible 9 × 7 wavelet defined in JPEG2000, a correction formula can be derived based on the same concept as described above. That is, the interleaved coefficient sequence
H0 L1 H1 L2 H2 L3 H3 L4 H4
In the case where the boundary between H2 and L3 is a tile boundary, the following correction formula is obtained.
Formula (v) H2 = (-0.0535H0 + 0.15644H1 + 0.09127L1-0.59127L2
+ 0.59127L3-0.09127L4) /0.60295
Formula (vi) H3 = (0.05754L2-0.05754L4 + 0.03372H4) /0.53372
However, in the case of 9 × 7 conversion, the case of the decomposition level 1 may be corrected.

  Next, the flow of the detiling process described above will be described in detail by taking, as an example, the decompression process of the JPEG 2000 code of four tile division and decomposition level 2. 7 to 14 are coefficient arrangement diagrams for explaining the detiling process. The decompression process proceeds in the order of entropy decoding of the code, inverse quantization of each decoded subband coefficient, two-dimensional inverse wavelet transform, and inverse color space transform. In the process of the two-dimensional inverse wavelet transform, A detile process is executed.

  FIG. 7 shows a coefficient array for four tiles after inverse quantization, and coefficients 2LL to 2HH of the composition level 2 are interleaved. When the 5 × 3 transformation is used, the coefficient of column A (corresponding to H2) shown in FIG. 8 is corrected by equation (iii), and the coefficient of column B (corresponding to H3) is corrected by equation (iv). In the case of 9 × 7 conversion, the coefficient of the column A is corrected by the equation (v), and the coefficient of the column B is corrected by the equation (vi). For such corrected coefficients, the inverse wavelet transform in the horizontal direction is performed (in JPEG 2000, the wavelet transform (forward transform) is performed in the order of the vertical direction and the horizontal direction, so the inverse transform is performed in the horizontal direction and the vertical direction. In order). FIG. 9 shows the coefficient array after the horizontal reverse conversion.

  Next, among the coefficients at decomposition level 2, the coefficient in row C (corresponding to H2) and the coefficient in row D (corresponding to H3) shown in FIG. 10 are corrected. In the case of 5 × 3 conversion, the coefficient of row C is corrected by equation (iii), and the coefficient of row D is corrected by equation (iv). In the case of 9 × 7 conversion, the coefficient of the row C is corrected by the equation (v), and the coefficient of the row D is corrected by the equation (vi). The inverse wavelet transform in the vertical direction is performed on the corrected coefficient, and the decomposition level 1 coefficient as shown in FIG. 11 is deinterleaved. When the coefficients are interleaved, the result is as shown in FIG.

  In the case of 5 × 3 conversion among the coefficients at decomposition level 1 shown in FIG. 12, the coefficients in column E shown in FIG. 13 are corrected by equation (iii). As described above, in the case of 5 × 3 conversion, at the decomposition level 1, the coefficient of the column F does not include a mirroring error, and thus correction is unnecessary. On the other hand, in the case of 9 × 7 conversion, the coefficient of column E is corrected by equation (v), and the coefficient of column F is corrected by equation (vi). After such correction, the inverse wavelet transform in the horizontal direction is performed, and the state shown in FIG. 14 is obtained. In the case of 5 × 3 conversion, the coefficient of the row G in FIG. 14 is corrected by equation (iii). In the case of 9 × 7 conversion, the coefficient of row G is corrected by equation (v), and the coefficient of row H is corrected by equation (vi). The inverse wavelet transform in the vertical direction is performed on the corrected coefficient. This completes the detiling process and the two-dimensional inverse wavelet transform process.

<Description of Embodiment of the Present Invention>
Embodiments of the present invention will be described below. FIG. 15 is a block diagram for explaining an embodiment of the present invention. The image processing apparatus shown in FIG. 15 has a function of taking in moving image frame data, outputting a compressed code, and taking in a compressed moving image code, and outputting the decompressed frame data. The color conversion / inverse conversion unit 1, the three-dimensional wavelet transform / inverse conversion unit 2, the quantization / inverse quantization unit 4, the entropy encoding / decoding unit 5, and the code formation / tag processing unit 6.

  The outline of the operation at the time of compression processing is as follows. The frame data of the moving image is input to the color conversion / inverse color conversion unit 1 (the frame data can be processed by being divided into tiles, but here, for the sake of simplicity, the tile division is not performed. To do). The color conversion / reverse color changing unit 1 converts the frame data (eg, RGB data) into luminance / color difference data such as YCbCr. The converted component data is converted into wavelet coefficient data by the three-dimensional wavelet transform / inverse transform unit 2 by applying a three-dimensional forward wavelet transform including the time direction (frame arrangement direction). The wavelet coefficient data of each component is quantized by the quantization / inverse quantization unit 4 and then entropy-encoded by the entropy encoding / decoding unit 5. Then, the code generation / tag processing unit 6 generates a packet from the necessary entropy code, arranges the packets in a predetermined order, and adds the necessary tags and tag information to form a code, which is output to the outside. Is done.

  The outline of the operation during the decompression process is as follows. The compressed code of the moving image is input to the code forming / tag processing unit 6 and separated into codes for each component. The code of each component is entropy-decoded by the entropy encoding / decoding unit 5 and returned to wavelet coefficient data. The wavelet coefficient data is inversely quantized by the quantization / inverse quantization unit 4 and then input to the three-dimensional wavelet transform / inverse transform unit 2 to be subjected to three-dimensional inverse wavelet transform. The three-dimensional wavelet transform / inverse transform unit 2 includes a coefficient correction unit 2 for suppressing the GOF boundary distortion with respect to a high-pass coefficient near the GOF boundary by the coefficient correction unit 2 in the process of the inverse wavelet transform in the time direction. Coefficient correction processing is performed. The coefficient correction process at the time of inverse wavelet transform in the time direction is the greatest feature of the present invention.

  Hereinafter, two embodiments of the present invention will be described more specifically. Here, a description will be given assuming that the JPEG2000 standard 5 × 3 conversion or 9 × 7 conversion is used as the wavelet conversion.

[Embodiment 1]
Regarding the present embodiment, the processing of the three-dimensional wavelet transform / inverse transform unit 2 during compression processing will be described first.

  FIG. 16 is a block diagram showing a functional configuration during the compression process of the three-dimensional wavelet transform / inverse transform unit 2, 21 is an intra-frame two-dimensional forward wavelet transform unit, and 22 is a time direction forward wavelet transform unit.

  During compression processing, moving image frame data (after color conversion) as schematically shown in FIG. 17 is sequentially input to the intra-frame two-dimensional order wavelet transform means 21, and the two-dimensional order wavelet transform is applied to each frame data. The FIG. 18 schematically shows the result of the two-dimensional forward wavelet transform up to two layers (decomposition level 2). The result of such a two-dimensional forward wavelet transform is input to the time direction forward wavelet transform means 22, and the forward wavelet transform is applied in the time direction in units of a predetermined number of consecutive frames (GOF). FIG. 19 schematically shows the result of time-direction wavelet transform up to two layers (decomposition level 2).

  FIG. 20 is a flowchart of such a three-dimensional forward wavelet transform process. In the intra-frame two-dimensional forward wavelet transform means 21, in the processing loop of steps 1, 2, 3, 4, 5, and 6, two frames (vertical direction and horizontal direction) are obtained in units of one frame from the first frame to the last frame of GOF. A forward wavelet transform is performed. That is, the forward wavelet transform in the vertical direction is performed in step 1, and then the forward wavelet transform in the horizontal direction is performed in step 2, so that wavelet coefficient data of decomposition level 1 is generated. When the forward wavelet transform of two or more layers is applied, the two-dimensional forward wavelet transform according to steps 1 and 2 is performed on the LL coefficient data of the wavelet coefficient data at the decomposition level 1, and the wavelet coefficient data at the decomposition level 2 is obtained. Is generated. When the conversion up to a predetermined layer is completed (step 3, Yes), the same two-dimensional forward wavelet conversion is performed on the next frame.

  When the two-dimensional wavelet transform up to a predetermined layer for the GOF final frame is completed (step 5, Yes), the time direction forward wavelet transform means 22 performs steps 7, 8, and 9 from the first frame of the GOF to the final frame. A one-dimensional forward wavelet transform in the time direction is performed on the corresponding coefficient sequence (wavelet coefficients are in the order of L, H, L, H... In the time direction). When two or more layers are applied in the time direction forward wavelet transform, the wavelet coefficients are deinterleaved (L coefficients and H coefficients are combined), and the coefficients in the same position of the frame are used for the L coefficient data in the time direction. The forward wavelet transform is repeated. When the conversion is completed up to a predetermined hierarchy (step 7, Yes), the three-dimensional order wavelet conversion for one GOF is completed.

  Next, the process of the three-dimensional wavelet transform unit 2 during the decompression process in the present embodiment will be described.

  FIG. 21 is a block diagram showing a functional configuration of the three-dimensional wavelet transform / inverse transform unit 2 during decompression processing. 3 is a coefficient correction unit, 31 is a time direction inverse wavelet transform unit, and 32 is a two-dimensional inverse in frame It is a wavelet transform means.

  During the decompression process, the wavelet coefficient data decoded by the entropy encoding / decoding unit 5 and dequantized by the quantization / inverse quantization unit 4 is input to the three-dimensional wavelet transform / inverse transform unit 2, and the highest layer ( The three-dimensional inverse wavelet transform is executed from the highest decomposition level) to the lower hierarchy. In the present embodiment, the time direction inverse wavelet transform unit 31 first performs the time direction inverse wavelet transform on the wavelet coefficient data of each GOF. In order to suppress the GOF boundary distortion prior to the inverse wavelet transform of each layer. The coefficient correction unit 3 performs the coefficient correction. For the result (wavelet coefficient data for each frame) in which the coefficient correction and the inverse wavelet transform in the time direction are repeated to the lowest layer, the intra-frame two-dimensional inverse wavelet transform means 32 changes the highest layer to the lower layer. On the other hand, two-dimensional inverse wavelet transform is performed, and frame data in which GOF boundary distortion is suppressed is generated. Since this frame data has been subjected to color conversion, the color conversion / reverse color conversion unit 1 performs reverse color conversion to restore the original color system frame data.

  FIG. 23 is a flowchart showing a processing flow by such a two-dimensional inverse wavelet transform means 32. This process is executed for each frame from the first frame of the GOF after the time direction inverse wavelet transform and coefficient correction to the last frame. In step 21, the two-dimensional inverse wavelet transform is applied to the wavelet coefficient data in the highest hierarchy, and the LL coefficient data in the next hierarchy is generated. Since the forward wavelet transform at the time of compression is performed in the order of the vertical direction and the horizontal direction, the inverse wavelet transform at the time of decompression is performed in the order of the horizontal direction and the vertical direction. Similar two-dimensional inverse wavelet transform is applied to the wavelet coefficient data of the next layer including the generated LL coefficient data. If the two-dimensional inverse wavelet transform is performed up to the lowest layer and returned to the frame data (step 22, Yes), the processing target is advanced to the next frame (step 25), and the same two-dimensional inverse wavelet transform is performed. When such a two-dimensional inverse wavelet is completed up to the last frame of GOF (step 24, Yes), data of all frames of one GOF is restored.

  The image processing apparatus according to the present embodiment described above in further detail below includes a CPU 1000, a memory 1001, a hard disk device 1002, a display device 1004, and a pointing device such as a keyboard and a mouse as shown in FIG. It can be realized by using computer resources such as a user input device 1003 such as the above, a communication interface 1005 for communication via a network, and the like connected by a bus 1006. That is, by loading a program that causes such a computer to function as the units 1, 2, 4, 5, and 6 shown in FIG. 15 into the memory 1001 and causing the CPU 1000 to execute the program, the computer as the image processing apparatus according to the present embodiment. Can be operated. Such a program and a computer-readable information recording (storage) medium such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor storage element on which the program is recorded are also included in the present invention. On the computer, the compression process and the expansion process are executed in the following flow, for example.

  In the case of compression processing, moving image frame data is read into the memory 1001 from the hard disk device 1002, for example. The frame data is subjected to the processes of color conversion, three-dimensional forward wavelet conversion, quantization, entropy encoding, and code formation as described above. The code of the generated moving image is stored in the hard disk device 1002 or transmitted from the communication interface 1005 to an external device.

  In the decompression process, the moving image code is read into the memory 1001 from the hard disk device 1002, for example, or received from the communication interface 1005 via the network and read into the memory 1001. For this code, each process of time direction inverse wavelet transform including entropy decoding, inverse quantization, coefficient correction, intra-frame two-dimensional inverse wavelet transform, and inverse color transform is executed. The restored moving image frame data is stored in the hard disk device 1002 or displayed on the display device 1004.

  Next, the processing of the coefficient correction unit 3 and the time direction inverse wavelet transform unit 31 during the expansion process will be further described with reference to five examples. For convenience of explanation, it is assumed that the intra-frame forward wavelet transform and the time direction forward wavelet transform at the time of compression are each performed up to two layers (decomposition level 2) unless otherwise specified.

(Example 1 of Embodiment 1)
Example 1 according to this embodiment will be described with reference to FIGS. 22 and 24 to 27. FIG. 22 is a flowchart showing a processing flow of coefficient correction and time direction inverse wavelet transform. With respect to the wavelet coefficient data of the highest hierarchy in the time direction, in order to suppress the GOF boundary distortion, the coefficient correction means 3 corrects the high-pass coefficient adjacent to the GOF boundary and the high-pass coefficient not adjacent to the boundary near the GOF boundary (step 11). , Step 12). The time direction inverse wavelet transform is performed on the coefficient data of the highest layer after the coefficient correction by the time direction inverse wavelet transform means 31 (step 13). The layer is lowered by one (step 15), the same coefficient correction is performed on the wavelet coefficients of that layer (step 11, step 12), and the inverse wavelet transform in the time direction is performed on the corrected coefficient data (step 13). The above process is repeated up to the wavelet coefficient of the decomposition level 1 in the time direction, and when the two-dimensional wavelet coefficient data of each frame in the GOF is returned (step 14, Yes), the coefficient correction and the time direction inverse wavelet transform process are completed. .

  The contents of such processing will be described with reference to three-dimensional wavelet coefficient data as shown in FIG. Here, JPEG2000 5 × 3 conversion is used as wavelet conversion, but 9 × 7 conversion is the same except that the coefficient correction formula is different. In addition, here, the correction of the time direction wavelet coefficient corresponding to the 2LL subband coefficient in the frame and the inverse wavelet transform will be described as an example, but the same applies to the time direction wavelet coefficient corresponding to the other subband coefficients in the frame. Think about it.

  First, the coefficients 2L and 2H of the decomposition level 2 in the time direction are interleaved (2L, 2H, and 2H are rearranged in the order of 2L, 2H, 2L, 2H,...). Regarding the interleaved coefficients 2L and 2H, a high-pass coefficient (2LL2H (t6)) adjacent to the GOF boundary indicated by the dotted line in FIG. 24B and a high-pass coefficient (2LL2H (t10)) not adjacent to the boundary near the GOF boundary Correct. The high pass coefficient correction formula is as shown in FIG.

The correction formula for 9x7 conversion is
2LL2H (t6) = (-0.0535x2LL2H (t2) + 0.15644x2LL2H (t2) + 0.09127x2LL2L (t0)
-0.59127x2LL2L (t4) + 0.591272x2LL2L (t8) -0.09127x2LL2L (t10)) / 0.60295
2LL2H (t10) = (0.05754x2LL2L (t4) -0.05754x2LL2L (t8) + 0.03372x2LL2H (t14)) / 0.53372
It becomes.

  The same coefficient correction is performed on the wavelet coefficients of the composition level 2 in the time direction corresponding to the 2LH, 2HL, 2HH, 1LH, 1HL, and 1HH subband coefficients in the frame. The same applies to the correction formula.

  The inverse wavelet transform in the time direction is performed on the wavelet coefficient at the decomposition level 2 after such coefficient correction. In the case of the wavelet coefficient at the decomposition level 2 in FIG. 24B, the coefficient at the decomposition level 1 (2LL1L) as shown in FIG. 24C is generated by the inverse wavelet transform.

  Next, coefficient correction and inverse wavelet transform are performed on the wavelet coefficients of the decomposition level 1 in the time direction. FIG. 25 is a diagram for explaining this. Interleave the coefficients 1L and 1H of the decomposition level 1 shown in FIG. 25 (a), and adjoin the high-pass coefficient (2LL1H (t7)) adjacent to the GOF boundary shown in FIG. 25 (b) and the boundary near the GOF boundary. Correct the high-pass coefficient (2LL1H (t9)). The coefficient correction formula is as shown in FIG. In the case of 5 × 3 conversion, the high-pass coefficient (2LL1H (t9)) not adjacent to the decomposition level 1 GOF boundary is not affected by mirroring and is not corrected. However, in the 9 × 7 conversion, a high-pass coefficient (2LL1H (t9)) that is not adjacent to the GOF boundary is also affected by mirroring, and is corrected.

The correction formula for 9x7 conversion is
2LL1H (t7) = (-0.0535x2LL1H (t3) + 0.15644x2LL1H (t5) + 0.09127x2LL1L (t4)
-0.59127x2LL1L (t6) + 0.591272x2LL1L (t9) -0.09127x2LL1L (t11)) / 0.60295
2LL1H (t9) = (0.05754x2LL1L (t6) -0.05754x2LL1L (t8) + 0.03372x2LL1H (t11)) / 0.53372
It becomes.

  The same coefficient correction is performed on the wavelet coefficients of the decomposition level 1 in the time direction corresponding to the 2LH, 2HL, 2HH, 1LH, 1HL, and 1HH subband coefficients in the frame. The same applies to the correction formula.

  The inverse wavelet transform in the time direction is performed on the wavelet coefficient of the composition level 1 after such coefficient correction. In the case of the wavelet coefficients at the decomposition level 1 in FIG. 25B, the inverse wavelet transform returns to the data at the decomposition level 0 as shown in FIG. 25C, that is, the in-frame two-dimensional wavelet coefficients. Become.

  The correction formula is an example, and other correction formulas may be used as long as they contribute to the suppression of GOF boundary distortion in a moving image. In this embodiment, the high-pass coefficient not adjacent to the GOF boundary near the GOF boundary is also subject to correction. However, the effect of suppressing the GOF boundary distortion by the correction is compared with the effect of correcting the high-pass coefficient adjacent to the GOF boundary. Since it is small, it is possible to exclude a high-pass coefficient that is not adjacent to the GOF boundary from the correction target.

  Also, the motion amount of the frame near the GOF boundary is estimated, and when the motion amount is smaller than a predetermined value, coefficient correction for suppressing the GOF boundary distortion is performed, and when the motion amount is larger than the predetermined value, the GOF boundary distortion is suppressed. Control may be performed so as not to perform coefficient correction. A known method may be used to estimate the amount of motion of the frame.

Further, the degree of correction for the coefficient near the GOG boundary may be controlled stepwise or continuously in accordance with the amount of motion of the frame near the GOF boundary. For example, when the motion amount of the frame near the GOF boundary is k, the coefficient value before correction is x0, and the coefficient value after correction is x1, the coefficient value considering the motion amount of the frame near the GOF boundary Where x is
x = (1-k) x0 + k x1
(The amount of motion k is used as a parameter for linear interpolation).

  The above-described deformation modes are the same for the examples described later.

(Example 2 of Embodiment 1)
Example 2 according to this embodiment will be described with reference to FIGS. FIG. 28 is a flowchart showing a processing flow of coefficient correction and time direction inverse wavelet transform. As is clear from a comparison between FIG. 28 and FIG. 22, the step (steps 31 and 32) of clipping processing for the high-pass coefficients corrected by steps 11 and 12 is added to the example 1 according to the present embodiment. It is a difference. This clipping process is executed by the coefficient correction means 3.

  FIG. 29 is an explanatory diagram of clipping processing. In this clipping process, when the correction value of the coefficient due to the GOF boundary suppression protrudes from the quantization interval of the coefficient, the coefficient value is returned to the value at the end closer to the quantization interval. In the case of the example in FIG. 29, the coefficient value after dequantization (before correction) is in the quantization interval [Q, 2Q], and the correction value of the coefficient is 0. The closer Q.

(Example 3 according to Embodiment 1)
Example 3 according to this embodiment will be described with reference to FIGS. FIG. 32 is a flowchart showing a processing flow of coefficient correction and time direction inverse wavelet transform. As is apparent from a comparison between FIG. 32 and FIG. 28, a coefficient quantization step number and threshold thQ comparison determination step step 41 is added, and coefficient correction is performed only when it is determined that the quantization step number> threshold thQ. However, if not, steps 11, 31, 12 and 32 are skipped, which is different from Example 2 according to the present embodiment.

  The reason why the coefficient correction (and therefore the correction value clipping process) is not performed when the number of quantization steps> threshold thQ or less is as follows.

  FIG. 30 is an explanatory diagram of clipping processing when the compression rate is small. Generally, when the compression rate is small, the number of quantization steps is small. Therefore, the example of FIG. 30 can be considered as an example in which the number of quantization steps is small.

  When the coefficient value (original signal) before quantization is between 2Q and 3Q as shown in FIG. 30 (Q is the number of quantization steps), this coefficient is dequantized to a value of 2Q + (3/8) Q. (When the reconstruction parameter is 3/8) Taking the case of 5 × 3 transformation (when the wavelet basis function is R5x3) as an example, the coefficient dequantized to the position of 2Q + (3/8) Q as described above is corrected to 0 (GOF boundary) If the coefficient is not adjacent to), it is then clipped back to the 2Q position. If the value of the quantization step number Q is small, the difference between the value before correction and the value after clipping is (3/8) Q. Therefore, if the number of quantization steps Q is sufficiently small, the difference (3/8) Q between the value before correction and the value after clipping is a minute value that is not so noticeable. That is, when the number Q of quantization steps is equal to or less than the predetermined threshold thQ, even if coefficient correction for suppressing GOF boundary distortion is small, the coefficient correction and clipping processing are not performed in this embodiment. Thus, the amount of processing for suppressing GOF boundary distortion is reduced.

  FIG. 31 is an explanatory diagram of clipping processing when the compression rate is large. In general, when the compression ratio is large, the number of quantization steps is large. Therefore, the example in the figure can be considered as an example in which the number of quantization steps is large. As shown in the figure, when the coefficient value (original signal) before quantization is between Q and 2Q, this coefficient is dequantized to the position of Q + (3/8) Q (the reconstruction parameter is 3/8) in the case of). Taking the case where the wavelet basis function is R5x3 as an example, the coefficient inversely quantized at the position of Q + (3/8) Q as described above is corrected to 0 (in the case of a coefficient not adjacent to the GOF boundary), Next, it is returned to the position Q by clipping. The difference between the value before correction and the value after clipping is (3/8) Q. If the number Q of quantization steps is large, the difference (3/8) Q between the value before correction and the value after clipping becomes a considerably large value. Therefore, in this embodiment, when the quantization step number Q is larger than the predetermined threshold thQ, coefficient correction for suppressing GOF boundary distortion is performed.

(Example 4 according to Embodiment 1)
FIG. 33 is a flowchart showing a processing flow of coefficient correction and time-direction inverse wavelet transform in Example 4 according to the present embodiment. As is clear from a comparison between FIG. 33 and FIG. 28, a comparison determination step step 51 between the decomposition level (number of hierarchies) and a predetermined threshold value thlev is added, and the GOF boundary is applied only to the coefficient of the decomposition level equal to or lower than the threshold value thlev. The coefficient correction for suppressing distortion and the clipping process (steps 11, 31, 12, 32) of the correction values are different from the second example according to the present embodiment.

  The reason why coefficient correction is not performed at a decomposition level higher than the threshold thlev is as follows. One is that the low frequency coefficient with a high decomposition level is repeatedly subjected to low-pass processing, so that it is originally corrected, and even if the coefficient is further corrected, the effect of suppressing the GOF boundary distortion is not effective. Since there are few, it is going to implement | achieve GOF boundary distortion suppression efficiently by excluding the decomposition level higher than predetermined threshold value thlev from the object of coefficient correction. The other is that the side effects of coefficient correction at a high decomposition level extend over many frames, so let's limit the range of side effects of coefficient correction by excluding decomposition levels higher than a predetermined threshold from the target of coefficient correction. It is what. Another reason is that at a high decomposition level, the number of coefficients may be insufficient depending on the image, and coefficient correction itself may not be possible in the first place.

  It should be noted that coefficient correction may be controlled in accordance with the number of quantization steps in the same manner as in the third embodiment with respect to the decomposition level equal to or lower than the threshold thlev, and this aspect is also included in this embodiment.

(Example 5 according to Embodiment 1)
FIG. 34 is a flowchart showing a processing flow of coefficient correction and time-direction inverse wavelet transform in Example 5 according to the present embodiment. As apparent from a comparison between FIG. 34 and FIG. 28, step step 61 for determining whether or not a correction is specified by the user is added, and coefficient correction for suppressing GOF boundary distortion and its correction only when correction is specified. The value clipping processing (steps 11, 31, 12, 32) is different from the second example according to the present embodiment. This is because some users may not want to increase the processing time for suppressing the GOF boundary distortion. In such a case, the process for suppressing the GOF boundary distortion is skipped. This is because it is more reasonable to increase the speed.

  An example of a correction designation method by the user will be described with reference to FIG. Here, it is assumed that the image processing apparatus according to the present embodiment is realized using a computer.

  As shown in the upper diagram of FIG. 35, when the compressed moving image is to be displayed and edited, the movie is selected by selecting “Open” from the “File” menu and specifying the file. Can open compressed image files. Thus, before opening the file, for example, by selecting “property”, it is possible to designate whether or not the GOF boundary distortion suppression function is present as shown in the lower diagram. When “Yes” is selected here, processing for suppressing GOF boundary distortion is executed in the decompression processing of the opened compressed file, and when “No” is selected, GOF is executed in the decompression processing of the compressed file. Does not execute processing for suppressing boundary distortion.

  In addition, it is desirable that the method for opening a file has a configuration in which various parameters can be designated. For example, the number of hierarchies to decode, components, image quality, position, etc. can be set as tabs and selected. One of the tabs may include “GOF boundary distortion suppression processing”. If you select “GOF Boundary Distortion” on the tab, you can select “Yes” or “No” for the GOF boundary suppression function as shown in the figure below, and if “Yes” is selected, “High Quality Mode (or High The image quality mode) and the high speed mode may be selected. In this case, for example, only the high-pass coefficient adjacent to the GOF boundary is corrected in the high-speed mode, and both the high-pass coefficient adjacent to the GOF boundary and the high-pass coefficient not adjacent to the GOF boundary near the GOF boundary are corrected in the high-quality mode. It may be configured as follows.

  Also in the present embodiment, the coefficient correction may be controlled in accordance with the number of quantization steps and / or the composition level as in the third and fourth embodiments, and this aspect is also included in the present embodiment. The

[Embodiment 2]
Regarding the present embodiment, the processing of the three-dimensional wavelet transform / inverse transform unit 2 during compression processing will be described first.

  FIG. 36 is a block diagram showing a functional configuration during the compression process of the three-dimensional wavelet transform / inverse transform unit 2, 21 is an intra-frame two-dimensional forward wavelet transform unit, and 22 is a time direction forward wavelet transform unit.

  At the time of compression processing, moving image frame data (after color conversion) as schematically shown in FIG. 17 is input to the time direction forward wavelet transform unit 22, and a group of a predetermined number of consecutive frames (GOF) is used as a unit. A forward wavelet transform is applied in the time direction. FIG. 38 schematically shows the results of time direction forward wavelet transform up to two layers (decomposition level 2). The result of such time direction forward wavelet transform is input to the intra-frame two-dimensional forward wavelet transform means 21, and forward wavelet transform is performed on each frame (time direction wavelet coefficient data) in the order of the vertical direction and the horizontal direction.

  FIG. 37 is a flowchart showing the flow of such forward wavelet transform processing. Steps 101, 102, and 103 form a processing loop for forward wavelet transform in the time direction. Steps 104, 105, 106, and 107 constitute a processing loop for two-dimensional forward wavelet transform in one frame. When this processing loop is executed up to the last frame of the GOF (step 108, Yes), the three-dimensional forward wavelet transform processing for one GOF is terminated.

  Next, the process of the three-dimensional wavelet transform / inverse transform unit 2 during the decompression process in the present embodiment will be described.

  FIG. 39 is a block diagram showing a functional configuration of the three-dimensional wavelet transform / inverse transform unit 2 during decompression processing. 3 is a coefficient correction unit, 31 is a time direction inverse wavelet transform unit, and 32 is an intra-frame two-dimensional inverse. It is a wavelet transform means.

  During the decompression process, the wavelet coefficient data decoded by the entropy encoding / decoding unit 5 and dequantized by the quantization / inverse quantization unit 4 is input to the three-dimensional wavelet transform / inverse transform unit 2, and the highest layer ( The three-dimensional inverse wavelet transform is executed from the highest decomposition level) to the lower hierarchy.

  In this embodiment, first, the intra-frame two-dimensional inverse wavelet transform means 32 performs two-dimensional inverse wavelet transform on each frame from the highest layer to the lower layer. The processing flow of this intra-frame two-dimensional inverse wavelet transform is as shown in FIG. Next, the time direction inverse wavelet transform means 31 performs the time direction inverse wavelet transform from the highest layer to the lower layer, and coefficient correction for suppressing the GOF boundary distortion prior to the inverse wavelet transform of each layer. Is performed by the coefficient correction means 3, and the frame data in which the GOF boundary distortion is suppressed is reproduced. Since this frame data has been subjected to color conversion, the color conversion / reverse color conversion unit 1 performs reverse color conversion to restore the original color system frame data.

  The image processing apparatus according to the present embodiment described above in further detail described above can be realized using, for example, computer resources as shown in FIG. That is, by loading a program that causes such a computer to function as the units 1, 2, 4, 5, and 6 shown in FIG. 15 into the memory 1001 and causing the CPU 1000 to execute the program, the computer as the image processing apparatus according to the present embodiment. Can be operated. Such a program and a computer-readable information recording (storage) medium such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor storage element on which the program is recorded are also included in the present invention. On the computer, the compression process and the expansion process are executed in the following flow, for example.

  In the case of compression processing, moving image frame data is read into the memory 1001 from the hard disk device 1002, for example. The frame data is subjected to the processes of color conversion, three-dimensional forward wavelet conversion, quantization, entropy encoding, and code formation as described above. The code of the generated moving image is stored in the hard disk device 1002 or transmitted from the communication interface 1005 to an external device.

  In the decompression process, the moving image code is read into the memory 1001 from the hard disk device 1002, for example, or received from the communication interface 1005 via the network and read into the memory 1001. Each process of entropy decoding, inverse quantization, intra-frame two-dimensional inverse wavelet transform, and time direction inverse wavelet transform inverse color transform including coefficient correction is performed on this code. The restored moving image frame data is stored in the hard disk device 1002 or displayed on the display device 1004.

  Next, the processing of the coefficient correction unit 3 and the time direction inverse wavelet transform unit 31 during the expansion process will be further described with reference to five examples. For convenience of explanation, it is assumed that the wavelet transform in the time direction is performed up to two layers (decomposition level 2) unless otherwise specified.

(Example 1 of Embodiment 2)
The processing flow of the coefficient correction and the time-direction inverse wavelet transform in Example 1 according to the present embodiment is as shown in FIG. 22, and since this has been described in relation to Example 1 according to Embodiment 1, The description will not be repeated.

  The contents of this processing will be described using the time direction wavelet coefficient data schematically shown in FIG. Here, JPEG2000 5 × 3 conversion is used as wavelet conversion, but 9 × 7 conversion is the same except that the coefficient correction formula is different.

  First, the coefficients 2L and 2H of the decomposition level 2 in the time direction are interleaved (2L, 2H, and 2H are rearranged in the order of 2L, 2H, 2L, 2H,...). Regarding the interleaved coefficients 2L and 2H, the high-pass coefficient (f62H (t6)) adjacent to the GOF boundary indicated by the dotted line in FIG. 40B and the high-pass coefficient (f102H (t10)) not adjacent to the boundary near the GOF boundary Correct. The high pass coefficient correction formula is as shown in FIG.

The correction formula for 9x7 conversion is
f62H (t6) = (-0.0535xf22H (t2) + 0.15644xf22H (t2) + 0.09127xf02L (t0)
-0.59127xf42L (t4) + 0.591272xf82L (t8) -0.09127xf102L (t10)) / 0.60295
f102H (t10) = (0.05754xf42L (t4) -0.05754xf82L (t8) + 0.03372xf142H (t14)) / 0.53372
It becomes.

  The inverse wavelet transform in the time direction is performed on the wavelet coefficient at the decomposition level 2 after such coefficient correction. In the case of the wavelet coefficient at the decomposition level 2 in FIG. 40B, the coefficient (1L) at the decomposition level 1 as shown in FIG. 40C is generated by the inverse wavelet transform.

  Next, coefficient correction and inverse wavelet transform are performed on the wavelet coefficients of the decomposition level 1 in the time direction. FIG. 41 is a diagram for explaining this. Interleaves the coefficients 1L and 1H of the decomposition level 1 shown in FIG. 41 (a), and adjoins the high-pass coefficient (f71H (t7)) adjacent to the GOF boundary shown in FIG. 41 (b) and the boundary near the GOF boundary. The high pass coefficient (f91H (t9)) not to be corrected is corrected. The coefficient correction formula is as shown in FIG. In the case of 5 × 3 conversion, the high-pass coefficient (f91H (t9)) that is not adjacent to the GOF boundary at decomposition level 1 is not affected by mirroring and is not corrected. However, in the 9 × 7 conversion, the high-pass coefficient (f91H (t9)) that is not adjacent to the GOF boundary is also affected by mirroring, and is corrected.

The correction formula for 9x7 conversion is
f71H (t7) = (-0.0535xf31H (t3) + 0.15644xf51H (t5) + 0.09127xf41L (t4)
-0.59127xf61L (t6) + 0.591272xf91L (t9) -0.09127xf111L (t11)) / 0.60295
f91H (t9) = (0.05754xf61L (t6) -0.05754xf81L (t8) + 0.03372xf111H (t11)) / 0.53372
It becomes.

  The inverse wavelet transform in the time direction is performed on the wavelet coefficient of the composition level 1 after such coefficient correction. In the case of the wavelet coefficient at the decomposition level 1 in FIG. 41B, the inverse wavelet transform returns to the data at the decomposition level 0 as shown in FIG. 41C, that is, the frame data.

  The correction formula is an example, and other correction formulas may be used as long as they contribute to the suppression of GOF boundary distortion in a moving image. In this embodiment, the high-pass coefficient not adjacent to the GOF boundary near the GOF boundary is also subject to correction. However, the effect of suppressing the GOF boundary distortion by the correction is compared with the effect of correcting the high-pass coefficient adjacent to the GOF boundary. Since it is small, it is possible to exclude a high-pass coefficient that is not adjacent to the GOF boundary from the correction target.

  Also, the motion amount of the frame near the GOF boundary is estimated, and when the motion amount is smaller than a predetermined value, coefficient correction for suppressing the GOF boundary distortion is performed, and when the motion amount is larger than the predetermined value, the GOF boundary distortion is suppressed. Control may be performed so as not to perform coefficient correction. A known method may be used to estimate the amount of motion of the frame.

Further, the degree of correction for the coefficient near the GOG boundary may be controlled stepwise or continuously in accordance with the amount of motion of the frame near the GOF boundary. For example, when the motion amount of the frame near the GOF boundary is k, the coefficient value before correction is x0, and the coefficient value after correction is x1, the coefficient value considering the motion amount of the frame near the GOF boundary Where x is
x = (1-k) x0 + k x1
(The amount of motion k is used as a parameter for linear interpolation).

  The above-described deformation modes are the same for the examples described later.

(Example 2 according to Embodiment 2)
The processing flow of coefficient correction and time direction inverse wavelet transform in Example 2 according to the present embodiment is as shown in FIG. That is, in this example, as in Example 2 according to the first embodiment, the coefficient correction unit 3 performs clipping processing on the correction value of the high-pass coefficient.

(Example 3 according to Embodiment 2)
The processing flow of coefficient correction and time-direction inverse wavelet transform in Example 3 according to the present embodiment is as shown in FIG. That is, in this example, as in Example 3 according to the first embodiment, it is determined whether the number of quantization steps exceeds a predetermined threshold thQ, and when the number of quantization steps exceeds the threshold thQ. Only coefficient correction is performed.

(Example 4 according to Embodiment 2)
The processing flow of coefficient correction and time-direction inverse wavelet transform in Example 4 according to the present embodiment is as shown in FIG. That is, in this example, as in Example 4 according to the first embodiment, it is determined whether the decomposition level exceeds a predetermined threshold value thlev, and coefficient correction is performed only for the decomposition level equal to or lower than the threshold value thlev. Do.

  Also in this embodiment, the coefficient correction may be controlled according to the number of quantization steps, and this aspect is also included in this embodiment.

(Example 5 according to Embodiment 2)
The processing flow of the coefficient correction and the time direction inverse wavelet transform in Example 5 according to the present embodiment is as shown in FIG. That is, in this example, as in Example 5 according to the first embodiment, the presence / absence of a coefficient correction instruction for suppressing GOF boundary distortion from the user is determined, and only when there is a coefficient correction instruction, the GOF boundary is determined. Coefficient correction for distortion suppression is performed.

  The method for giving an instruction by the user has been described in connection with Example 5 according to the first embodiment, and therefore the description thereof will not be repeated. Also in the present embodiment, “present” and “none” of the GOF boundary suppression function can be selected, and when “present” is selected, “high quality mode (or high image quality mode)” and “high speed mode” are selected. It may be configured to be able to. In this case, for example, only the high-pass coefficient adjacent to the GOF boundary is corrected in the high-speed mode, and both the high-pass coefficient adjacent to the GOF boundary and the high-pass coefficient not adjacent to the GOF boundary near the GOF boundary are corrected in the high-quality mode. It may be configured as follows.

  Also in the present embodiment, the coefficient correction may be controlled according to the decomposition level and / or the number of quantization steps as in the third and fourth embodiments, and this aspect is also included in the present embodiment. The

  Although the image processing apparatus according to the present invention has been described above, it is obvious that the above description is also the description of the image processing method according to the present invention. Therefore, the description of the image processing method according to the present invention will not be repeated.

  In the above description, wavelet transformation is used as frequency transformation in the time direction, but the present invention can also be applied to cases where frequency transformation other than wavelet transformation is used. In the above description, the intra-frame wavelet transform (generally frequency transform) is performed in two dimensions. However, the present invention can also be applied to the case of performing in one dimension. Furthermore, the present invention can also be applied to the case where only the time direction wavelet transform (generally frequency conversion) in units of GOF is performed without performing the intra-frame wavelet transform (generally frequency conversion).

It is a figure which shows the original image and coordinate system for demonstrating wavelet transformation. It is a figure which shows the coefficient arrangement | sequence after the forward wavelet transform to a perpendicular direction. It is a figure which shows the coefficient arrangement | sequence after the forward wavelet transform to a horizontal direction. It is a figure which shows the coefficient arrangement | sequence in the state of deinterleaving of the decomposition level 1. It is a figure which shows the coefficient arrangement | sequence in the state of deinterleaving of the decomposition level 2. It is explanatory drawing of mirroring. It is a figure which shows the coefficient of the state of the interleaved state of the decomposition level 2 about 4 tiles. It is a figure which shows the high-pass coefficient which should be correct | amended for horizontal detiling in the decomposition level 2. FIG. It is a figure which shows the coefficient after coefficient correction | amendment and a horizontal direction reverse wavelet transform. It is a figure which shows the high pass coefficient which should be correct | amended for the vertical detiling in the decomposition level 2. FIG. FIG. 6 is a diagram illustrating a coefficient (deinterleaved state) after coefficient correction and vertical inverse wavelet transform. It is a figure which shows the coefficient of the decomposition level 1 interleaved. It is a figure which shows the coefficient which should be corrected for a horizontal direction detiling in the decomposition level 1. FIG. It is a figure which shows the coefficient which should be corrected for a vertical direction detiling in the decomposition level 1. FIG. 1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. 3 is a block diagram illustrating a functional configuration of a three-dimensional wavelet transform / inverse transform unit during compression processing according to Embodiment 1. FIG. It is a schematic diagram which shows the frame data and GOF of a moving image. It is a schematic diagram which shows the frame structure after the in-frame two-dimensional order wavelet transform. FIG. 5 is a schematic diagram illustrating time direction forward wavelet transform in units of GOF in the first embodiment. 6 is a flowchart illustrating a processing flow of forward wavelet transform in the first embodiment. 3 is a block diagram illustrating a functional configuration of a three-dimensional wavelet transform / inverse transform unit during decompression processing according to Embodiment 1. FIG. It is a flowchart which shows the processing flow of the coefficient correction | amendment and time direction reverse wavelet transform in Example 1 which concerns on Embodiment 1,2. 6 is a flowchart illustrating a processing flow of intra-frame two-dimensional forward wavelet transform in the first and second embodiments. FIG. 6 is a schematic diagram for explaining the coefficient correction of the decomposition level 2 and the time-direction inverse wavelet transform in the first embodiment. FIG. 5 is a schematic diagram for explaining coefficient correction at a decomposition level 1 and time-direction inverse wavelet transform in the first embodiment. It is a figure which shows the coefficient correction type | formula in the decomposition level 2. FIG. It is a figure which shows the coefficient correction formula in the decompression level 1. It is a flowchart which shows the processing flow of the coefficient correction | amendment and time direction reverse wavelet transform in Example 2 which concerns on Embodiment 1,2. It is explanatory drawing of clipping. It is explanatory drawing of clipping. It is explanatory drawing of clipping. It is a flowchart which shows the processing flow of the coefficient correction | amendment and time direction reverse wavelet transform in Example 3 which concerns on Embodiment 1,2. It is a flowchart which shows the processing flow of the coefficient correction | amendment and time direction reverse wavelet transform in Example 4 which concerns on Embodiment 1,2. It is a flowchart which shows the processing flow of the coefficient correction | amendment and time direction reverse wavelet transform in Example 5 which concerns on Embodiment 1,2. It is a figure for demonstrating the method of designating the presence or absence of GOF boundary distortion suppression by a user. 10 is a block diagram illustrating a functional configuration of a three-dimensional wavelet transform / inverse transform unit during compression processing in Embodiment 2. FIG. 10 is a flowchart illustrating a processing flow of forward wavelet transform in the second embodiment. FIG. 10 is a schematic diagram illustrating time direction forward wavelet transform in units of GOF in the second embodiment. 10 is a block diagram illustrating a functional configuration of a three-dimensional wavelet transform / inverse transform unit during decompression processing in Embodiment 2. FIG. FIG. 10 is a schematic diagram for explaining coefficient correction at a decomposition level 2 and time-direction inverse wavelet transform in the second embodiment. FIG. 10 is a schematic diagram for explaining coefficient correction at a decomposition level 2 and time-direction inverse wavelet transform in the second embodiment. It is a figure which shows the coefficient correction type | formula in the decomposition level 2. FIG. It is a figure which shows the coefficient correction formula in the decompression level 1. It is a block diagram which shows the structural example of the computer which can be utilized for realization of the image processing apparatus which concerns on this invention.

Explanation of symbols

1 color conversion / inverse color conversion unit 2 3D wavelet conversion / inverse conversion unit 3 coefficient correction means 4 quantization / inverse quantization unit 5 entropy encoding / decoding unit 6 code formation / tag processing unit 21 2D order in frame Wavelet transformation means 22 Time direction forward wavelet transformation means 31 Time direction inverse wavelet transformation means 32 Intra-frame two-dimensional inverse wavelet transformation means

Claims (16)

Inverse frequency conversion means for performing reverse frequency conversion in the time direction on a frequency coefficient generated by performing frequency conversion in the time direction for each group of moving image frames (hereinafter referred to as GOF);
The high frequency coefficient adjacent to the GOF boundary in the frequency coefficient subjected to inverse frequency conversion by the inverse frequency conversion means is low when the inverse frequency conversion value centered on the high frequency coefficient is not subjected to GOF division prior to the inverse frequency conversion. Coefficient correction means for correcting to be equal to the inverse frequency conversion value using only the frequency coefficient ;
An image processing apparatus comprising: Inverse frequency conversion means for performing reverse frequency conversion in the time direction on the frequency coefficient generated by hierarchical frequency conversion in the time direction for each group of moving image frames (hereinafter referred to as GOF);
Prior to the inverse frequency conversion, the inverse frequency conversion value centered on the high frequency coefficient is subjected to GOF division before the inverse frequency conversion of the high frequency coefficient adjacent to the GOF boundary in the frequency coefficient below the specific layer to be inverse frequency converted by the inverse frequency conversion means. Coefficient correction means for correcting to be equal to the inverse frequency conversion value using only the low frequency coefficient when not
An image processing apparatus comprising: The frequency transform is a 5 × 3 wavelet transform, and the inverse frequency transform is a 5 × 3 wavelet inverse transform;
  The coefficient correction means is a sequence of frequency coefficients in the vicinity of the interleaved GOF boundary before being subjected to inverse frequency conversion by the inverse frequency conversion means.
  L1 H1 L2 H2 L3 H3 L4 H4
(However, L1, L2, L3, and L4 are low frequency coefficients, H1, H2, H3, and H4 are high frequency coefficients, and there is a boundary between two GOFs temporally continuous between H2 and L3, and L1 , H1, L2, and H2 belong to the GOF on the earlier side in time, and L3, H3, L4, and H4 belong to the GOF on the later side in time), the correction value of the high-frequency coefficient H2 adjacent to the GOF boundary
  H2 = (1/3) H1- (2/3) L2 + (2/3) L3
The image processing apparatus according to claim 1, wherein the image processing apparatus is calculated by: The frequency transform is a 9 × 7 wavelet transform, and the inverse frequency transform is a 9 × 7 wavelet inverse transform;
  The coefficient correction means is a sequence of frequency coefficients in the vicinity of the interleaved GOF boundary before being subjected to inverse frequency conversion by the inverse frequency conversion means.
  H0 L1 H1 L2 H2 L3 H3 L4 H4
(However, L1, L2, L3, and L4 are low frequency coefficients, H0, H1, H2, H3, and H4 are high frequency coefficients, and there are two temporally continuous GOF boundaries between H2 and L3. H0, L1, H1, L2, and H2 belong to the GOF on the earlier side in time, and L3, H3, L4, and H4 belong to the GOF on the later side in time), and the high-frequency coefficient H2 adjacent to the GOF boundary Correction value
  H2 = (-0.0535H0 + 0.15644H1 + 0.09127L1-0.59127L2 + 0.59127L3-0.09127L4) /0.60295
The image processing apparatus according to claim 1, wherein the image processing apparatus is calculated by: Correction for the high frequency coefficients, the image processing of any one of claims 1 to 4, characterized in that the number of quantization steps is performed only if less than a predetermined value adjacent to the GOF boundary by said coefficient correction means apparatus. 6. The correction of the high frequency coefficient adjacent to the GOF boundary by the coefficient correction means is performed only when GOF boundary distortion suppression is designated by the user. Image processing device. When the correction value of the high-frequency coefficient adjacent to the GOF boundary protrudes from the quantization section to which the value before correction of the high-frequency coefficient belongs , the coefficient correction unit is configured to detect the quantum near the correction value of the high-frequency coefficient. The image processing apparatus according to claim 1, wherein clipping processing for correcting the correction value of the high-frequency coefficient is performed to a value at an end of the conversion interval . A reverse frequency conversion step of performing reverse frequency conversion in the time direction on a frequency coefficient generated by performing frequency conversion in the time direction for each group of moving image frames (hereinafter referred to as GOF);
Prior to the inverse frequency conversion , the high frequency coefficient adjacent to the GOF in the frequency coefficient subjected to the inverse frequency conversion in the inverse frequency conversion step is a low frequency when the inverse frequency conversion value centering on the high frequency coefficient is not subjected to the GOF division. A coefficient correction step for correcting to be equal to the inverse frequency conversion value using only the coefficient;
An image processing method comprising: A reverse frequency conversion step of performing reverse frequency conversion in the time direction on a frequency coefficient generated by hierarchical frequency conversion in the time direction for each group of moving image frames (hereinafter referred to as GOF);
Prior to the inverse frequency conversion, the inverse frequency conversion value centered on the high frequency coefficient is divided into GOF divisions before the inverse frequency conversion of the high frequency coefficient adjacent to the GOF boundary in the frequency coefficient below the specific layer that is inverse frequency converted by the inverse frequency conversion step. A coefficient correction step for correcting to be equal to the inverse frequency conversion value using only the low frequency coefficient when not ,
An image processing method comprising: The frequency transform is a 5 × 3 wavelet transform, and the inverse frequency transform is a 5 × 3 wavelet inverse transform;
  The coefficient correction step includes a sequence of frequency coefficients in the vicinity of the interleaved GOF boundary before the reverse frequency conversion is performed by the reverse frequency conversion step.
  L1 H1 L2 H2 L3 H3 L4 H4
(However, L1, L2, L3, and L4 are low frequency coefficients, H1, H2, H3, and H4 are high frequency coefficients, and there is a boundary between two GOFs temporally continuous between H2 and L3, and L1 , H1, L2, and H2 belong to the GOF on the earlier side in time, and L3, H3, L4, and H4 belong to the GOF on the later side in time), the correction value of the high-frequency coefficient H2 adjacent to the GOF boundary
  H2 = (1/3) H1- (2/3) L2 + (2/3) L3
The image processing method according to claim 8, wherein the image processing method is calculated by: The frequency transform is a 9 × 7 wavelet transform, and the inverse frequency transform is a 9 × 7 wavelet inverse transform;
  The coefficient correction step includes a sequence of frequency coefficients in the vicinity of the interleaved GOF boundary before the reverse frequency conversion is performed by the reverse frequency conversion step.
  H0 L1 H1 L2 H2 L3 H3 L4 H4
(However, L1, L2, L3, and L4 are low frequency coefficients, H0, H1, H2, H3, and H4 are high frequency coefficients, and there are two temporally continuous GOF boundaries between H2 and L3. H0, L1, H1, L2, and H2 belong to the GOF on the earlier side in time, and L3, H3, L4, and H4 belong to the GOF on the later side in time), and the high-frequency coefficient H2 adjacent to the GOF boundary Correction value
  H2 = (-0.0535H0 + 0.15644H1 + 0.09127L1-0.59127L2 + 0.59127L3-0.09127L4) /0.60295
The image processing method according to claim 8, wherein the image processing method is calculated by: 12. The image processing according to claim 8, wherein the correction for the high frequency coefficient adjacent to the GOF boundary by the coefficient correction step is performed only when the number of quantization steps is smaller than a predetermined value. Method. The correction for the high-frequency coefficient adjacent to the GOF boundary by the coefficient correction step is performed only when GOF boundary distortion suppression is specified by the user. Image processing method. In the coefficient correction step, when the correction value of the high-frequency coefficient adjacent to the GOF boundary protrudes from the quantization section to which the value before the correction of the high-frequency coefficient belonged, the coefficient on the side close to the correction value of the high-frequency coefficient The image processing method according to claim 8, wherein clipping processing for correcting the correction value of the high-frequency coefficient is performed on the end value of the conversion interval . A program that causes a computer to function as an inverse frequency conversion unit and a coefficient correction unit of an image processing apparatus according to any one of claims 1 to 7. 8. A computer-readable information recording medium in which a program for causing a computer to function as an inverse frequency conversion unit and a coefficient correction unit of the image processing apparatus according to claim 1 is recorded.

Images