JP4219303B2 - Encoding apparatus, encoding control method, program, and recording medium - Google Patents

Description

  The present invention relates to encoding of moving images, and more particularly to frame-based encoding of interlaced images.

  In a moving image photographing device such as a video camera, generally, a moving image is photographed by interlace scanning at intervals of 1/60 seconds. As a method of encoding moving image data shot by such a video camera, a non-interlaced image synthesized from field-based encoding in which individual interlaced images (fields) are directly encoded and two adjacent interlaced images is used. There is frame-based encoding that encodes (frame).

  Since the scanning lines are skipped in the image of each field, the pixel correlation in the vertical direction is weaker than the image of the frame in which the scanning lines are continuous. Therefore, frame-based encoding is generally advantageous in terms of compression efficiency. However, the frame-based coding has a problem that the compression efficiency is lowered when the amount of motion between fields constituting the frame is large, and an unnatural image quality deterioration is likely to occur.

  In order to solve the above-described problem of reduction in compression efficiency, Patent Document 1 discloses an invention of an encoding device that adaptively switches between frame encoding and field encoding on a block basis. In the encoding apparatus according to the present invention, a non-interlaced image of a frame is synthesized from two consecutive interlaced images, divided into blocks, and a vertical pixel correlation (1) for two blocks arranged in the vertical direction. ) Further, these two blocks are reconverted into an interlaced image, and the same pixel correlation (2) is examined for the image. Then, the strengths of the pixel correlation (1) and the pixel correlation (2) are compared. When the pixel correlation (1) is stronger, a non-interlaced image of 2 blocks is encoded. If it is stronger, a 2-block interlaced image is encoded. Information for identifying which method is used for coding is added to the code of each block.

  Next, another problem will be described. (A) in FIG. 1 is an interlaced image of the (n) field, (b) is an interlaced image of the (n + 1) field after 1/60 seconds, and (c) is a composite of the interlaced images of these two fields. Examples of non-interlaced images of the obtained frames are shown. When the subject moves to the right as shown in the drawing between two yields, the left and right edge portions of the subject are shifted in a comb shape by a plurality of pixels for each scanning line. (D) is an enlargement of the comb-shaped edge portion, and L corresponds to the amount of motion between fields.

  In the frame-based encoding, the “comb” is a cause of lowering the encoding efficiency and a cause of peculiar unnatural image quality degradation.

  This unnatural image quality degradation will be further described with reference to FIG. When the vertical line that was in the position shown in FIG. 2A in the (n) field has moved to the position shown in (b) in the next (n + 1) field, on the frame synthesized from both fields, The vertical lines are two dotted lines as shown in (c). When this frame is decoded after frame-based encoding, due to the influence of quantization (including truncation described later) at the time of encoding, on the decoded frame, as shown in FIG. It seems that the space is connected with a pale color. Therefore, in each field decomposed from the decoded frame, two dotted lines as shown in (e) and (f) appear. As a result, when the decoded frame is field-decomposed and interlaced on a television receiver or the like, an original vertical line and a vertical line such as an “afterimage” appear in each field, which is originally a single vertical line. Will look like two vertical lines. Although FIG. 2 shows an example in which the vertical line is moved, when the surface is moved instead of the line, the surface appears to be doubled or to the left and right. Such a phenomenon may be referred to as an “afterimage phenomenon” in this specification.

  An invention related to frame-based encoding using two-dimensional wavelet transform is described in Patent Document 2. The outline is that in order to control the image quality for each block area of the image, the amount of motion is detected by inter-frame difference, and the weighting factor determined according to the detected amount of motion and the type of subband is used as a quantized wavelet. Multiply by a coefficient.

JP 2002-64830 A JP 2001-326936 JP Wenjun Zeng, Scott Daly and Shawmin Lei, "VISUAL OPTIMIZATION TOOLS IN JPEG 2000", The 2000 International Conference on Image Processing (ICIP-2000) J. Katto and Y. Yasuda, “Performance evaluation of subband coding and optimization of its filter coefficients,” Journal of Visual Communication and Image Representation, vol. 2, Dec. 1991, pp. 303-313. Marcus J. Nadenau and Julien Reichel, "Opponent color, human vision and wavelets for image compression", Proceedings of the Seventh Color Imaging Conference, Scottsdale, Arizona, November 16-19, 1999, IS & T, pp. 237-242 Marcus J. Nadenau, Julien Reichel, and Murat Kunt, "Wavelet-based color image compression: Exploiting the contrast sensitivity function", IEEE Transactions on Image Processing, 2000 D.S.Taubman and M.W.Marcellin, "JPEG 2000: Image Compression Fundamentals, Standards and Practice", Kluwer, 2002

  Therefore, an object of the present invention is to suppress the above-mentioned unnatural image quality degradation and achieve an improvement in image quality in frame-based encoding of interlaced images.

  The applicant has applied for patents related to the present invention in Japanese Patent Application Nos. 2002-289807, 2002-300468, 2002-300476, and 2002-360809. The outline of the invention relating to each patent application is as follows.

  Japanese Patent Application No. 2002-289807: In frame-based encoding of an interlaced image (field), two-dimensional wavelet transform of the frame (non-interlaced) image is performed. Focusing on the fact that the comb-like effect due to the motion between fields appears strongly in the 1LH subband coefficient, the motion amount (movement) between the fields is based on the coefficient value of at least 1LH subband or the code amount of at least 1LH subband coefficient. Speed).

  Japanese Patent Application No. 2002-300468: In frame-based encoding of interlaced images, two-dimensional wavelet transform of frame images is performed. Even when the amount of motion is large but the moving object is small, in order to accurately determine the amount of motion between fields, the image is divided into sub-blocks (for example, code blocks), and the coefficient value of at least 1 LH sub-band of each sub-block Alternatively, the amount of motion between fields of the entire image is determined based on the code amount.

  Japanese Patent Application No. 2002-300476: In frame-based encoding of interlaced images, two-dimensional wavelet transform of frame images is performed. The image is divided into sub-blocks, and the amount of motion between fields is determined for each sub-block unit based on the coefficient value or code amount of at least 1 LH sub-band. If the camera that shoots the video is stationary, there are moving parts (and moving parts at high speed and moving parts at low speed) and non-moving parts in the image. The amount of movement of the part can be determined.

  Japanese Patent Application No. 2002-360809: In frame-based encoding of interlaced images, frequency coefficient quantization obtained by frequency conversion (for example, wavelet conversion) of frame images, frequency coefficient truncation after quantization, or code truncation By controlling according to the amount of motion between fields, unnatural image quality deterioration is suppressed.

In general, image encoding (compression)
(A) Conversion of image data into frequency domain coefficients → Quantization of coefficients for each frequency → Entropy coding of coefficients after quantization or
(B) Conversion of image data into frequency domain coefficients → Quantization of coefficients for each frequency → Entropy only the final required part (eg, required bit plane or sub bit plane) of the quantized coefficient Encoding or
(C) Conversion of image data into frequency domain coefficients → Quantization of coefficients for each frequency → Entropy coding of coefficients after quantization → Discarding unnecessary entropy codes (truncation)
The processing flow is often taken. The truncation of the entropy code in the processing flow (C) is also called post-quantization. In the processing flow (B), it is equivalent to discarding the bit plane or sub bit plane of the coefficient that is not the target of entropy coding. That is, truncation is performed in the coefficient state.

  In the present invention, in order to suppress unnatural image quality degradation such as the above-mentioned “afterimage phenomenon” and to obtain good image quality, a flow of frame-based encoding of an interlaced image (for example, (A ) (B) (Flow as in (C)) Optimal quantization or truncation control is performed so as to leave a comb shape due to movement between fields.

  As described in Non-Patent Document 1, it is known that an optimal method of quantization and truncation depends on a quantization error and a compression rate (although the compression rate and the quantization error are different). A similar effect on the optimal method).

  In general, when the quantization error and compression ratio are small and image quality degradation is difficult to understand, quantization using weights based on specific visual sensitivity (human contrast sensitivity to the frequency described later) determined by the observation distance is performed. preferable. However, when the compression ratio and quantization error become large and image quality deterioration becomes conspicuous, it is necessary to use a weight different from the weight in order to obtain a preferable image quality. This is thought to be because human sensitivity to errors at each frequency is a function of the error amount itself. In general, it is said that it is better to refrain from quantizing low frequency coefficients as the error amount itself is larger, and to quantize more high frequency coefficients, and in coding using wavelet transform, It can be easily confirmed by experiment. (As mentioned above, the quantization error and the compression rate are different, but usually the compression rate increases when the error amount itself is large. Can be replaced with a compression ratio).

  Therefore, in frame-based encoding of an interlaced image, it is effective to perform quantization or truncation that reflects both the amount of motion between fields constituting a frame and the compression rate or quantization error.

Hereinafter, features of the present invention will be sequentially described.
According to the first aspect of the present invention, an encoding processing means for performing frame-based encoding processing of an interlaced image and a motion amount between fields constituting the frame are detected, and based on the motion amount and a compression rate or a quantization error. And an encoding control means for controlling quantization of frequency coefficients in the encoding processing means ,
Using wavelet transform in the frame-based encoding process,
As the amount of movement between the fields,
1LHmean, 1HL subband coefficient absolute value 1HLmean, 2LH subband coefficient absolute value 2LHmean, 2HL subband coefficient absolute value Average value of 2HLmean
(1LHmean / 1HLmean) / (2LHmean / 2HLmean) ratio, or
Regarding the luminance component, when the entropy coding of 1LH, 1HL, 2LH, and 2HL subbands is performed, the respective code amounts of 1LHcode, 1HLcode, 2LHcode, and 2HLcode
Ratio of (1LHcode / 1HLcode) / (2LHcode / 2HLcode)
It is a coding apparatus characterized by having a means to calculate.
According to this configuration, appropriate quantization is performed according to the amount of motion between fields and the compression rate or quantization error, and even when there is motion between fields, unnatural image quality degradation is suppressed, and at the same time good Image quality can be obtained.

According to a second aspect of the present invention, an encoding processing means for performing frame-based encoding processing of an interlaced image and a motion amount between fields constituting the frame are detected, and based on the motion amount and a compression rate or a quantization error. An encoding control means for controlling frequency coefficient or code truncation in the encoding processing means ,
Using wavelet transform in the frame-based encoding process,
As the amount of movement between the fields,
1LHmean, 1HL subband coefficient absolute value 1HLmean, 2LH subband coefficient absolute value 2LHmean, 2HL subband coefficient absolute value Average value of 2HLmean
(1LHmean / 1HLmean) / (2LHmean / 2HLmean) ratio, or
Regarding the luminance component, when the entropy coding of 1LH, 1HL, 2LH, and 2HL subbands is performed, the respective code amounts of 1LHcode, 1HLcode, 2LHcode, and 2HLcode
Ratio of (1LHcode / 1HLcode) / (2LHcode / 2HLcode)
It is a coding apparatus characterized by having a means to calculate.
According to such a configuration, truncation of an appropriate frequency coefficient (wavelet coefficient) or code corresponding to the amount of motion between fields and the compression rate or quantization error is performed, and it is unnatural even when there is motion between fields. It is possible to suppress image quality deterioration and obtain good image quality at the same time.

  Quantization and truncation are performed for image quality control. However, when it is desired to perform the most precise control, it is appropriate to control quantization and truncation for each applicable minimum unit. In the present specification, this minimum unit is referred to as an image quality control unit.

  The image quality control unit is a frequency coefficient itself (for example, DCT coefficient for JPEG, wavelet transform coefficient for JPEG2000), a rectangular block of a predetermined size (for example, DCT coefficient for JPEG, code block for JPEG2000), and the like.

Therefore, one aspect of the present invention is characterized in that, in the encoding apparatus according to the first aspect of the present invention , the encoding control means controls the quantization for each image quality control unit.
By controlling quantization for each image quality control, more precise and optimum image quality control is possible.

  Note that according to the invention of Japanese Patent Application No. 2002-300476, it is possible to detect the inter-field motion amount (comb amount) in such an image quality control unit.

According to an aspect of the present invention, in the encoding device according to the second aspect of the present invention , the encoding control means controls truncation for each image quality control unit.
By controlling truncation for each image quality control unit, more precise and optimum image quality control is possible.

  On the other hand, when it is desired to perform image quality control at high speed and simply or when there are implementation restrictions, quantization or truncation control is performed for each unit larger than the image quality control unit, for example, for each frequency band of the frequency conversion coefficient. It is convenient to do.

Therefore, one aspect of the present invention is characterized in that, in the encoding device of the invention of claim 1, the encoding control means controls the quantization for each frequency band of the frequency coefficient.
Thus, if the unit of quantization control is a frequency band, the quantization control is simplified, and the image quality control can be speeded up.
In this case, the motion amount detection between fields may be performed on the entire image. Such detection can be performed by the invention according to Japanese Patent Application No. 2002-289807 or Japanese Patent Application No. 2002-300468.

According to another aspect of the present invention, in the encoding device according to the second aspect of the present invention , the encoding control means controls truncation for each frequency band of the frequency coefficient.
Thus, if the truncation control unit is a frequency band, truncation control is simplified and image quality control can be speeded up.
In this case, the motion amount detection between fields may be performed on the entire image.

  In recent years, the use of wavelet transform (DWT: discrete wavelet fret transform) is increasing as a frequency transform in place of DCT (discrete cosine transform) employed in the JPEG (Joint Photographic Coding Experts Group). A representative example is JPEG2000, an image compression / decompression method succeeding JPEG, which became an international standard in 2001.

  In the case of DCT, each DCT coefficient has a frequency band. In the case of wavelet transform, the frequency band of the frequency coefficient means each decomposition level, and the smaller the decomposition level, the higher the frequency.

  The above-mentioned “quantization or truncation that leaves a comb shape” means to refrain from quantization or truncation of high frequency band coefficients. This is because, as is apparent from FIG. 2, the comb shape occurs because the pixel value changes greatly for each pixel, and takes the highest frequency.

  Here, the procedure of wavelet transform and the definition of the composition level, resolution level, and subband will be described.

  JPEG2000 employs wavelet transformation called 5 × 3 transformation and wavelet transformation called 9 × 7 transformation. Each forward conversion formula and reverse conversion formula are shown below.

5 × 3 conversion (forward conversion)
C (2i + 1) = P (2i + 1) −floor ((P (2i) + P (2i + 2)) / 2) [step1]
C (2i) = P (2i) + floor (((C (2i-1) + C (2i + 1) +2) / 4) | [step2] (1)
(Inverse transformation)
P (2i) = C (2i) −floor ((C (2i-1) + C (2i + 1) +2) / 4) [step1]
P (2i + 1) = C (2i + 1) + floor ((P (2i) + P (2i + 2)) / 2) [step2] (2)
Here, floor (x) represents a floor function of x (a function that replaces the real number x with an integer that does not exceed x and is closest to x).

9x7 conversion (forward conversion)
C (2n + 1) = P (2n + 1) + α * (P (2n) + P (2n + 2)) [step1]
C (2n) = P (2n) + β * (C (2n-1) + C (2n + 1)) [step2]
C (2n + 1) = C (2n + 1) + γ * (C (2n) + C (2n + 2)) [step3]
C (2n) = C (2n) + δ * (C (2n-1) + C (2n + 1)) [step4]
C (2n + 1) = K * C (2n + 1) [step5]
C (2n) = (1 / K) * C (2n) [step6] (3)
(Inverse transformation)
P (2n) = K * C (2n) [step1]
P (2n + 1) = (1 / K) * C (2n + 1) [step2]
P (2n) = X (2n) -δ * (P (2n-1) + P (2n + 1)) [step3]
P (2n + 1) = P (2n + 1) -γ * (P (2n) + P (2n + 2)) [step4]
P (2n) = P (2n) -β * (P (2n-1) + P (2n + 2)) [step5]
P (2n) = P (2n + 1) -α * (P (2n) + P (2n + 2)) [step6] (4)
However, α = -1.586134342059924
β = -0.052980118572961
γ = 0.882911075530934
δ = 0.443506852043971
K = 1.230174104914001

  A process of performing wavelet transform called 5 × 3 transform in two dimensions (vertical direction and horizontal direction) on a monochrome image of 16 × 16 pixels will be described with reference to FIGS.

  As shown in FIG. 3, the XY coordinates are taken, and the pixel value of a pixel whose Y coordinate is y is expressed as P (y) (0 ≦ y ≦ 15) for a certain X coordinate. In JPEG2000, a coefficient C (2i + 1) is obtained by first applying a high-pass filter in the vertical direction (Y-coordinate direction) around an odd-numbered pixel (y = 2i + 1). Next, a low-pass filter is applied around the pixel whose Y coordinate is an even number (y = 2i) to obtain a coefficient C (2i) (this is performed for all X coordinates). Step 1 in the forward conversion (1) represents a high-pass filter, and step 2 represents a low-pass filter.

  Note that there may be no adjacent pixel at the center of the image at the edge of the image. In this case, the pixel value is supplemented by a predetermined rule. It will not be described in detail.

  For the sake of simplicity, if the coefficient obtained by the high-pass filter is denoted by H and the coefficient obtained by the low-pass filter is denoted by L, the image in FIG. 3 is arranged as shown in FIG. Is converted to

  Subsequently, a high-pass filter is applied to the coefficient array shown in FIG. 3 in the horizontal direction centering on an odd-numbered (x = 2i + 1) coefficient, and then an even-numbered (x = 2i) coefficient. (This is performed for all y. In this case, P (2i) etc. in the equation (1) is read as a coefficient value).

  For simplicity, LL is obtained by applying a low-pass filter centered on the L coefficient, HL is obtained by applying a high-pass filter centered on the L coefficient, and is obtained by applying a low-pass filter centered on the H coefficient. 4 is converted into a coefficient array as shown in FIG. 5, if the coefficient obtained is expressed as LH and the coefficient obtained by applying a high-pass filter around the H coefficient as HH. Here, the coefficient group with the same symbol is called a subband, and FIG. 5 includes four subbands.

  With the above, one wavelet transform (one decomposition (decomposition)) is completed, and only the LL coefficients are collected (collected for each subband as shown in FIG. 6 and only the LL subband is taken out). An “image” having half the resolution of the image is obtained (in this way, classification for each subband is referred to as deinterleaving, and arrangement in the state shown in FIG. 5 is referred to as interleaving).

In the second wavelet transform, the LL subband may be regarded as an original image and the same transformation as described above may be performed. When the second wavelet transform is performed and the coefficients are rearranged, a schematic diagram 7 is obtained.
Here, in FIGS. 6 and 7, the prefixes 1 and 2 of the coefficient indicate how many times the wavelet transform has been obtained, and are called the decomposition level. In the wavelet transform, this decomposition level corresponds to the frequency band. Further, FIG. 8 shows the definition of the resolution level that is almost opposite to the composition level.

  In the above discussion, when only one-dimensional wavelet transform is desired, processing in only the vertical or horizontal direction may be performed.

  In such inverse transformation of 5 × 3 wavelet transform, an inverse low-pass operation is first performed on the interleaved coefficient array as shown in FIG. 5 centering on a coefficient whose X coordinate is an even number (x = 2i). Apply a filter, and then apply an inverse high-pass filter centered on a coefficient whose X coordinate is an odd number (x = 2i + 1) (this is performed for all y). The expression of step 1 in the inverse transformation expression (2) represents an inverse low-pass filter, and the expression of step 2 represents an inverse high-pass filter. As in the case of forward conversion, there may be no adjacent coefficient for the central coefficient at the edge of the image. In this case, the coefficient value is appropriately supplemented by a predetermined rule, but the description is omitted. .

  As a result, the coefficient array in FIG. 5 is converted (inversely converted) into a coefficient array as shown in FIG. Next, in the vertical direction, apply an inverse low-pass filter centered on the coefficient whose Y coordinate is an even number (y = 2i), and then apply an inverse high-pass filter centered on the coefficient whose Y coordinate is an odd number (y = 2i + 1). If this is done (for all X coordinates), one wavelet inverse transformation is completed, and the image of FIG. 3 is returned (reconstructed). If wavelet transformation is performed a plurality of times, FIG. 3 is regarded as an LL subband, and similar inverse transformation may be repeated using other coefficients such as HL.

  As described above, in the 5 × 3 wavelet transform, one low-pass filter output (low-pass coefficient) is obtained using five pixels, and one high-pass filter output (high-pass coefficient) is obtained using three pixels. It is a conversion. On the other hand, the 9 × 7 wavelet transform is a conversion in which 9 pixels are used to obtain one low-pass filter output (low-pass coefficient), and 7 pixels are used to obtain one high-pass filter output (high-pass coefficient). is there. As described above, the main difference between the 5 × 3 conversion and the 9 × 7 conversion is the difference in the filter range, and the low pass filter is applied to the center of the even position and the high pass filter is applied to the center of the odd position. . Therefore, FIGS. 4 to 7 apply to the 9 × 7 conversion as well.

Therefore, one aspect of the present invention is characterized in that, in the encoding apparatus according to the first aspect of the present invention , the encoding control means controls the quantization for each decomposition level of the wavelet transform.
If the quantization control is performed for each decomposition level in this way, the quantization control when using the wavelet transform is simplified, and the image quality control can be speeded up.

According to another aspect of the present invention, in the encoding device according to the second aspect of the present invention , the encoding control means controls truncation for each decomposition level of the wavelet transform.

  If truncation control is performed for each decomposition level in this way, truncation control when using wavelet transform is simplified, and high-speed image quality control is possible.

  Now, the “comb shape” generated by the movement between fields changes the pixel value greatly in the vertical direction for each pixel, but the pixel value does not always change in the horizontal direction (the pixel value is changed only at the end of the comb). (It does not change). That is, the “comb” is a high-frequency component in the vertical direction. Referring to FIG. 1D, when the amount of movement is large, the length of the “comb” horizontal edge increases, whereas the vertical edge is substantially constant. Since the horizontal edge component is a high-frequency component in the vertical direction, the higher the amount of motion (the more conspicuous the comb shape), the higher the high-frequency component in the vertical direction.

  Therefore, when a “comb” is generated, 1LH coefficient (decomposition level 1, vertical high frequency, horizontal low frequency coefficient) takes the largest value among the wavelet transform coefficients, and 1HH coefficient (decomposition level 1). , Vertical high-frequency and horizontal high-frequency coefficients) take the next largest value, and 1HL coefficient (decomposition level 1, vertical low-frequency, horizontal high-frequency coefficient) takes the smallest value. In this way, even in the same frequency band (decomposition level), the degree of reflection of the “comb” differs for each subband.

Therefore, the invention of claim 3 is characterized in that, in the encoding apparatus of claim 1, the encoding control means controls the quantization for each subband of the wavelet transform.

  Controlling the quantization for each subband means controlling the quantization taking into account both the frequency band and direction, so high-accuracy and high-speed image quality control is possible with simple control of quantization. It is.

According to a fourth aspect of the present invention, in the encoding device according to the second aspect of the present invention, the encoding control means controls truncation for each subband of the wavelet transform.
Controlling truncation for each subband means that truncation is controlled by taking both the frequency band and direction into consideration, so high-precision and high-speed image quality control is possible with simple truncation control. It is.

The invention according to claim 5 is the encoding apparatus according to claim 3 , wherein the encoding control means controls the subbands in the same resolution level so as to minimize the quantization of the LH subband. It is what.

  The LH subband is a high-frequency component in the vertical direction, and if the loss is large, the “afterimage” described with reference to FIG. 2 is likely to occur. Therefore, since the high frequency components in the vertical direction are preferentially preserved by leastly quantizing the LH subband, the “afterimage phenomenon” that is an unnatural image quality degradation is effectively suppressed.

The invention according to claim 6 is the encoding apparatus according to claim 4 , wherein the encoding control means controls the subbands in the same resolution level so that the truncation of the LH subband is minimized. It is what.

  Reducing the truncation of the LH subband is equivalent to repressing the quantization of the LH subband the most, so that the “afterimage phenomenon”, which is an unnatural image degradation, is effectively suppressed.

  As can be understood from the above description, the effect of the “comb” appears most at the decomposition level 1 and becomes smaller as the decomposition level increases. According to the applicant's experiment, it has been confirmed that a sufficient effect can be obtained if the quantization or truncation up to the decomposition level 2 is appropriately controlled.

Therefore, a seventh aspect of the invention is characterized in that, in the encoding device of the third or fifth aspect of the invention, the encoding control means controls the quantization only for the decomposition level 1 and the decomposition level 2. Is.
According to an eighth aspect of the present invention, in the encoding device according to the fourth or sixth aspect of the present invention, the encoding control means controls truncation only for the decomposition level 1 and the decomposition level 2. It is.
Thus, by limiting the object of quantization or truncation control to the decomposition level 2 or lower, the control can be simplified and speeded up, and sufficient effects can be obtained.

  Here, the quantization, truncation, and image quality control in JPEG2000 will be further described. JPEG2000 encoding processing is generally performed according to the flow shown in FIG. FIG. 10 is a schematic diagram showing the relationship between images, tiles, subbands, precincts, and code blocks.

  A tile is an image divided into rectangles. When the number of divisions = 1, the image and the tile are the same. Each tile is regarded as an independent image, wavelet transformation is performed, and subbands are generated. In the basic specification of JPEG2000, when 9 × 7 conversion is used as wavelet conversion, coefficients included in the same subband can be divided by the same number and linearly quantized. Therefore, image quality control by linear quantization is possible in subband units (that is, image quality control units by linear quantization are subbands).

  The precinct is a subband divided into rectangles (of a size that can be designated by the user) and represents a rough position in the image. With respect to the precinct obtained by dividing the HL, LH, and HH subbands, precincts (a total of three) at the corresponding positions of the subbands are handled as a group. However, one precinct obtained by dividing the LL subband is handled as one group. The precinct can be the same size as the subband. A code block is obtained by dividing a precinct into rectangles (of a size that can be specified by the user). FIG. 10 shows the relationship among tiles, subbands, precincts, and code blocks in the case of decomposition level 3.

  The subband coefficients after quantization are bit-plane encoded in units of code blocks (one bit plane is decomposed into three sub-bit planes and encoded). A packet is obtained by extracting and collecting a part of codes from all code blocks included in the precinct (for example, collecting codes of MSBs of all code blocks to the third bit plane). Since the “part” may be “empty” (from), the content of the packet may be “empty” (from) in terms of code.

  Collecting packets of all precincts (= all code blocks = all subbands), a part of the code of the entire image (for example, the code of the MSB to the third bit plane of the wavelet coefficients of the entire image) This is called a layer. Since the layer is roughly a part of the code of the bit plane of the entire image, the image quality increases as the number of layers to be decoded increases. Therefore, a layer is a unit of image quality. If all the layers are collected, it becomes the sign of all the bit planes throughout the image.

  FIG. 11 shows an example of the layer configuration when the number of wavelet transform layers (decomposition level) = 2 and the precinct size = subband size. In addition, examples of packets included in each layer are shown in FIG.

  In this example, the precinct size is equal to the subband size, and since the code block having the same size as the size of the printinct is adopted, the subband at the decomposition level 2 is divided into four code blocks. The sub-band of decomposition level 1 is divided into nine code blocks. Since the packet is based on a precinct, when precinct = subband, the packet extends across the HL to HH subbands.

  Here, the packet is “a collection of code blocks extracted and collected”, and unnecessary codes need not be generated as packets. For example, the code of the lower bit plane as contained in the layer 9 in FIG. 11 is usually discarded (truncated).

  Therefore, image quality control by truncation is possible in units of code blocks (and units of sub-bit planes). That is, the image quality control unit by truncation is a code block. Note that the sequence of packets is called a progression order.

Now, when performing linear quantization on a subband basis, in order to improve the subjective image quality, the number of quantization steps (the denominator of division during linear quantization) is:
Number of quantization steps = Number determined by normalization to maximize PSNR / Visual weight (commonly called Visual Weight)
(A)
It is normal. When each coefficient is inverse frequency transformed (inverse wavelet transform for JPEG2000) and returned to the RGB value, the influence of the quantization error generated in each coefficient on the final RGB value is the frequency band (for wavelet transform, subband) The ratio is determined by a constant (so-called subband gain) at the time of inverse frequency conversion (inverse wavelet conversion in JPEG2000).

  In order to improve the PSNR, it is necessary to make the influence uniform between subbands, and quantization is performed so as to cancel the subband gain for each subband in order to make the influence uniform. The details are described in Non-Patent Document 2. In short, at a certain compression ratio, in order to minimize the mean square of errors (= maximize PSNR) generated in the signal after inverse transformation (= consisting of multiple signal values), each subband is In general, linear quantization is performed using the reciprocal of the square root of the subband gain (a value that is a constant multiple thereof). In the present specification, this “number determined by normalization for maximizing PSNR” is referred to as “basic quantization step number”.

  However, since PSNR does not match the subjective image quality, the number of basic quantization steps divided by the visual weight (visual weight) is generally used as the number of quantization steps, as in equation (a) above. Is.

  FIG. 13 shows a measurement example of visual sensitivity described in Non-Patent Document 3. The horizontal axis is the fringe frequency (cycle / degree), and the vertical axis is the reciprocal of the minimum contrast that humans perceive at that frequency, that is, the sensitivity to the contrast (relative value). The sensitivity to contrast measured for each stripe of luminance Y, color difference Cb, and color difference Cr is shown in FIG. From FIG. 13, human vision is sensitive to changes in contrast at low spatial frequencies, insensitive to high frequencies, most sensitive to Y components, and most insensitive to Cb components. I understand that.

Based on this visual sensitivity, JPEG2000 standard books illustrate weights described later. The weight of each subband is obtained as an integral value of the visual sensitivity curve in the frequency band occupied by the subband, and details thereof are described in Non-Patent Document 4. A large weight is given to the low frequency subband, and a small weight is given to the high frequency subband. As such visual weight, what is originally obtained from visual sensitivity should be used, but it is also possible to obtain a desired image quality by consciously operating the value. For example, on the visual sensitivity, 1LH weight = 1HL weight,
1LH weight> 1HL weight,
By doing so, it becomes possible to refrain from quantization of 1 LH than quantization of 1 HL.

Accordingly, the invention of claim 9 is characterized in that, in the encoding device of the invention of claim 1, 3, 5, or 7 , the encoding control means changes the visual weight for each subband in the control of quantization. It is.
Thus I by the altering the visual weighting for each sub-band in the control of the quantization, it is possible to image quality control more accurate.

Similarly for truncation, the visual weight can be added as the importance of the code. In general, the value of a code is evaluated by “increase in quantization error when the code is discarded / the amount of code”, and a code with a small value (an increase in error compared to a decrease in code amount due to the discard). Discarded in order starting from the few codes. In order to improve subjective image quality, the value of a code is expressed as “increment of quantization error when the code is discarded × visual weight / the amount of code”
It is normal to evaluate with.

Accordingly, the invention of claim 10 is characterized in that, in the encoding device of the invention of claim 2, 4, 6 or 8 , the encoding control means changes the visual weight for each subband in the control of truncation. is there.
Thus, by changing the visual weight for each subband in the truncation control, it is possible to control the image quality with higher accuracy.

  The “weight” described above is in units of subbands, and this weight is often multiplied by the characteristics of a subblock (code block in JPEG2000) obtained by dividing the subband. Human vision is a function of the contrast intensity itself as well as the frequency. The higher the contrast, the less sensitive the contrast error (change), even in the same frequency band. This is because the threshold of whether or not a human feels or feels the contrast error depends not on the absolute value of the error but on the relative size of the error (ratio of error value / original value) (so-called Weber's Is the law). In this way, the effect of making the error less noticeable when the original value (original value) is large is generally referred to as “masking”.

Here, since the frequency-converted coefficient generally reflects the contrast of the original image, the magnitude of the coefficient value can be regarded as the magnitude of the contrast. Therefore, if an index indicating whether the value of the coefficient included in the sub-block is large or small is called a “masking index” in this specification, the importance of the code is
"Increment of quantization error when discarding the code x visual weight / (masking index x number of codes)"
Can be evaluated.

Therefore, the invention according to claim 11 is the encoding apparatus according to claim 2, 4, 6 or 8 ,
(Sum of absolute values of wavelet coefficients included in image quality control unit) / ((number of wavelet coefficients included in image quality control unit) ^ α)) (where 0 <α ≦ 1)
And a coding control means that reflects the masking for each image quality control unit in the truncation control using the masking index .
Thus, in the truncation control, the image quality control with higher accuracy can be performed by reflecting the masking after the image quality control unit.

  Similarly, instead of changing the “weight” in units of subbands, or changing the weight in units of subbands, a comb-shaped polygon is obtained for each subblock into which each subband is divided, and the weight applied to each subblock is determined. It is also possible to modify by a comb-like shape in the sub-block. Since the weight is originally derived from the visual frequency characteristics, it is the same value in the same subband (= same frequency band), but by the above modification, the weight is changed for each subblock. . As described above, comb-shaped multiples can be detected for each sub-block (code block in JPEG2000). For example, if the comb-shaped multiple is multiplied by the weight for each sub-band, a different weight can be applied to each sub-block. .

Therefore, the invention of claim 12 is the encoding apparatus of the invention of claim 2, 4, 6, 8 or 10 , wherein the encoding control means changes the visual weight for each image quality control unit in the truncation control. It is what.

According to a thirteenth aspect of the present invention, there is provided a control step for controlling wavelet coefficient quantization in a frame-based encoding process of an interlaced image using wavelet transform, and a detection step for detecting a motion amount between fields constituting a frame. And
The control step controls the quantization based on the amount of motion detected in the detection step and the compression rate or quantization error,
In the detection step, as the amount of movement,
1LHmean, 1HL subband coefficient absolute value 1HLmean, 2LH subband coefficient absolute value 2LHmean, 2HL subband coefficient absolute value Average value of 2HLmean
(1LHmean / 1HLmean) / (2LHmean / 2HLmean) ratio, or
Regarding the luminance component, when the entropy coding of 1LH, 1HL, 2LH, and 2HL subbands is performed, the respective code amounts of 1LHcode, 1HLcode, 2LHcode, and 2HLcode
Ratio of (1LHcode / 1HLcode) / (2LHcode / 2HLcode)
This is a coding control method characterized by calculating .

According to the fourteenth aspect of the present invention, there is provided a control step for controlling truncation of wavelet coefficients or codes in frame-based coding processing of interlaced images using wavelet transform, and detection for detecting a motion amount between fields constituting a frame. A process,
The control step controls truncation based on the amount of motion detected in the detection step and the compression rate or quantization error,
In the detection step, as the amount of movement,
1LHmean, 1HL subband coefficient absolute value 1HLmean, 2LH subband coefficient absolute value 2LHmean, 2HL subband coefficient absolute value Average value of 2HLmean
(1LHmean / 1HLmean) / (2LHmean / 2HLmean) ratio, or
Regarding the luminance component, when the entropy coding of 1LH, 1HL, 2LH, and 2HL subbands is performed, the respective code amounts of 1LHcode, 1HLcode, 2LHcode, and 2HLcode
Ratio of (1LHcode / 1HLcode) / (2LHcode / 2HLcode)
This is a coding control method characterized by calculating .

A fifteenth aspect of the invention is a program that causes a computer to execute a control step and a detection step in the encoding control method of the thirteenth or fourteenth aspect of the invention .

The invention of claim 16 is a computer-readable recording medium on which a program for causing a computer to execute the control step and the detection step in the encoding control method of the invention of claim 13 or 14 is recorded.

As can be understood from the above description,
(1) According to the first to fourteenth aspects of the present invention, it is possible to simultaneously achieve suppression of unnatural image quality deterioration due to movement between fields and improvement of image quality.
(2) According to the third and fourth aspects of the invention, since quantization or truncation is easily controlled, higher-speed image quality control is possible.
(3) According to the inventions of claims 5 and 6 , since the high-frequency component in the vertical direction is preferentially stored by storing the high-frequency signal in the vertical direction, unnatural image quality degradation is effective due to the movement between the fields. Can be suppressed.
(4) According to the present invention 7,8, control of quantization or truncation becomes simple, allowing faster image quality control. According to the ninth to twelfth aspects, more accurate image quality control is possible.
(5) According to the inventions of claims 15 and 16 , encoding control for simultaneously achieving suppression of unnatural image quality degradation and image quality improvement in frame- based encoding of interlaced images is facilitated using a computer. Can be implemented.
And so on.

  FIG. 14 shows the configuration of the encoding apparatus according to the embodiment of the present invention. The encoding apparatus shown here performs frame-based encoding of an interlaced image. Here, a frame (non-interlaced image) synthesized from two consecutive fields (interlaced image) is input. . However, this does not exclude a mode in which means for synthesizing a frame from a field is provided inside the encoding apparatus. This encoding apparatus detects an amount of motion between two fields constituting a frame by using an encoding processing unit 100 that performs frame-based encoding processing, and based on the amount of motion and a desired compression rate or quantization error. The coding processing unit 100 is configured to control the frequency coefficient quantization, or the coding processing unit 100 controls the frequency coefficient or code truncation.

  Here, it is assumed that the encoding processing unit 100 performs encoding processing using the algorithm of JPEG2000 (however, it is not limited to JPEG2000 only). In addition, 9 × 7 conversion is used as wavelet conversion. The frame image is assumed to be composed of three RGB components. The specific contents of motion amount detection and quantization or truncation control will be described in connection with examples described later.

  The content of movement amount detection and quantization or truncation control in such an encoding device is also the content of the encoding control method of the present invention. Therefore, the description related to the encoding apparatus of the present invention is also the description of the encoding control method of the present invention.

  The coding apparatus as shown in FIG. 14 can also implement the whole or the function of the coding control means 101 by a program using a computer such as a personal computer or a microcomputer. In other words, the procedure of the encoding control method of the present invention can be executed by a program on a computer. Such a program and various recording (storage) media such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor storage element that can be read by a computer on which the program is recorded are also included in the present invention.

  An embodiment in which the present invention is implemented using a computer will be briefly described with reference to FIG. In FIG. 15, 110 is a central processing unit (CPU), 111 is a memory for temporarily storing data, programs, etc., 112 is a hard disk device as auxiliary storage, and 113 is a system bus. It is assumed that moving image data to be encoded is stored in the hard disk device 112 in the form of frames composed of fields.

  The outline of the flow of processing is as follows. First, one frame of a moving image is read from the hard disk device 112 into the memory 111 according to a command from the CPU 110 (1). The CPU 110 reads a frame on the memory 111 and executes JPEG2000 encoding processing. At this time, the CPU 110 detects a motion amount (comb amount) between fields, and controls quantization or truncation (2). The CPU 110 writes the encoded data in another area on the memory 111 (3). When the processing for one frame is completed, the encoded data is recorded in the hard disk device 112 according to a command from the CPU 110 (4). The above processes (1) to (4) are repeated for the number of frames of the moving image.

  Prior to the description of the embodiment, the description of JPEG2000 will be supplemented.

  As described above, the basic flow of JPEG2000 compression / decompression processing is as shown in FIG. When a color image composed of RGB3 components is compressed, wavelet conversion is performed after DC level shift and component conversion (color conversion) to luminance / color difference components. The wavelet coefficients are quantized as necessary, and then entropy-coded in units of bit planes (precisely, the bit plane is subdivided into three sub-bit planes and encoded).

  At the time of decompression, the wavelet coefficients restored by entropy decoding in bit plane units are inversely quantized as necessary. The obtained wavelet coefficients for each component are subjected to inverse wavelet transformation, and the obtained pixel values of each component are subjected to inverse component transformation, thereby returning to RGB pixel values.

  The conversion formula for the DC level shift of JPEG2000 is as follows.

I (x, y) ← I (x, y) -2 ^{Ssiz (i)} forward conversion
I (x, y) ← I (x, y) +2 ^{Ssiz (i)} Inverse transformation (5)
Here, Ssiz (i) is the bit depth of each component i of the original image (i0, 1, 2 for RGB images).

  When this DC level shift is a positive number such as an RGB signal value, the forward conversion subtracts half of the dynamic range of the signal from each signal value, and the inverse conversion uses the signal dynamics for each signal value. A level shift that adds half the range is performed. However, this level shift is not applied to signed integers such as Cb and Cr signals of YCbCr signals.

  In JPEG 2000, reversible transformation (RCT) and irreversible multiple component transformation (ICT) are defined as component transformations.

RCT forward and inverse transforms are expressed by the following equations.
Forward conversion
Y ₀ (x, y) = floor ((I ₀ (x, y) + 2 * (I ₁ (x, y) + I ₂ (x, y)) / 4)
Y ₁ (x, y) = I ₂ (x, y) -I ₁ (x, y)
Y ₂ (x, y) = I ₀ (x, y) -I ₁ (x, y)
Reverse transformation
I ₁ (x, y) = Y ₀ (x, y) -floor ((Y ₂ (x, y) + Y ₁ (x, y)) / 4)
I ₀ (x, y) = Y ₂ (x, y) + I ₁ (x, y)
I ₂ (x, y) = Y ₁ (x, y) + I ₁ (x, y) (6)
In the equation, I represents an original signal, and Y represents a signal after conversion. In the case of the RGB signal, 0 = R, 1 = G, 2 = B in the I signal, and 0 = Y, 1 = Cb, 2 = Cr in the Y signal.

The forward conversion and the reverse conversion of ICT are expressed by the following equations.
Forward conversion
Y ₀ (x, y) = 0.299 * I ₀ (x, y) + 0.587 * I ₁ (x, y) + 0.144 * I ₂ (x, y)
Y ₁ (x, y) =-0.16875 * I ₀ (x, y) -0.33126 * I ₁ (x, y) + 0.5 * I ₂ (x, y)
Y ₂ (x, y) = 0.5 * I ₀ (x, y) -0.41869 * I ₁ (x, y) -0.08131 * I ₂ (x, y)
Reverse transformation
I ₀ (x, y) = Y ₀ (x, y) + 1.402 * Y ₂ (x, y)
I ₁ (x, y) = Y ₀ (x, y) -0.34413 * Y ₁ (x, y) -0.71414 * Y ₂ (x, y)
I ₂ (x, y) = Y ₀ (x, y) + 1.772 * Y ₁ (x, y) (7)
In the equation, I represents an original signal, and Y represents a signal after conversion. In the case of the RGB signal, 0 = R, 1 = G, 2 = B in the I signal, and 0 = Y, 1 = Cb, 2 = Cr in the Y signal.

  As described above, when the 9 × 7 wavelet transform is selected in JPEG2000, the wavelet coefficients can be linearly (scalar) quantized for each subband. A common number of quantization steps is used in the same subband. The quantization equation is shown in equation (8), and the number of quantization steps (Δb) is shown in equation (9).

q _b (u, v) = sign (a _b (u, v)) * floor (| a _b (u, v) | / Δb) (8)
Where a _b (u, v) is the coefficient in subband b
q _b (u, v) is the coefficient in subband b
Δb is the quantization step in subband b Δb = 2 ^Rb−εb * floor (1 + ^μb / 2 ¹¹ ) (9)
Where Rb is the dynamic range in subband b
εb is the quantization index in subband b
μb is the mantissa of quantization in subband b

  The exponent εb and mantissa μb define all subbands at each decomposition level, only the LL subband at the lowest decomposition level, and the remaining subbands using a predetermined formula There are two types of methods to specify. The former is called explicit quantization or explicit quantization, and the latter is called implicit quantization (derived quantization or implicit quantization). The implicit quantization exponent and mantissa pair (εb, μb) is determined by equation (10).

(ε _b , μ _b ) = (ε ₀ -N _L + n _b, μ ₀ ) (10)
Where n _b is the number of decomposition levels

The inverse quantization equation is shown in equation (11).
_{Rq b (u, v) =} (q b (u, v) + r * 2Mb-Nb (u, v)) * Δb q b (u, v)> time of 0
_{= (q b (u, v} ) -r * 2 Mb-Nb (u, v)) * Δb q b (u, v) < 0 When
= 0 q _b (u, v) = 0 (11)

  When the 5 × 3 wavelet transform is used, the quantization step (Δb) is always 1. This means no quantization.

  In JPEG2000, substantial quantization (post-quantization) is possible by discarding unnecessary lower bitplane codes after encoding wavelet coefficients. Further, the same effect can be obtained by not encoding unnecessary lower bit planes from the beginning (that is, discarding them in the coefficient state). In this specification, both the discard before encoding and the discard after encoding are collectively referred to as truncation.

  In this embodiment, the linear quantization of wavelet transform coefficients is controlled based on the amount of motion (comb amount) and the compression rate or quantization error, and the processing flow is shown in FIG. Steps S100, S101, S104, and S105 in FIG. 16 are processing steps of the encoding processing means 100 in FIG. Step S102 is a processing step of the encoding control means 101 of FIG. Step S <b> 103 is a processing step related to both the encoding processing unit 100 and the encoding control unit 101. That is, the linear quantization itself is executed by the encoding processing unit 100, but the motion amount detection and the linear quantization control are executed by the control unit 101. A more detailed processing flow of step S103 in FIG. 16 is shown in FIG. The contents of the processing steps will be described in order.

<< Step S100 >>
The DC level shift and component conversion described above are executed for each frame. Therefore, the frame after this processing step consists of three components of luminance Y and color differences Cb and Cr.

<< Step S101 >>
A two-dimensional wavelet transform is applied to each component of the frame. Here, the 9 × 7 wavelet transform is used as described above. In addition, although conversion up to decomposition level 4 is performed here, the present invention is not limited to this.

<< Step S102 >>
The amount of motion between the fields (comb amount) is detected. Here, as in the invention according to Japanese Patent Application No. 2002-289807 which is the prior application of the present applicant, focusing on the 1LH subband coefficient, the amount of motion between the fields for the entire image (comb amount) is detected. The following method is used to enable accurate detection even when the original image contains many high-frequency waves in the vertical direction other than the comb shape.

That is, regarding the wavelet coefficients of the luminance component, the average value of the absolute values of the coefficients of the 1LH subband (1LHmean), the average value of the absolute values of the coefficients of the 1HL subband (1HLmean), and the average value of the absolute values of the coefficients of the 2LH subband (2LHmean), find the average value (2HLmean) of the absolute values of the coefficients of the 2HL subband, and use it as an evaluation value for the amount of movement (comb amount)
(1Lhmean / 1HLmean) / (2Lhmean / 2HLmean)
The “coefficient ratio” is calculated.

  As described above, the size of 1Lhmean / 1HLmean reflects the comb shape. However, this ratio also increases when the original image originally contains many high frequencies in the vertical direction other than the comb shape. On the other hand, at the decomposition level 2, the influence of the comb shape on the wavelet coefficient is small, and the influence of the high frequency other than the comb shape included in the original image becomes more remarkable. Therefore, if 1Lhmean / 1HLmean is normalized by the ratio (2Lhmean / 2HLmean) of the high frequency other than the comb included in the original image in the vertical and horizontal directions (2Lhmean / 1HLmean), the amount of the comb can be detected accurately.

  Note that the magnitude of the average value of the absolute values of the wavelet coefficients is generally reflected in the magnitude of the code amount after entropy coding. Therefore, similar to the invention according to Japanese Patent Application No. 2002-289807, the necessary subband coefficients are entropy. It is also possible to perform encoding and use the code amount.

That is, entropy coding is performed on the luminance components 1LH, 1HL, 2LH, and 2HL subbands to obtain respective code amounts 1Lhcode, 1Hlcode, 2Lhcode, and 2HLcode,
(1Lhcode / 1HLcode) / (2Lhcode / 2HLcode)
The “code amount ratio” can be calculated as an evaluation value of the motion amount (comb amount). A mode of calculating such a code amount ratio is also included in the present embodiment.

  In this embodiment, the coefficient ratio (or code amount ratio) and predetermined thresholds A and B (where A <B, the values of A and B differ depending on the coefficient ratio and the code amount ratio). The amount of motion is evaluated in three stages: “small” (less than A), “medium” (A or more, less than B), and “large” (B or more). However, this evaluation can also be performed in step S103, and this aspect is also included in this embodiment.

<< Step S103 >>
The entire processing in this step will be described later with reference to FIG. 17, and here, a method for determining the number of quantization steps for each subband of each component will be described. This determination process is related to the encoding control means 101 in FIG.

  Desired compression ratio (or quantization error estimated at the time of encoding) and predetermined threshold values X and Y (where X <Y, the values of X and Y are different between the compression ratio and the quantization error) , The compression ratio (or quantization error) is evaluated in three stages: “low” (less than X), “medium” (X or more, less than Y), and “high” (Y or more).

  More simply, the user may designate one of the three options “low (relative) compression ratio, medium compression ratio, high compression ratio”. Such an embodiment is also included in the present embodiment.

  Then, one of the nine visual weights 1 to 9 shown in FIG. 18 is determined from the motion amount evaluation and the compression rate (or quantization error) evaluation. However, in this embodiment, the five visual weights A to E shown in FIG.

FIG. 20 shows a specific numerical example of the visual weights A to E in an arrangement corresponding to FIG. In this example,
Visual weight A <visual weight B <visual weight C
In this order, the weights for the low-decomposition level and subband are increased so that a desirable image quality can be obtained corresponding to the high compression rate (or the magnitude of the quantization error). Also,
Visual weight A <visual weight D <visual weight E
In this order, the weights for the high-frequency composition level and the sub-band are increased so that comb-shaped quantization (or truncation) is suppressed for images with a large amount of motion.

  FIG. 21 shows another numerical example of the visual weights A to E in an arrangement corresponding to FIG. By applying the visual weight shown here, a favorable image quality can be obtained corresponding to the high compression rate (or the magnitude of quantization error), and at the same time, the LH subband coefficients up to decomposition level 2 are preferentially treated. Comb quantization (or truncation) is refrained.

  Using the visual weight determined as described above, the number of quantization steps applied to the quantization of each subband of each decomposition level of each component is obtained by the following equation.

Number of quantization steps (Denominator of division during linear quantization)
= Constant x number of basic quantization steps / visual weight (12)

  Here, the value shown in FIG. 22 is used as the number of basic quantization steps. Note that it is not uncommon for the number of quantization steps to be smaller than 1 when quantization is not intended to reduce the dynamic range.

  By using such a compression step (or quantization error) and the number of quantization steps reflecting the amount of motion that reflects the amount of motion, it is possible to perform quantization that reflects the quantization error and the comb shape.

  The “constant” in the equation (12) is for actively reducing the dynamic range, and the compression rate and the amount of quantization error are controlled according to the size. The value of this constant should be specified as appropriate, and of course can be 1.

  FIG. 23 shows the value of the number of quantization steps calculated using the visual weight value shown in FIG. 20 in a form corresponding to FIG. 24 shows the quantization step number value calculated using the visual weight value shown in FIG. 21 in a form corresponding to FIG.

  In the case of linear quantization, the normal mathematical quantization error is the sum of all the coefficients of “1/2 of the number of quantization steps × the square root of the subband gain”. Alternatively, the estimated magnitude of the quantization error can be reflected as the magnitude of the number of quantization steps (when visual quantization error is used as an error amount index, Multiply the 2 × subband gain square root by the visual weight to be applied in that subband to get the sum of all coefficients, as mentioned above, since the visual weight itself is a function of error, The visual weight itself must be selected accordingly).

  Next, the entire process of step S103 will be described with reference to FIG.

  First, based on the amount of motion and the compression rate (or quantization error), the visual weight shown in FIG. 20 or FIG. 21 is selected as described above (step S110).

  One component is selected (step S111), one composition level of the selected component is selected (step S113), and one subband of the selected composition level of the selected component is selected. (Step S115). Then, the number of quantization steps applied to the subband is determined (step S117), and all the coefficients of the subband are linearly quantized (step S118).

  Returning to step S115, another subband is selected, the number of quantization steps for it is determined, and all the coefficients of the subband are linearly quantized (steps S117 and S118).

  When the linear quantization for all subbands of the selected component and decomposition level is completed (step S116, Yes), the process returns to step S113, and another one of the decomposition levels is selected and all of the decomposition levels are selected. The process for linear quantization on the subband is repeated (steps S113 to S118).

  When the linear quantization for all subbands of all the decomposition levels of the selected component is completed (step S114, Yes), the process returns to step S111 to select another component and all the decompositions of the component. The process for linear quantization of all subbands of the level is repeated (steps S113 to S118).

  When linear quantization has been completed for all components (step S112, Yes), step S103 in FIG. 16 is terminated.

  Returning to the flowchart of FIG.

<< Step S104 >>
Entropy coding is performed for each subband on the linearly quantized wavelet coefficients. In JPEG2000, for each subband, coefficients are encoded in bit plane units from upper bits (MSB) to lower bits (LSB).

  For simplicity, a general description will be given assuming that the composition level = 2. Assume that the coefficients of the 2LL subband in FIG. 7 have values as shown in FIG. These values are expressed in binary numbers, and each bit is divided into bit planes. The coefficients in FIG. 25 can be divided into four bit planes as shown in FIG. Since the binary representation of the decimal number 15 is “1111”, “1” is set in all the bit planes at the position corresponding to the coefficient of the value 15 in FIG.

  In JPEG 2000, these bit planes are individually entropy encoded. More specifically, each bit plane is divided into three sub bit planes, and entropy encoding is performed for each sub bit plane. An entropy coding uses an arithmetic coder called MQ coder, but the description thereof is omitted.

<< Step S105 >>
A packet is generated from the entropy code, arranged in a predetermined progression order, and a code of a predetermined format to which a necessary header is added is formed.

  In the case of the composition level = 2, for example, a code having a configuration as shown in FIG. 27 can be formed. This example shows that the code starts with the 2LL subband and ends with the 1HH subband. Each bit plane is encoded in order from the most significant bit.

  In this embodiment, truncation in the entropy coding stage, that is, discarding of wavelet transform coefficients is controlled based on the motion amount (comb amount) and the compression rate or quantization error, and the processing flow is shown in FIG. Shown in Steps S200, S201, S202, S206, and S207 in FIG. 28 are processing steps of the encoding processing means 100 in FIG. Steps S203, S204, and S205 are processing steps of the encoding control means 101. The contents of each processing step will be described in order.

<< Step S200 >>
The DC level shift and component conversion described above are executed for each frame.

<< Step S201 >>
A two-dimensional wavelet transform is applied to each component of the frame. Here, the 9 × 7 wavelet transform is used as described above. Here, it is assumed that conversion up to decomposition level 4 is performed.

<< Step S202 >>
The wavelet coefficients are quantized (normalized) for each subband using the basic quantization step number shown in FIG.

<< Step S203 >>
As in step S102 in the first embodiment, a “coefficient ratio” is obtained as the amount of motion between the fields (comb-shaped amount). By comparing this coefficient ratio with two predetermined thresholds, the amount of motion is “small”. It is evaluated in three levels, “medium” and “large”.

<< Step S204 >>
As described with reference to step S103 of the first embodiment, one of visual weights A to E shown in FIG. 19 is selected based on the compression rate (or quantization error) and the amount of motion. Specifically, one of the visual weight values shown in FIG. 20 or FIG. 21 is selected.

<< Step S205 >>
The truncation amount for each subband is calculated by the following equation. However, it is converted to an integer by rounding off.

  Truncation amount = log2 (constant x 1 / visual weight of the subband) (13)

  As described above, the “constant” in the above equation becomes larger as the quantization error is larger (because the quantization error increases as the coefficient is discarded). Here, three values of 16, 32, and 64 are used. is doing. FIG. 29 and FIG. 30 show the truncation amount calculated using the visual weight value of FIG. 20 and the truncation amount calculated using the visual weight value of FIG. 21 in a form corresponding to FIG. Therefore, in this step, the corresponding truncation value is actually selected from the table of FIG. 29 or FIG.

<< Step S206 >>
For each subband, only the upper bitplanes are entropy-encoded except for the lower subbitplane corresponding to the truncation amount (FIG. 29 or FIG. 30) determined for the subband. Here, truncation of coefficients is performed in units of bit planes, but truncation in units of sub-bit planes is also possible.

<< Step S207 >>
A packet is generated from the entropy code, and a code in which the packets are arranged in a predetermined progression order is formed.

In this embodiment, after entropy coding, truncation of codes is controlled in units of code blocks. In this embodiment, in addition to the visual weight for each subband similar to the above embodiments,
(A) “Decomposition level 1” “comb index” in code block units
(B) “Masking index” for each code block
The truncation for each code block is controlled in consideration of the above two, and the compression rate is precisely controlled near the desired value. The code block is an image quality control unit in JPEG2000 truncation.

  A processing flow of this embodiment is shown in FIG. Steps S300, S301, S302, S304, S309, and S310 are steps executed by the encoding processing means 100 in FIG. 14, and steps S303, S305, S306, S307, and S308 are executed by the encoding control means 101 in FIG. Are the steps A more detailed flow of step 308 is shown in FIG. Hereinafter, processing contents will be described in order.

<< Step S300 >>
The DC level shift and component conversion described above are executed for each frame.

<< Step S301 >>
A two-dimensional wavelet transform is applied to each component of the frame. Here, the 9 × 7 wavelet transform is used as described above. Here, it is assumed that conversion up to decomposition level 4 is performed.

<< Step S302 >>
The wavelet coefficients are quantized (normalized) for each subband using the basic quantization step number shown in FIG.

<< Step S303 >>
A masking index for each code block is calculated.

  As described above, human vision is a function of the contrast intensity itself, and even in the same frequency band, the higher the contrast, the less sensitive to the contrast error (change). This is because the threshold of whether or not a human feels or feels a contrast error depends on the relative magnitude (the ratio of error value / original value) rather than the absolute value of the error (so-called Weber's law) The effect is called "masking". An index indicating whether the value of the coefficient included in a code block (the coefficient whose frequency has been converted generally reflects the contrast of the original image, so that the magnitude of the coefficient value can be regarded as the magnitude of the contrast) is large or small, In this specification, it is called a “masking index”.

In general, as a typical “masking index”
(Sum of absolute values of coefficients included in code block) / ((Number of coefficients included in code block) ^ α)
A ratio is used. Experimentally, it is said that 0 <α ≦ 1, so in this example, α = 1 was selected,
The masking index for each code block is
Masking index = (sum of absolute values of coefficients included in code block) / (number of coefficients included in code block) (14)
Is calculated by The masking index for each code block is retained.

<< Step S304 >>
Entropy coding is performed for all bit planes of all subbands. At this time, the code amount of each bit plane of each code block is held. In addition, for each bit plane of each code block, the quantization error amount when the code is discarded is also calculated and stored.

<< Step S305 >>
"Large code ratio" from the 1LH, 1HL, 2LH, and 2HL subband code amounts 1Lhcode, 1Hlcode, 2Lhcode, and 2HLcode of the luminance component
(1Lhcode / 1HLcode) / (2Lhcode / 2HLcode)
Is calculated. The amount of motion is “small” (less than A), “medium” (A or more, less than B), “large” by comparing and determining the “code amount ratio” and predetermined thresholds A and B (A <B). ”(B or higher).

  Since this code amount ratio is an evaluation value of the motion amount (comb amount) of the entire image, in this embodiment, as will be described later, a “comb code amount index” for each code block is obtained and this is calculated. By multiplying the visual weight determined based on the “code amount ratio”, fine adjustment of the motion amount in units of code blocks is performed (see steps S307 and S308).

<< Step S306 >>
In the same manner as described in relation to step S103 of the first embodiment, the visual weights A to E shown in FIG. 19 are based on the compression rate (or quantization error) and the motion amount (= code amount ratio). Select one. Specifically, one of the visual weight values shown in FIG. 20 or FIG. 21 is selected.

<< Step S307 >>
As related to FIG. 10, the precinct is three rectangles composed of coefficients corresponding to the same position of the original image, and the code block is obtained by dividing the precinct. There is one “code block” in each of the 1HL, 1LH, and 1HH subbands. Therefore, wavelet coefficients and code amounts included in these code blocks are related to the same position in the original image.

Therefore, as taught by the invention of Japanese Patent Application No. 2002-300476, which is the prior application of the present applicant, the comb-shaped multiples for each code block are
(Code amount of LH coefficient included in the code block) / (Code amount of HL coefficient included in the code block at the same position in the original image)
Can be evaluated.

Therefore, in the present embodiment, the following “comb code amount index” is calculated as a comb-shaped index for each code block.
Comb code amount index = ((Code amount of 1LH coefficient included in the code block) / (Code amount of 1HL coefficient included in the code block at the same position)) / ((Total code amount of 1LH subband) / (1HL Total code amount of subband)) (15)

  The comb code amount index is treated as the same value for three code blocks located at the same position in the original image.

  In the next step S308, the “comb code amount index” is multiplied by the visual weight determined based on the code amount ratio and the compression ratio (or quantization error) of the entire subband. As a result, if the amount of motion at a certain code block position is larger than the amount of motion of the entire subband, the value of the weight for that code block is adjusted so as to be increased accordingly. In other words, the amount of movement is finely adjusted in code units.

<< Step S307 >>
When entropy coding is performed on all bit planes of a code block, the basic importance of the code is “increment of quantization error when the code is discarded × visual weight / the amount of code”
It is evaluated with.

  In this embodiment, in this step, the importance of the sign of a certain code block of a certain bit plane is obtained by the following equation.

  Importance = ((Increment of quantization error when the code is discarded) × Visual weight × Comb code amount index)) / (Masking index × its code amount) (16)

  The visual weight is common within the same subband, but since the comb code amount index and the masking index are different for each code block, the code can be differentiated for each code block by the weight calculated by the above formula. .

  Then, by sorting the importance of the code blocks of all components in ascending order, the order of which component and which code block bit plane code should be discarded is obtained.

  There are various ways of obtaining “increment of quantization error when the code is discarded”, and a mathematically exact method is described in Non-Patent Document 5, for example.

In this embodiment, in order to simplify the process,
“Quantization error per wavelet coefficient when the code of the n (n> 0) th bit plane counted from the least significant bit (LSB) is discarded” = 2 ^ (n−1) ”
Is considered. Discarding one bit plane from the bottom is equivalent to dividing the coefficient by 2 from an error point of view, and the error is 2 ^ 0 = 1 stochastically. Since the wavelet coefficients are normalized from the viewpoint of errors, the quantization error per wavelet coefficient can be treated as an RGB value error. Therefore, in this case, “the increment of the quantization error when the code of the nth bit plane counted from the LSB of the code block is discarded” is obtained by the following equation.

Quantization error increment =
((2 ^ (n-1) -2 ^ (n-2)) x number of coefficients included in the code block (17)

  This processing step will be further described along the flow of FIG. One component is selected (step S320), and one code block of the selected component is selected (step S322).

  For the selected code block, one bit plane is selected (step S324), and the importance of the bit plane is calculated by the above equation (16) (step S326). When calculation of the importance of all the bit planes of the selected code block is completed (step S325, Yes), the process returns to step S322, and another code block of the selected component is selected, and for each bit plane. Importance is calculated.

  When the importance calculation is completed for all the code blocks of the selected component, the process returns to step S320, another one component is selected, and the calculation of the importance for the component is repeated. When the importance calculation is completed for all the components (Yes in step S321), the importance is sorted and the order of code discard is determined (processing step S330).

  Reference is again made to FIG.

<< Step S309 >>
The codes are discarded until the desired code amount (compression rate) is reached according to the order determined from the importance. However, according to the specification of JPEG2000, since the code of each code block needs to be discarded in order from the LSB side, if the order is inconsistent with that, the code discard order is adjusted.

  For example, when the composition level = 2, it is assumed that a part of the order is as shown in FIG. 33 (in FIG. 33, in order to avoid complication, 1 to 1 of some bit planes of some code blocks). Up to 22 ranks are shown). In this example, since the code of the order “21” is closer to the LSB side than the code of the order “19”, the codes are discarded in the order of “18” → “20” → “21” → “19”. become.

  The implementation of processing for discarding codes in order from the LSB and in the order of the order may be performed by sorting as described above, but may be performed by a general Lagrange's undetermined multiplier method. Non-Patent Document 5 describes in detail the implementation method of Lagrange's undetermined multiplier method in JPEG2000.

  A modification of this embodiment will be described below. In this modified example, in step S305, the motion amount is evaluated based on the “coefficient ratio” similar to that in the first embodiment. In step S306, instead of the “comb code amount index”, a “comb coefficient index” for each code block calculated by the following equation is obtained.

Comb coefficient index =
((Sum of absolute values of 1LH coefficients included in the code block) / (sum of absolute values of 1HL coefficients included in the code block at the same position)) / ((sum of absolute values of all coefficients of 1LH subband)) / (Sum of absolute values of all coefficients of 1HL subband)) (18)

  In step S308, the importance of one code block of one bit plane is calculated by the following equation.

Importance =
((Increment of quantization error by discarding the code) x visual weight x comb coefficient index)) / (masking index x its code amount) (19)

  The control of truncation in units of code blocks similar to the present modification and the third embodiment can be applied to the control of truncation of wavelet coefficients as in the second embodiment.

  Although the embodiments of the present invention have been described in detail with respect to some examples and modifications, it is obvious that the embodiments of the present invention are not limited to them.

It is a figure for demonstrating the motion between fields and a comb shape. It is a figure for demonstrating the "afterimage phenomenon" by the motion between fields. It is a figure which shows the original image and coordinate system for description of two-dimensional wavelet transformation. It is a figure which shows the coefficient arrangement | sequence obtained by the filtering of the orthogonal | vertical direction to the original image of FIG. FIG. 5 is a diagram showing a coefficient array obtained by horizontal filling to the coefficient array in FIG. 4. It is a figure which shows the coefficient arrangement | sequence which deinterleaved the coefficient arrangement | sequence of FIG. It is a figure which shows the coefficient arrangement | sequence which de-interleaved the coefficient obtained by performing wavelet transformation twice. It is a figure which shows the relationship between a decomposition level and a resolution level. It is a block diagram which shows the flow of the compression / decompression process of JPEG2000. It is a figure for demonstrating the relationship between an image, a tile, a subband, a precinct, and a code block. It is a figure which shows an example of a multilayer structure. It is a figure which shows the packet which comprises the layer in FIG. It is a graph which shows the example of a visual characteristic. It is a device block diagram for demonstrating embodiment of this invention. It is a figure for demonstrating the form which implements this invention with a computer. It is a flowchart which shows the flow of the process in Example 1 of this invention. It is a flowchart which shows the processing flow of step S103 in FIG. It is a figure for demonstrating the determination method of the visual weight based on a motion amount, a compression rate, or a quantization error. It is a figure for demonstrating the determination method of the visual weight based on the amount of motion, a compression rate, or a quantization error employ | adopted in an Example. It is a figure which shows the numerical example of visual weight. It is a figure which shows another numerical example of visual weight. It is a figure which shows the number of basic quantization steps. It is a figure which shows the numerical example of a quantization step number. It is a figure which shows another numerical example of the number of quantization steps. It is a figure which shows the example of a coefficient value of 2LL subband. It is a figure which shows the bit plane of 2LL subband of FIG. It is a figure which shows the code structural example of JPEG2000. It is a flowchart which shows the processing flow in Example 2 of this invention. It is a figure which shows the numerical example of the truncation amount. It is a figure which shows another numerical example of the truncation amount. It is a flowchart which shows the processing flow in Example 3 of this invention. 32 is a flowchart showing a process flow of step S308 in FIG. It is a figure which shows the example of a code discard order.

Explanation of symbols

100 Encoding processing means 101 Encoding control means

Claims (16)

In the encoding processing means that performs frame-based encoding processing of an interlaced image, and detects the amount of motion between fields constituting the frame, and based on the amount of motion and the compression rate or quantization error in the encoding processing means An encoding apparatus comprising an encoding control means for controlling quantization of a frequency coefficient ,
Using wavelet transform in the frame-based encoding process,
As the amount of movement between the fields,
1LHmean, 1HL subband coefficient absolute value 1HLmean, 2LH subband coefficient absolute value 2LHmean, 2HL subband coefficient absolute value Average value of 2HLmean
(1LHmean / 1HLmean) / (2LHmean / 2HLmean) ratio, or
Regarding the luminance component, when the entropy coding of 1LH, 1HL, 2LH, and 2HL subbands is performed, the respective code amounts of 1LHcode, 1HLcode, 2LHcode, and 2HLcode
Ratio of (1LHcode / 1HLcode) / (2LHcode / 2HLcode)
An encoding device comprising means for calculating In the encoding processing means that performs frame-based encoding processing of an interlaced image, and detects the amount of motion between fields constituting the frame, and based on the amount of motion and the compression rate or quantization error in the encoding processing means A coding apparatus comprising coding control means for controlling frequency coefficient or code truncation ,
Using wavelet transform in the frame-based encoding process,
As the amount of movement between the fields,
1LHmean, 1HL subband coefficient absolute value 1HLmean, 2LH subband coefficient absolute value average 2LHmean, 2HL subband coefficient absolute value Average value of 2HLmean
(1LHmean / 1HLmean) / (2LHmean / 2HLmean) ratio, or
Regarding the luminance component, the entropy coding of 1LH, 1HL, 2LH, and 2HL subbands for each code amount of 1LHcode, 1HLcode, 2LHcode, and 2HLcode
Ratio of (1LHcode / 1HLcode) / (2LHcode / 2HLcode)
An encoding device comprising means for calculating In the encoding device according to claim 1, coding device, characterized in that the coding control means controls the quantization in each sub-band. In the encoding apparatus according to claim 2, encoding apparatus characterized by coding control means controls the truncation for each sub-band. 4. The encoding apparatus according to claim 3 , wherein the encoding control means controls the subbands in the same resolution level to minimize the quantization of the LH subband. 5. The encoding apparatus according to claim 4 , wherein the encoding control means controls the subbands in the same resolution level so as to minimize truncation of the LH subband. 6. The encoding apparatus according to claim 3, wherein the encoding control means controls the quantization only for the composition level 1 and the composition level 2. 7. The encoding apparatus according to claim 4 or 6 , wherein the encoding control means controls truncation for only the composition level 1 and the composition level 2. 8. The encoding apparatus according to claim 1, wherein the encoding control means changes a visual weight for each subband in the control of quantization. 9. The encoding apparatus according to claim 2, wherein the encoding control means changes the visual weight for each subband in the control of truncation. The encoding device according to claim 2, 4, 6 or 8 ,
(Sum of absolute values of wavelet coefficients included in image quality control unit) / ((number of wavelet coefficients included in image quality control unit) ^ α)) (where 0 <α ≦ 1)
A coding apparatus comprising: means for calculating a masking index, wherein the coding control means reflects masking for each image quality control unit in truncation control using the masking index . 11. The encoding apparatus according to claim 2, wherein the encoding control means changes the visual weight for each image quality control unit in the truncation control.   A control process for controlling the quantization of wavelet coefficients in a frame-based encoding process of an interlaced image using a wavelet transform, and a detection process for detecting a motion amount between fields constituting the frame,
  The control step controls the quantization based on the amount of motion detected in the detection step and the compression rate or quantization error,
  In the detection step, as the amount of movement,
1LHmean, 1HL subband coefficient absolute value 1HLmean, 2LH subband coefficient absolute value average 2LHmean, 2HL subband coefficient absolute value Average value of 2HLmean
(1LHmean / 1HLmean) / (2LHmean / 2HLmean) ratio, or
Regarding the luminance component, the entropy coding of 1LH, 1HL, 2LH, and 2HL subbands for each code amount of 1LHcode, 1HLcode, 2LHcode, and 2HLcode
Ratio of (1LHcode / 1HLcode) / (2LHcode / 2HLcode)
A coding control method characterized by calculating.   A control step for controlling the truncation of wavelet coefficients or codes in a frame-based encoding process of an interlaced image using a wavelet transform, and a detection step for detecting a motion amount between fields constituting the frame,
  The control step controls truncation based on the amount of motion detected in the detection step and the compression rate or quantization error,
  In the detection step, as the amount of movement,
1LHmean, 1HL subband coefficient absolute value average 1HLmean, 2LH subband coefficient absolute value average 2LHmean, 2HL subband coefficient absolute value Average value of 2HLmean
(1LHmean / 1HLmean) / (2LHmean / 2HLmean) ratio, or
Regarding the luminance component, the entropy coding of 1LH, 1HL, 2LH, and 2HL subbands for each code amount of 1LHcode, 1HLcode, 2LHcode, and 2HLcode
Ratio of (1LHcode / 1HLcode) / (2LHcode / 2HLcode)
A coding control method characterized by calculating. The program which makes a computer perform the control process and detection process in the encoding control method of Claim 13 or 14. 15. A computer-readable recording medium on which a program for causing a computer to execute the control step and the detection step in the encoding control method according to claim 13 or 14 is recorded.

Images