JP4737665B2 - Code processing apparatus, code processing method, program, and information recording medium - Google Patents

Description

  The present invention relates to a code processing device (encoder) that generates encoded data from an image and a code processing device (transcoder or parser) that generates new encoded data from encoded data of an image. The present invention relates to a code processing device that performs rate control according to.

  As for human visual characteristics, the sensitivity of the color signal varies depending on the magnitude of saturation. For example, when the saturation of the red signal is small, the distortion of the red signal is visually difficult to understand, but when the saturation of the red signal is large, the distortion of the red signal is visually noticeable. On the basis of this, Patent Document 1 describes an invention for performing nonlinear quantization of a color difference signal by referring to the DC level of the color difference signal after orthogonal transformation in encoding of an image signal including luminance and color difference signals. Yes.

  Further, Patent Document 2 discloses an invention in which nonlinear quantization is performed on color difference components so that the quantization error is visually uniform.

Japanese Patent No. 2903785 JP 2002-44458 A

  In image coding, in order to obtain better subjective image quality under a high compression rate, it is common to use the non-linearity of human visual characteristics. A typical example of this nonlinearity is a phenomenon called contrast masking.

  Contrast masking is an effect that the visual tolerance increases as the signal value itself increases. In encoding using frequency conversion, the coefficient value after conversion corresponds to the signal value. Generally, except for a small coefficient value range, it is said that the allowable quantization error is proportional to the coefficient value α power (0 <α ≦ 1) (experimentally, it is said that α = 0.7) , Α = 1 is the so-called Weber's law). In simple terms, larger coefficients should be quantized more coarsely, and smaller coefficients should be quantized finer.

  However, for example, in the basic specification of JPEG2000 (Part 1), in order to reduce the amount of calculation, only linear quantization is possible for wavelet coefficients (frequency coefficients). (None), non-linear quantization is possible only with the extended specification of JPEG2000 (Part 2).

  Therefore, non-linear quantization is impossible with the encoding within the basic specification of JPEG2000, but with embedded encoding represented by JPEG2000, the entropy code after bit-plane encoding is discarded, thereby quantizing the coefficient. It is possible to obtain the same effect as the quantization (so-called post-quantization). This is because discarding the code for one coefficient bit plane has the same effect as quantizing the coefficient by 2.

  The first object of the present invention is to easily obtain the same effect as nonlinear quantization of frequency coefficients by discarding entropy codes (post-quantization) in an encoding system that does not allow nonlinear quantization of frequency coefficients. ("Reflection of non-linearity"). In addition to an encoder, a transcoder or a parser whose main purpose is to discard an entropy code (post-quantization) for an encoded file once generated is also a suitable application target of the present invention. Therefore, in this specification, an encoder and a transcoder / parser are collectively referred to as a code processing device.

  Next, the inventor has found that when the coefficients are treated as binary values, for example, the visual importance of each binary value bit itself is not uniform. For example, the MSB of a coefficient is the most important bit responsible for the existence of the coefficient itself. This is because when the MSB is lost due to quantization (when it becomes 0), it is normal that the coefficient remains 0 (is lost) even after the inverse quantization at the time of decoding. It is.

  On the other hand, the bits in the vicinity of the LSB of the coefficient (for example, the lower 3 bits of the coefficient) are also important bits in terms of visual characteristics that bear the texture component in the image, although they do not bear the presence of the coefficient itself. This is because if these lower bits are lost, the texture of the image may be lost, giving an over-smoothed impression.

  In contrast to these two examples, it can be said that bits that are neither the MSB nor the low-order bits are relatively less important in terms of visual characteristics. Therefore, if such special characteristics in visual characteristics can be added to the coefficient bits, rate control with higher subjective image quality can be performed. This is the second purpose of the present invention ("reflection of speciality").

  Hereinafter, the invention according to each claim will be described in order.

  First, JPEG2000 is an example of an encoding method suitable as an application target of the present invention, so an outline of JPEG2000 will be described. However, the present invention is not limited to JPEG2000, and a coding unit having a plurality of image quality control units is formed by dividing a frequency coefficient into a plurality of pieces in the absolute value direction, such as bit plane coding in block units. Widely applicable to methods.

  JPEG2000 encoding (compression) processing is generally performed according to the flow shown in FIG. The image is divided into rectangular tiles (number of divisions ≧ 1) and processed in tile units.

  First, each tile is color-converted into components such as luminance and color difference (step 1). At this time, the RGB color image or the like is also subjected to DC level shift to reduce half of its dynamic range.

  The component after transformation (referred to as a tile component) is divided into subbands by two-dimensional wavelet transformation (step 2). That is, it is divided into four subbands abbreviated as LL, HL, LH, and HH by one two-dimensional wavelet transform. Two-dimensional wavelet transform (decomposition) is recursively repeated for the LL subbands, and finally one LL subband and a plurality of HL, LH, and HH subbands are generated. The coefficients of each subband are linearly quantized as required (step 3).

Next, each subband is divided into rectangles called precincts. As shown in FIG. 2, the precinct is a subband divided into rectangles, and roughly represents a position in the image. Precincts for the three subbands HL, LH, and HH are treated as a group of three spatially corresponding ones. However, one precinct obtained by dividing the LL subband is handled as one unit. The precinct can be the same size as the subband. A code block is obtained by further dividing the precinct into rectangles (FIG. 2). Therefore, the physical size order is image ≧ tile> subband ≧ precinct ≧ code block. For the following explanation, FIG. 3 shows the relationship between the decomposition level (the number of wavelet transforms to be performed) and the resolution level.

  After the above division, entropy coding (bit plane coding) called MQ coding is performed on the coefficients for each code block and in the bit plane order (more precisely, in the sub bit plane order) (step 4). The sub bit plane refers to three subsets of bit planes obtained as a result of classifying the bits included in one bit plane into three types. Therefore, in the case of JPEG2000, the code of one sub bit plane of one code block is the minimum code unit.

  Next, from the sub-bit plane codes, only the codes necessary to fit within the desired total code amount (file capacity) are selected (unnecessary codes are discarded (truncated)) to generate packets, so-called rate A process called control is performed (step 5). Then, encoded data (code stream) is generated by arranging the generated packets in a predetermined order and adding necessary tags or tag information (step 6).

  The rate control in step 5 is performed not only to match the total code amount to the target, but also to obtain the maximum PSNR (Peak Signal-to-Noise Ratio) and the highest image quality at that code amount. For this purpose, the code values (importance) are ordered for each code block and each sub-bit plane, and the codes are selected.

When aiming to obtain a good PSNR, the value of the code of each sub-bit plane of each code block is typically
“Increment of pixel value quantization error when the code is discarded / code amount” (A)
Can be expressed as That is, if an error does not increase even if a certain code is discarded, the code is an unnecessary code. When a certain code is maintained, if the error is reduced and the total code amount does not increase, the code is a useful code to be maintained.

  In this way, the index indicating the importance of the sign, which is defined by the increment value of the error amount and the increment value of the code amount, is generally called a distortion slope. The calculation formula of the distortion slope itself is typically the above formula (A), but it varies depending on the purpose.

Now, the code length, which is the denominator of the above formula (A), is an amount obtained as actual data at the time of entropy coding. On the other hand, the increment of quantization error, which is a numerator , is the increment of error in units of code blocks, and is calculated as the increment of square error in units of code blocks when the objective is to obtain the maximum PSNR. (To maximize PSNR, the square error needs to be minimized). That is, the numerator is obtained by multiplying the sum of square errors that increase when bits are discarded for all the coefficients included in the code block by a subband gain (described later). ΔDi / ΔLi in FIG. 4 indicates the distortion slope used to maximize the PSNR.

  Here, it is important that the value of the code is calculated in “bit plane units”. That is, a code in bit plane coding has an attribute of “bit position”, such as a code in which bits in MSB positions are encoded and a code in which bits at LSB positions are encoded. That is, it is possible to weight the code value using the bit position as a parameter, and to select the code based on the weighting result. For example, the operation of lowering the importance of the sign of the upper bits is that a larger number of codes are discarded for a coefficient having a large absolute value (a code block that includes) than when the operation is not performed. Yes, it is a reflection of the contrast masking effect. The distortion slope that has been used in the past is a mathematical calculation value defined by the quantization error and the code length, but by adding consideration to the bit position, distortion and nonlinearity are newly added. It can be reflected in the slope.

  In the above description, the code importance level is calculated in units of code blocks. However, it is also possible to perform rate control by calculating the level of importance for each set of code blocks, for example, sub-bit planes in subband units. It is. In addition, although it is a code block unit, it is possible to calculate the importance level for each bit plane instead of for each sub-bit plane. Which unit is used as the unit of importance calculation is determined by a balance of accuracy, complexity, processing speed, etc. of rate control. Therefore, in the present specification, such units composed of a set of code units (each code block and each sub-bit plane) are collectively referred to as “image quality control unit”.

  Such image quality control is applied in the case of encoding processing, and also when performing so-called “transcoding” or “parsing”, in which a new file with a reduced file size is generated from a generated code file. Is also applicable. In an encoding method (so-called embedded encoding) in which the entire code is formed by concatenating code units, such as JPEG2000, entropy decoding (MQ in the case of JPEG2000) is performed by truncating unnecessary code units after the code file is generated. This is because the file capacity can be reduced without performing decryption. Therefore, in this specification, an encoder that generates a code file from an image and a transcoder or parser that transcodes the generated code file are collectively referred to as a code processing device. In order to control the image quality in the transcoder / parser, for example, the slope for each code block and the code length according to the expression L described later are stored in advance in the code file or in a form corresponding to the code file. However, it may be calculated (estimated) by the transcoder / parser.

  In order to easily reflect the nonlinearity, a normal distortion slope may be multiplied by a nonlinear function using the bit position of the image quality control unit as a parameter. The distortion slope that has been used in the past is a calculated value that is mathematically calculated as the ratio between the quantization error and the code length, and this reflects the nonlinearity (or specialness) in consideration of the bit position. Is not known.

Therefore, in the invention according to claim 1, in the rate control for the code of the encoding scheme in which the frequency coefficient is divided into a plurality in the absolute value direction to form the code unit, a predetermined unit (hereinafter referred to as a set of code units) In the code processing device that selects a code based on the distortion slope that is an index of the importance of the code calculated for each image quality control unit), the image quality control A code processing apparatus including multiplication of a nonlinear function using a division position of a code unit included in a unit as a parameter. The invention according to claim 2 is the code processing device according to claim 1, characterized in that the division is a bit unit of a binary value. According to such a configuration, it is possible to easily reflect nonlinearity in the distortion slope of the image quality control unit and to perform favorable rate control such as subjective image quality.

  Next, in order to easily reflect the peculiarity, a nonlinear function having a parameter such as whether the bit at the bit position of the code unit of the coefficient constituting the image quality control unit (code unit) is MSB or LSB is used. The quantization error that increases due to the discarding of the bit may be multiplied. Since whether or not the bit is an MSB differs for each coefficient, the multiplication is performed for each coefficient. In JPEG2000, the distortion slope numerator is usually the sum of the square errors that increase when the bits in the discarded bit plane position of the coefficient included in the code block are discarded. Therefore, a non-linear function with a parameter such as whether the bit position is MSB or LSB is added to the square error that increases when the bits in the bit plane position to be discarded of the coefficient included in the code block are discarded. What is multiplied is the sum of the code blocks.

Therefore, the invention according to claim 3 provides a predetermined unit (hereinafter referred to as a set of code units) in rate control for a code of a coding scheme in which a frequency unit is divided into a plurality of frequency coefficients in the absolute value direction to form a code unit. In the code processing device that selects a code based on a distortion slope that is an index of the importance of the code calculated for each image quality control unit), the image quality control unit is represented in the calculation formula for the importance of the image quality control unit. A distortion processing formula included in the image processing apparatus includes: a non-linear function multiplication using a parameter as to whether or not a frequency coefficient division part included in the image quality control unit is a specific part. is there. The invention according to claim 4 is the code processing apparatus according to claim 3, wherein the division is a bit unit of a binary value. According to such a configuration, the above-mentioned special property can be easily reflected in the distortion slope of the image quality control unit, so that favorable rate control such as subjective image quality can be performed.

In the inventions according to claims 1 to 4 , how the nonlinearity is given depends on what purpose (effect) is to be obtained. For example, if the objective is to obtain higher subjective image quality, the higher the bit position is, or when the bit is MSB, the non-linear function takes a smaller value in order to use a masking effect. Good. On the other hand, if the objective is to obtain an image quality with as many edges as possible, rather than high or low subjective image quality (for example, if you want to specify the criminal's human phase with a security camera, etc., contour, wrinkle, mole) Such as edge information is important), the higher the bit position (ie, the higher the probability that the bit is the MSB that bears the edge), or the larger the nonlinear function when the bit is the MSB What should I do?

The invention according to claim 5 is the code processing apparatus according to claim 2, wherein the nonlinear function takes a smaller value as the divided bit position of the code unit included in the image quality control unit is higher. The code processing device. According to this configuration, it is possible to perform rate control that reflects the masking effect.

The invention according to claim 6 is the code processing apparatus according to claim 4, wherein the nonlinear function is obtained when a divided part of the frequency coefficient included in the image quality control unit is the most significant bit (MSB). Takes a minimum value. According to this configuration, it is possible to perform rate control that does not place importance on edge maintenance.

The invention according to claim 7 is the code processing apparatus according to claim 2, wherein the nonlinear function takes a larger value as the divided bit position of the code unit included in the image quality control unit is higher. This is a characteristic code processing apparatus. According to this configuration, it is possible to perform rate control that places importance on maintaining edges.

The invention according to claim 8 is the code processing device according to claim 4, wherein the nonlinear function is used when a divided part of the frequency coefficient included in the image quality control unit is the most significant bit (MSB). Takes the maximum value. According to this configuration, it is possible to perform rate control that places importance on maintaining edges.

  As mentioned above, it is desirable to maintain the bit plane near the LSB when you want to preserve the texture.

Therefore, the invention according to claim 9 is the code processing device according to claim 2, wherein the nonlinearity is obtained when a divided bit position of a divided unit of the code unit included in the image quality control unit is lower than a predetermined bit position. The function is a code processing device characterized by not taking a minimum value. According to this configuration, it is possible to perform rate control that places importance on the maintenance of texture.

The invention according to claim 10 is the code processing device according to claim 4, wherein the divided part of the frequency coefficient included in the image quality control unit is a bit lower than a predetermined bit position. The nonlinear function is a code processing device characterized by not taking a minimum value. According to this configuration, it is possible to perform rate control that places importance on the maintenance of texture.

  Now, since a still image is literally still, it is easy to observe details, so a higher resolution and more high-frequency components are required than a moving image. On the other hand, moving images are displayed as a series of screens, and the person who sees them is more sensitive to differences between screens than details in the screen. In particular, when the error of the low frequency component is large, the difference increases over a wide area between the screens, and the difference may be observed as undulation. Therefore, with regard to moving images, it is often desirable to deal with low frequency components more importantly than still images, and conversely, handling with high frequency components neglected. The neglect of the high-frequency component is the neglect of the edge, which is nothing but reducing the importance of the sign of the upper bits.

Therefore, the invention according to claim 11 is the code processing device according to any one of claims 1 to 4, wherein the nonlinear function is made different depending on whether the processing target is a moving image or a still image. This is a code processing device characterized by the above. According to such a configuration, it is possible to perform rate control in consideration of the difference in characteristics between moving images and still images.

The invention according to claim 12 is the code processing device of the invention according to claim 2, wherein when the processing target is a moving image, the higher the divided bit position of the code unit included in the image quality control unit, the more nonlinear The code processing apparatus is characterized in that the function takes a small value. According to such a configuration, it is possible to perform rate control in consideration of moving image characteristics.

The invention according to claim 13 is the code processing device of the invention according to claim 4, wherein when the processing target is a moving image, the division part of the frequency coefficient included in the image quality control unit is the most significant bit (MSB). ), The non-linear function takes a minimum value. According to such a configuration, it is possible to perform rate control in consideration of moving image characteristics.

The invention according to claim 17 is a program causing a computer to function as a coding apparatus of the invention according to any one of claims 1 to 13, the invention according to claims 1 to 13 by using a computer A code processing device can be realized.

The invention according to claim 18 is a computer-readable information recording medium in which a program for causing a computer to function as the code processing apparatus according to any one of claims 1 to 13 is recorded. The code processing device according to the first to thirteenth aspects of the present invention can be realized using the above.

As described above, according to the inventions according to claims 1 to 16 , in an encoder, a transcoder, or a parser, rate control with good subjective image quality and edge maintenance can be performed without using nonlinear quantization of frequency coefficients. Rate control that does not place importance, rate control that places importance on maintaining edges, rate control that places importance on maintaining textures, rate control that takes into account the difference between video and still image characteristics, and rate control that takes into account video characteristics It becomes. Further, according to the inventions according to claims 17 and 18 , the code processing device according to the inventions according to claims 1 to 13 can be easily realized using a computer.

FIG. 5 is a block diagram of a code processing device according to an embodiment of the present invention. The code processing apparatus according to the present embodiment is an encoder that generates a code file from an image in accordance with the JPEG2000 algorithm, and performs processing corresponding to steps 1, 2, 3 , 4 , 5 , and 6 shown in FIG. The process block 100, 101, 102, 103, 104, 105 for execution is comprised. The basic processing operation of this code processing apparatus is the same as that of a conventional JPEG2000-compliant encoder, but the present invention is applied to the calculation of the distortion slope in the processing block 103, and the processing content is specific to the present invention. Then, the distortion slope obtained in the processing block 103 is applied to the rate control in the processing block 104. FIG. 13 shows the flow of processing by the processing blocks 103, 104, and 105.

  Prior to the description along the flowchart of FIG. 13, a method for calculating the code length, error calculation, and distortion slope necessary for rate control will be described.

  First, a method for calculating the code length of each sub-bit plane in JPEG2000 will be described. This calculation is performed in process block 103.

JPEG2000 entropy coding is performed in bit plane order in units of code blocks. For example, the nested for loop
for (component) {
for (resolution) {
for (subband) {
for (Precinct) {
for (code block) {
for (subbit plane) {
MQ coding
}
}
}
}
}
}
(For the physical positional relationship of each for loop, see FIG. 2 and FIG. 3).

  In this case, the MQ codes generated by the MQ encoding are generated in the order of the sub bit planes as shown in FIG. 7, and are sequentially concatenated as shown in FIG. The MQ encoder outputs the total L (k) of the code lengths generated up to that point (exactly, in units of bytes) when the encoding of each sub-bit plane k is completed. Therefore, by storing this total value and taking the difference between the code length total value in a certain sub-bit plane and the code length total value up to the sub-bit plane immediately before it, the code for each sub-bit plane is obtained. The length can be calculated.

In rate control (processing block 104), unnecessary portions are discarded (truncated) from the lower sub-bitplane side in the sequential codes of FIG. 8, and only necessary codes are collected to generate packets. However, in order to correctly decode the MQ code in which the code below the sub-bit plane k is discarded, a code corresponding to the sum of the code lengths generated up to the sub-bit plane k plus 5 bytes is required at the maximum. (Since the code is output from the MQ encoder in byte units, there are codes less than bytes remaining inside the encoder, etc.). So here,
“When each sub-bit plane has been encoded, the total length L (k) +5 of the code lengths generated up to that point” Expression L
And the code length ΔL for each sub-bit plane is calculated by taking this difference.

Next, a method of calculating the square error increment when each sub bit plane is discarded will be described. As shown in FIG.
・ The absolute value of wavelet coefficient before quantization is y
・ The wavelet coefficient after truncating p bit planes (quantized by 2 ^p and converted to an integer) is v _p
・ After truncating p bit planes, wavelet coefficients after dequantization
y _p ^
・ The wavelet coefficient y divided by 2 ^p is the fractional part of the wavelet coefficient v _p ^~
And In truncation, wavelet coefficients after quantization are converted into integers by a floor function, but are considered by dividing them into an integer part v _p and a decimal part v _p ^~ . Since a sub bit plane is a subset of bit planes, when a bit plane including the sub bit plane is truncated, each sub bit plane is also truncated.

The square error increment Δ (D ² ) when truncating bitplanes from p to p + 1 is
Δ (D ² ) = (y _{(p + 1)} ^ −y) ² − (y _p ^ −y) ²
This is expressed as a function of the fractional part v _p ^~ .

First, depending on the magnitude relationship between the magnitude of the coefficient y and the truncation amount, one coefficient (case A) v _p = 0 and v _{p + 1} = 0 (At this time Δ (D ² ) = 0)
(Case B) v _p = 1 and v _{p + 1} = 0 (The MSB with the coefficient y is truncated on the p + 1th sheet)
(Case C) v _p ≧ 2 and _{v p + 1 ≧ 1 (p} + 1 th in even still survive MSB of the coefficient y)
One of the following three holds (be careful when v _{p + 1} = 0).

From the definition of the decimal part above, v _p ^~ = y / 2 ^p −v _p , so
y = 2 ^p (v _p + v _p ^~ ) (1)
Similarly, from v _{p + 1} ^~ = y / 2 ^{(p + 1)} −v _{p + 1} ,
y = 2 ^{p + 1} (v _{p + 1} + v _{(p + 1)} ^~ ) (2)
However, when v _{p + 1} = 0,
y = 2 ^{p + 1} · v _{(p + 1)} ^~ (3)

Also, the coefficient after inverse quantization is decoded at the center of the section,
y _p ^ = 2 ^p (v _p +0.5) (where v _p ≠ 0) (4)
However, since the coefficient that became 0 after quantization is decoded to 0,
y _p ^ = 0 (where v _p = 0) (5)
Similarly,
y _{(p + 1)} ^ = 2 ^{p + 1} (v _{p + 1} +0.5) (where v _{p + 1} ≠ 0) (6)
y _{(p + 1)} ^ = 0 (where v _{p + 1} = 0) (7)
It becomes.

As in (Case A), with respect to the attention coefficient, when v _p = 0, the error does not increase even if the number of truncation is increased from p to p + 1.

As in (Case B), when v _p = 1 and v _{p + 1} = 0, the square error increment Δ (D ² ) is
Δ (D ² ) = (y _{(p + 1)} ^ −y) ² − (y _p ^ −y) ²
= (0−y) ² − (y _p ^ −y) ² (Equation (7))
＝ y ² − (y _p ^ −y) ²
= (2 ^{p + 1} · v _{(p + 1)} ^~ ) ² − (2 ^p (v _p +0.5) −2 ^p (v _p + v _p ^~ )) ² (Equation (3), (4), (1))
＝ (2 ^{p + 1}・ v _{(p + 1)} ^~ ) ² − (2 ^p (0.5−v _p ^~ )) ²
= 2 ^2p {(2 ・ v _{(p + 1)} ^~ ) ² − (v _p ^~ −0.5) ² }
Where v _p ^~ = 2 ・ v _{(p + 1)} ^~ − | _2 ・ v _{(p + 1)} ^~ _ | (8)
In Equation (8), | _x_ | represents a floor function of x (a function that replaces the real number x with an integer that does not exceed x and is closest to x).

When v _{p + 1} = 0, | _2 · v _{(p + 1)} ^~ _ | ≧ 1, so v _p ^~ = 2 · v _{(p + 1)} ^~ −1
∴Δ (D ² ) = 2 ^2p {(2 ・ v _{(p + 1)} ^~ ) ² − (2 ・ v _{(p + 1)} ^~ −1.5) ² } ≡2 ^2p・ S (v _{(p + 1)} ^~ )
It becomes.

In case C, the square error increment Δ (D ² ) is
Δ (D ² ) = (y _{(p + 1)} ^ −y) ² − (y _p ^ −y) ²
= (2 ^{p + 1} (v _{p + 1} +0.5) −2 ^{p + 1} (v _{p + 1} + v _{(p + 1)} ^to )) ² − (2 ^p (v _p +0.5) −2 ^p ( v _p + v _p ^~ )) ²
(Formula (6), (2), (4), (1))
= (2 ^{p + 1} (0.5−v _{(p + 1)} ^~ )) ² − (2 ^p (0.5−v _p ^~ )) ²
= 2 ^2p {(2 ・ v _{(p + 1)} ^~ −1) ² − (v _p ^~ −0.5) ² }
= 2 ^2p {(2 ・ v _{(p + 1)} ^~ −1) ² − (2 ・ v _{(p + 1)} ^~ − | _2 ・ v _{(p + 1)} ^~ _ | −0.5) ² } ≡2 ^2p・ M (v _{(p + 1)} ^~ )
It becomes.

From the above, the sum ΣΔ (D ² ) of the square error increments in the target code block in each sub bit plane is the sum in the code block of the (Case B) or (Case C) increment, and the wavelet coefficient y ^Let h be the fractional part after dividing by 2 ^{p + 1} .
ΣΔ (D ² ) = 2 ^2p {S (h) + M (h)}
= 2 ^2p · Σ {(2h) ² − (2h−1.5) ² } (when v _p = 1 and MSB)
+2 ^2p · Σ {(2h−1) ² − (2h− | _2h_ | −0.5) ² } (v _p ≧ 2, other than MSB)
.... Formula D1
It becomes.

However, when p = 0, truncation is not performed, and in the case of 5x3 conversion, it is lossless.
Δ (D ² ) = (y _{(p + 1)} ^ −y) ² − (y _p ^ −y) ² = (y _{(p + 1)} ^ −y) ²
ΣΔ (D ² ) = 2 ^2p {S (h) + M (h)}
= 2 ^2p · Σ {(2h) ² } (v _p = 1, MSB)
+2 ^2p · Σ {(2h−1) ² } (v _p ≧ 2, other than MSB)
.... Formula D2
It becomes.

  Note that an error generated in a coefficient is converted into an error of a pixel value through an inverse wavelet transform, but the square error of the pixel value can be approximately treated as a constant multiple of the square error of the coefficient. The constant depends on the subband and is called a subband gain. Further, when component conversion to YCbCr or the like is performed on the original image and wavelet transform is performed on the component, the component conversion gain Gc that is a magnification of a square error when the component is inversely converted to the RGB pixel value. (Gc takes a value that does not depend on the subband).

  From the above, for each code block belonging to subband b, the distortion slope of each sub bit plane can be calculated as follows.

(Gb × Gc × STEPb ² ) × ΣΔ (D ² ) / ΔL... S0
Here, Σ is taken for all coefficients in the code block, Gb is the subband gain of subband b, and STEPb is the number of quantization steps for subband b (STEPb = 1 if not quantized). Therefore, if this is applied to all the sub-bit planes in the code block, a distortion curve (a polygonal line shape as shown in FIG. 4) relating to the code block can be obtained.

  Normally, the distortion curve has a monotonously decreasing shape that protrudes downward as shown in FIG. 4, but an exceptionally concave portion may be formed as shown in FIG. In this case, as shown in FIG. 10, a slope obtained by integrating a plurality of sub-bit planes is recalculated and shaped to be monotonously decreased. This process is also an exception process, and it is easy to determine the unevenness by comparing the inclinations, and it is also easy to obtain an integrated inclination, and the detailed procedure is omitted.

  By the above processing, so-called Lagrangian rate control is performed after generating a distortion curve that decreases monotonously for all subbands and all code blocks.

That is, assuming that the number of sub-bit planes to be truncated for a certain code block i is t, the maintained code length is R (t), and the square error generated in the coefficient is ΣD ² (t).
In order to maximize the PSNR under the desired code amount R,
Σ {ΣD ² (t)} → minimum (the first sum is for all code blocks, the next sum is for all coefficients in code block i)
And ΣR (t) −R = 0 (the sum is taken for all code blocks)
Should be satisfied. At this time, the Lagrangian function Lag (t, λ) where the undetermined multiplier is λ is
Lag (t, λ) = Σ {ΣD ² (t)} + λ {ΣR (t) −R}
It becomes.

Here, the condition that the first term Σ {ΣD ² (t)} takes an extreme value is that the partial differentiation of the Lagrangian function is 0, so that ΣΣ D ² (t) / ∂t + λ {∂ R (t) / ∂t} = 0
∴∂ΣD ² (t) / ∂R (t) = − λ
And ΣR (t) = R
It is.

  Therefore, for each distortion curve, the sign of the sub-bit plane having a slope larger than an arbitrary value λ (> 0) as shown in FIG. 11 (the portion on the left side of the contact point with the line segment of slope −λ in FIG. 10). ) Only, discard other codes. Then, the sum R (λ) of code lengths corresponding to the value λ is calculated for all subbands and all code blocks.

  Here, if R (λ) is larger than the desired value R, λ is increased and R (λ) is recalculated. Conversely, if R (λ) is smaller than the desired value, λ is decreased and R (λ ) Is recalculated. This is repeated until R (λ) becomes “desired value R ± allowable error”. The search for λ can be easily performed by a so-called binary search method.

The code thus generated has the best PSNR in order to minimize ΣD ² (t) in the capacity R (λ). More specifically, in addition to the sum of the code lengths maintained as described above, a header and a main header of a packet generated from the codes are required, so these header lengths are also taken into account.

  As described above, the rate can be controlled by the Lagrange's undetermined multiplier method as long as the distortion slope can be defined. In the present invention, the following slope is used.

The number of the bit plane to which each sub bit plane belongs (LSB number is 0) is p, and for example, a non-linear function with respect to p is, for example, f1 (p) = (0.8) ^p
And That is, f1 (p) is monotonically decreasing with respect to p (Claims 1 and 2 ),
Slope of the image quality control unit = f1 (p) × (Gb × Gc × STEPb ² ) × ΣΔ (D ² ) / ΔL
... S1
And In this way, by lowering the importance of the high-order bit code, it is possible to discard a larger number of codes for a coefficient having a large absolute value (including a code block) than when the operation is not performed. it can.

  FIG. 12 is a schematic explanatory diagram regarding the above-described operation, and shows slope values before and after application of the present invention for three code blocks A, B, and C. FIG. The absolute values of the coefficients included in the code blocks A, B, and C are A <B <C on average. This is suggested by the fact that the maximum value of the slope is A <B <C. This is because the larger the absolute value of the coefficient, the larger the error when the MSB is truncated, resulting in a larger maximum value of the slope.

  The upper part of FIG. 12 shows the slope before multiplying by f1 (p) (before application of the present invention). For example, if the slope portion of 2000 or less is discarded, the code of the shaded portion is discarded. On the other hand, the slope obtained by multiplying f1 (p) (the slope after application of the present invention) is the lower stage. For example, if the slope portion of 1000 or less is discarded, the code of the shaded portion is discarded.

  As apparent from comparison of the shaded portions in FIG. 12, after the present invention, the truncation amount of the code block having a large absolute value on the average increases and the truncation amount of the other code block decreases. Therefore, the contrast masking effect can be used.

  FIG. 13 is a flowchart for explaining the processing of the processing blocks 103, 104, and 105 in the code processing device according to this embodiment. Steps 1300 to 1306 are processing steps in the processing block 103, and perform MQ coding and code length and slope calculation for each sub bit plane. Step 1307 is a rate control processing step in the processing block 104, and step 1308 is a code formation processing step in the processing block 105.

Steps 1300-1306 are nested for loops like
for (component) {
for (resolution) {
for (subband) {
for (Precinct) {
for (code block) {
for (subbit plane) {
MQ coding, holding the code length of each sub-bit plane (step 1305)
}
for (subbit plane) {
Calculate and hold slope (step 1306)
}
}
}
}
}
}
It is clear that

  In step 1307, λ is determined by a known binary search using the calculated slope, unnecessary MQ codes are discarded, and packets are formed using necessary MQ codes. In the next step 1308, these packets are arranged, and necessary tags and tag information are added to generate a JPEG2000 code file.

In step 1306, as described above, for each sub-bit plane, ΔL is calculated using equation L, error ΣΔD ² is calculated using equations D1 and D2, and f1 ( Calculate and hold the slope using p) and equation S1.

In the above, the function f1 (p) = (0.8) ^p is simply used. However, in order to reflect the contrast masking effect, it may be a non-linear function and may be defined in a table format. The function is typically monotonically decreasing with respect to p (Claim 5 ). Strictly speaking, since the contrast masking effect is established in the range where the signal value is equal to or greater than the predetermined value Th, for example, a function that takes a value 1 when p <3 and monotonously decreases when p ≧ 3 may be used.

In the above description, each sub bit plane for each code block is collectively multiplied by f1 (p). However, there is a method of multiplying for each coefficient. This is because the error Δ (D ² ) is calculated for each coefficient and summed as described above. Therefore
f2 (p) = 1 (when p is not the MSB of the coefficient)
f2 (p) = 0.8 (when p is the MSB of the coefficient)
F2 (p)
ΣΔ (D ² ) = 2 ^2p {S (h) + M (h)}
^{= 2 2p · Σ [{(} 2h) 2 - (2h-1.5) 2} × f2 (p)] (v case of _{p = 1, MSB, f2 (} p) = 0.8)
+2 ^2p · Σ [{(2h−1) ² − (2h− | _2h_ | −0.5) ² } × f2 (p)]
(v _p ≧ 2, other than MSB)
... Formula D3
However, when p = 0,
ΣΔ (D ² ) = 2 ^2p {S (h) + M (h)}
= 2 ^2p · Σ [{(2h) ² } × f2 (p)] (v _p = 1, f2 (p) = 0.8 for MSB)
+2 ^2p · Σ [{(2h−1) ² } × f2 (p)] (v _p ≧ 2, other than MSB)
... Formula D4
Assuming that ΣΔ (D ² ) is calculated, the code block containing more MSBs has a smaller error increase amount ΣΔ (D ² ) and a smaller slope (claims 3, 4, 6 ). As a result, compared to a case where f2 (p) is not multiplied, a code block including more MSBs is more likely to be truncated. Therefore, although the reproducibility of the edge is lowered, a lot of codes are assigned to the non-edge, that is, the low-frequency component accordingly, and a stable image quality can be obtained particularly for moving images. It can be said that the same effect as contrast masking can be produced in that a code is preferentially assigned to low frequency components.
Therefore, according to the second embodiment, in step 1306, for each sub-bit plane, ΔL is calculated using equation L, error ΣΔD ² is calculated using f2 (p) and equations D3, D4, and equation S0 is used. The slope is calculated and held (claims 3, 4, 6 ).

Now, unlike the above, if you want to give priority to edge reproducibility,
f3 (p) = (1.1) ^p
And That is, f3 (p) monotonically increases with respect to p (Claim 7 ).
Slope of the image quality control unit = f3 (p) × (Gb × Gc × STEPb ² ) × ΣΔ (D ² ) / ΔL
... Formula S3
And In this way, by relatively increasing the importance of the high-order bit code, it is possible to perform a smaller number of code discards for a coefficient having a large absolute value (including a code block) than when the operation is not performed. it can.

Therefore, according to the third embodiment, in step 1306, for each sub-bit plane, ΔL is calculated using equation L, error ΣΔD ² is calculated using equations D1 and D2, and f3 (p) and equation S3 are used. The slope is calculated and held (claims 1, 2, 7 ).

Also, in the method of multiplying by coefficient,
f4 (p) = 1 (when p is not the MSB of the coefficient)
f4 (p) = 1.2 (where p is the MSB of the coefficient)
F4 (p)
ΣΔ (D ² ) = 2 ^2p {S (h) + M (h)}
= 2 ^2p · Σ [{(2h) ² − (2h−1.5) ² } × f4 (p)] (v _p = 1, MSB, f4 (p) = 1.2)
^{+2 2p · Σ [{(2h} -1) 2 - (2h- | _2h_ | -0.5) 2} × f4 (p)] (v otherwise _p ≧ 2, MSB)
... Formula D5
However, when p = 0,
ΣΔ (D ² ) = 2 ^2p {S (h) + M (h)}
= 2 ^2p · Σ [{(2h) ² } × f4 (p)] (v _p = 1, f4 (p) = 1.2 for MSB)
+2 ^2p · Σ [{(2h−1) ² } × f4 (p)] (v _p ≧ 2, other than MSB)
... D6
If ΣΔ (D ² ) is calculated as follows, the error increase amount ΣΔ (D ² ) increases and the slope increases as the code block includes more MSBs. As a result, code blocks that contain more MSBs are not truncated and edge reproducibility is improved.

Therefore, according to the fourth embodiment, in step 1306, for each sub-bit plane, ΔL is calculated using equation L, error ΣΔD ² is calculated using f4 (p) and equations D1, D2, and equation S0 is used. The slope is calculated and held (claims 3, 4, 8 ).

In addition, unlike the above, when it is desired to maintain the texture of the original image, for example, f5 (p) = 1 (0 ≦ p ≦ 2)
f5 (p) = 0.8 (p> 2)
F5 (p) is prepared. That is, f5 (p) is set so as not to take a minimum value with respect to p equal to or less than a predetermined value (claim 9 ). And
Slope of the image quality control unit = f5 (p) × (Gb × Gc × STEPb ² ) × ΣΔ (D ² ) / ΔL
... Formula S5
And In general, texture is converted into a frequency coefficient having a small value (for example, less than an absolute value of 8). Thus, by relatively increasing the importance of the sign of the lower bits, the absolute value is smaller than that in the case where the operation is not performed. It is possible to perform a small code discard for a coefficient with a small value (including a code block) (the slope of the upper bit of the subband with the higher decomposition level is lower than the slope of the lower bit of the subband with the lower level). However, this effect occurs mainly when the compression ratio is low).

Therefore, according to the fifth embodiment, in step 1306, for each sub-bit plane, ΔL is calculated using equation L, error ΣΔD ² is calculated using equations D1 and D2, and f5 (p) and equation S5 are used. The slope is calculated and held (claims 1, 2, 9 ).

Also, in the method of multiplying by coefficient,
f6 (p) = 1 (when the coefficient is less than 7)
f6 (p) = 0.8 (when the coefficient is 8 or more)
F6 (p)
ΣΔ (D ² ) = 2 ^2p {S (h) + M (h)}
= 2 ^2p · Σ [{(2h) ² − (2h−1.5) ² } × f6 (p)] (when v _p = 1 and MSB)
+2 ^2p · Σ [{(2h−1) ² − (2h− | _2h_ | −0.5) ² } × f6 (p)]
(v _p ≧ 2, other than MSB)
... Formula D7
However, when p = 0,
ΣΔ (D ² ) = 2 ^2p {S (h) + M (h)}
= 2 ^2p · Σ [{(2h) ² } x f6 (p)] (when v _p = 1, MSB)
+2 ^2p · Σ [{(2h−1) ² } × f6 (p)] (v _p ≧ 2, other than MSB)
... Formula D8
If ΣΔ (D ² ) is calculated as follows, the amount of increase in error ΣΔ (D ² ) increases and the slope increases as the code block includes many coefficients less than 7. As a result, code blocks containing many coefficients less than 7 are not truncated, and texture reproducibility is improved.

Therefore, according to the sixth embodiment, in step 1306, for each sub-bit plane, ΔL is calculated using equation L, error ΣΔD ² is calculated using f6 (p) and equations D7, D8, and equation S0 is used. The slope is calculated and held (claims 3, 4, 10 ).

The non-linearity can also be switched for the purpose of emphasizing edges for still images and emphasizing low frequencies for moving images (claims 11 and 12 ).

For example, according to the seventh embodiment, when the encoding target is a still image, in step 1306, ΔL is calculated for each sub-bit plane using equation L, error ΣΔD ² is calculated using equations D1 and D2, and f1 (p) and equation S1 are used to calculate and hold the slope, but when the encoding target is a moving image, in step 1306, ΔL is calculated for each sub-bit plane using equation L, and error ΣΔD is calculated using equations D1 and D2. ² is calculated and the slope is calculated and stored using f3 (p) and equation S3.

Further, according to the eighth embodiment, when the encoding target is a still image, ΔL is calculated for each sub-bit plane in step 1306 using equation L, and error ΣΔD ² is calculated using f2 (p) and equations D3 and D4. , And the slope is calculated and stored using Expression S0. When the encoding target is a moving image, ΔL is calculated using Expression L for each sub-bit plane in Step 1306, and f4 (p) and Expression D5 are calculated. , D6, the error ΣΔD ² is calculated, and the slope is calculated and held using the equation S0.

  As described above, the present invention can also be applied to rate control when transcoding a once generated code file.

  FIG. 14 is a simplified block diagram for explaining a code processing apparatus (transcoder / parser) according to the second embodiment of the present invention that performs such transcoding. The code processing apparatus according to the present embodiment includes a processing block 200 that performs code analysis on a code file to be processed, and a processing block 201 that discards unnecessary codes and regenerates packets (rate control) using necessary codes. The processing block 202 generates a new code file by performing code reformation from the regenerated packet.

FIG. 15 is a flowchart for explaining the processing flow of the code processing device. Steps 2100 to 2105 are processing steps in the processing block 200 of the code analysis, and the content thereof is a nested for loop as follows.
for (component) {
for (resolution) {
for (subband) {
for (Precinct) {
for (code block) {
for (subbit plane) {
Read the slope corresponding to the code of the code block and the required code length (result of expression L) (step 2105)
}
}
}
}
}
}
It is clear that

  Then, in step 2106, the processing block 201 determines λ by a known binary search, calculates the number of sub bit planes to be maintained for each code block and the necessary code length, discards unnecessary codes from the packet, Correspondingly, the packet header is regenerated to reproduce the rate-controlled packet. In step 2107, the processing block 202 arranges the regenerated packet and regenerates a new code file.

The data necessary for regenerating the packet header is
(a) Whether the packet is empty
(b) Code block included in the packet
(c) Number of zero bit planes in each code block
(d) Number of sub bit planes included in each code block
(e) MQ code length of each code block, (c) reuses the value written in the packet header of the original code, and the others are the number of sub bit planes and code length reflecting the result of rate control Etc. will be used.

  As described above, the transcoder / parser can perform code length calculation, error calculation, distortion slope calculation, and the like necessary for rate control from the code, and perform rate control using the information. Aspects are also included in this embodiment.

  For subband gain, see “J. Katto and Y. Yasuda,“ Performance evaluation of subband coding and optimization of its filter coefficients, ”“ Journal of Visual Communication and Image Representation, vol. 2, pp. 303-313, Dec. Details on “Gsbc” in “1991”, FIG. 21 shows subband gains of 5 × 3 conversion and 9 × 7 conversion. In JPEG2000, component conversion called RCT is used for 5x3 conversion, and component conversion called ICT is used for 9x7 conversion, and the gain at the time of inverse conversion is also shown in FIG.

The code processing apparatus (encoder) according to the present invention described with reference to FIG. 5 is a computer in which a CPU 150, a memory 151, a hard disk device (HDD) 152, etc., as shown in FIG. However, it can also be realized by software. That is, a program for causing the computer to function as the processing blocks 100 to 105 in FIG. 5 is loaded into the memory 151 and executed by the CPU 150, thereby causing the computer to function as the code processing device according to the present invention. The flow of processing in this case is generally as follows.
(1) One frame of the original image stored in the HDD 152 is read into the memory 151 by a command from the CPU 150.
(2) The CPU 150 reads an image on the memory 151 and performs encoding in the JPEG2000 format. The present invention is applied to rate control (code discard) at that time.
(3) The CPU 150 writes the generated encoded file in another area on the memory 151.
(4) The encoded file is stored in the HDD 152 according to a command from the CPU 150.

  It is obvious that the code processing apparatus (transcoder / parser) according to the present invention described with reference to FIG. 14 can also be realized by a program using a computer. A program for realizing the code processing apparatus according to the present invention by a computer, and various information recordings readable by a computer, such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor storage element, on which the program is recorded (Storage) media are also encompassed by the present invention.

Moreover, since it is clear that the above description is also the description of the embodiment of the code processing method according to claims 14 , 15, and 16 , the description thereof will not be repeated.

  Further, as is clear from the above description, not only JPEG2000 but also a code unit is formed by dividing a frequency coefficient into a plurality of pieces in the absolute value direction, such as bit plane coding in block units, and a plurality of image quality controls. The present invention can be similarly applied to a code processing apparatus using an arbitrary encoding method having a unit.

It is explanatory drawing of the flow of an encoding (compression) process of JPEG2000. It is a figure which shows the relationship between the image in JPEG2000, a tile, a subband, a precinct, and a code block. It is a figure which shows the relationship between a decomposition level and a resolution level. It is explanatory drawing of a distortion slope. 1 is a block diagram of a code processing device (encoder) according to an embodiment of the present invention. It is a block diagram for description of the aspect using a computer. It is explanatory drawing of the encoding order of the sub bit plane in 1 code block. It is a figure for demonstrating the calculation method of the code amount of a sub bit plane. It is a figure for explanation of a symbol used for calculation of an error and a slope. It is a figure for demonstrating the recessed part of a distortion slope, and its process. It is a figure for explanation of Lagrange rate control. It is a figure for demonstrating the effect | action by this invention application. It is a flowchart for demonstrating the process in the code processing apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram of a code processing device (transcoder / parser) according to a second embodiment of the present invention. It is a flowchart for demonstrating the process in the code processing apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows a subband gain and an inverse component conversion gain.

Explanation of symbols

100 Processing Block for DC Level Shift and Color Conversion 101 Processing Block for Wavelet Transform 102 Processing Block for Quantization 103 Processing Block for Computation of Distortion Slope for Bit-Plane Coding and Rate Control 104 Rate Processing block for control and packet generation 105 Processing block for code formation 200 Processing block for code analysis 201 Processing block for rate control 202 Processing block for code reconstruction

Claims (18)

In rate control for coding scheme codes that divide the frequency coefficient into multiple values in the absolute value direction to form code units, calculation is performed for each predetermined unit (hereinafter referred to as image quality control unit) that is composed of a set of code units. In a code processing device that selects a code based on a distortion slope that is an index of the importance of a generated code,
  A code processing apparatus characterized in that a distortion slope calculation formula of an image quality control unit includes a multiplication of a nonlinear function using a division position of a code unit included in the image quality control unit as a parameter. The code processing device according to claim 1, wherein the division is a bit unit of a binary value. In rate control for coding scheme codes that divide the frequency coefficient into multiple values in the absolute value direction to form code units, calculation is performed for each predetermined unit (hereinafter referred to as image quality control unit) that is composed of a set of code units. In a code processing device that selects a code based on a distortion slope that is an index of the importance of a generated code,
  Non-linearity with the calculation formula of the importance of the image quality control unit as a parameter in the calculation formula of the distortion slope included in the image quality control unit and whether the frequency coefficient division part included in the image quality control unit is a specific part A code processing apparatus comprising a multiplication of a function. 4. The code processing device according to claim 3, wherein the division is a bit unit of a binary value. The code processing device according to claim 2, wherein the nonlinear function takes a smaller value as the divided bit position of the code unit included in the image quality control unit is higher. 5. The code processing apparatus according to claim 4, wherein the non-linear function takes a minimum value when a divided part of a frequency coefficient included in the image quality control unit is a most significant bit (MSB). apparatus. The code processing device according to claim 2, wherein the nonlinear function takes a larger value as the divided bit position of the code unit included in the image quality control unit is higher. 5. The code processing apparatus according to claim 4, wherein the non-linear function takes a maximum value when a divided part of a frequency coefficient included in the image quality control unit is a most significant bit (MSB). apparatus. 3. The code processing apparatus according to claim 2, wherein the nonlinear function does not take a minimum value when a divided bit position of a divided unit of the code unit included in the image quality control unit is lower than a predetermined bit position. A code processing device. 5. The code processing apparatus according to claim 4, wherein the non-linear function does not take a minimum value when a divided part of a frequency coefficient included in the image quality control unit is a bit lower than a predetermined bit position. A code processing device. 5. The code processing device according to claim 1, wherein the nonlinear function is made different depending on whether a processing target is a moving image or a still image. 3. The code processing apparatus according to claim 2, wherein when the processing target is a moving image, the nonlinear function takes a smaller value as the divided bit position of the code unit included in the image quality control unit is higher. Processing equipment. 5. The code processing device according to claim 4, wherein when the processing target is a moving image, the nonlinear function has a minimum value when a divided portion of the frequency coefficient included in the image quality control unit is the most significant bit (MSB). A code processing device characterized by that. In rate control for coding scheme codes that divide the frequency coefficient into multiple values in the absolute value direction to form code units, calculation is performed for each predetermined unit (hereinafter referred to as image quality control unit) that is composed of a set of code units. In a code processing method for selecting a code based on a distortion slope that is an index of importance of a generated code,
  A code processing method characterized in that a distortion slope calculation formula of an image quality control unit includes multiplication of a nonlinear function using a division position of a code unit included in the image quality control unit as a parameter. In rate control for coding scheme codes that divide the frequency coefficient into multiple values in the absolute value direction to form code units, calculation is performed for each predetermined unit (hereinafter referred to as image quality control unit) that is composed of a set of code units. In a code processing method for selecting a code based on a distortion slope that is an index of importance of a generated code,
  Non-linearity with the calculation formula of the importance of the image quality control unit as a parameter in the calculation formula of the distortion slope included in the image quality control unit and whether the frequency coefficient division part included in the image quality control unit is a specific part A code processing method comprising multiplication of a function. 16. The code processing method according to claim 14, wherein the division is a bit unit of a binary value. A program that causes a computer to function as the code processing device according to claim 1. 14. A computer-readable information recording medium in which a program for causing a computer to function as the code processing device according to claim 1 is recorded.

Images