JP4379869B2 - Moving image processing apparatus, program, and information recording medium - Google Patents

Description

  The present invention relates to processing of encoded moving images, and more particularly, to frame thinning for reducing the amount of code of moving images encoded within a frame.

Video coding methods can be broadly divided into two types: intra-frame coding and inter-frame coding. Intra-frame coding is a method in which individual frames of a moving image are independently coded, and the code of each frame is connected to generate a moving image code. Typical examples are DV, Motion-JPEG,
Motion-JPEG2000. Inter-frame coding is a method in which a plurality of continuous frames of a moving image are grouped into one group, coding is performed for each group, and a code for each group is generated to generate a moving image code. Are MPEG1, MPEG2, and MPEG4.

  The encoding of each frame includes field encoding that encodes an odd-numbered field and an even-numbered field image that are interlaced images, and frame-based encoding that encodes odd-numbered frames and even-numbered frames at once. There is.

  The moving image targeted by the present invention is a moving image obtained by frame-based encoding an interlaced image independently for each frame.

  Known documents related to the present invention include, for example, Patent Document 1 and Patent Document 2.

  In Patent Document 1, on the encoding side, the motion between frames is detected by block matching between adjacent frame images, etc., and only the frame in which the motion changes is encoded (the frame with uniform motion is thinned), and the decoding side Describes a technique for synthesizing frame images thinned out by motion compensation frame interpolation processing using a decoded frame image as a reference image.

  Patent Document 2 describes a “comb shape” related to frame-based encoding of an interlaced image, in which a subject moves between fields constituting a frame.

Japanese Patent No. 2919211 JP 2002-64830 A

  Nowadays, there is an increasing need to extract codes of some frames of a moving image without decoding the encoded moving image. For example, if a frame with little motion is thinned out from a moving image on a server on the network and a moving image consisting only of a code with a large amount of motion is created and transmitted, the load on the network and the transfer time can be reduced. it can. In addition, since the code of a frame with a large motion is stored, the motion of the decoded moving image is not so unnatural.

  Since the intra-frame encoded moving image is configured as a set of codes for each frame, the code can be thinned out for each frame. The problem is how to select a frame with less motion from the frame group constituting the moving image without decoding the moving image.

  Therefore, an object of the present invention relates to a moving image obtained by frame-based encoding of an interlaced image independently for each frame, and selects a frame with less motion without decoding it. It is an object of the present invention to provide a novel moving image processing apparatus and method for generating a new moving image with a reduced code amount by deleting the code.

The invention of claim 1
  A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  Means for calculating the code amount of the 1LH subband based on the information of the portion of the header of the code of each frame in which information relating to the code amount of the 1LH subband is described, and the code amount calculated by the means is The moving image processing apparatus includes means for evaluating a larger amount of motion as a larger value and selecting a frame with a smaller amount of motion.

The invention of claim 2
  A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  The code amount of the 1LH subband and the code amount of the 1HL subband are determined based on the information of the header in the code of each frame and the information about the code amount of the 1LH subband and the code amount of the 1HL subband. Using the means for calculating each and the code amount calculated by the means,
  Code amount ratio = (Code amount of 1LH subband) / (Code amount of 1HL subband)
And a means for selecting a frame with a small amount of motion by evaluating that the amount of motion is larger as the code amount ratio calculated by (1) is larger.

The invention of claim 3
  A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  Based on the information in the header in the code of each frame, which describes the information about the code amount of 1LH subband, the code amount of 1HL subband, the code amount of 2LH subband, and the code amount of 2HL subband. A code amount for 1LH subband, a code amount for 1HL subband, a code amount for 2LH subband, and a code amount for 2HL subband, and a code amount calculated by the means,
  Code quantity ratio = [(1LH subband code quantity / 1HL subband code quantity) / (2LH subband code quantity
                Code amount / 2HL subband code amount)]
And a means for selecting a frame with a small amount of motion by evaluating that the amount of motion is larger as the code amount ratio calculated by (2) is larger.

The invention of claim 4
  A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  1LH subband code in units of blocks smaller than the subband based on the header information in the code of each frame and the information of the portion describing the information about the code amount of the 1LH subband and the code amount of the 1HL subband. A unit for calculating the amount and the code amount of the 1HL subband, and the code amount of the block unit corresponding to the positional relationship calculated by the unit,
  Block code amount ratio = (Code amount of 1LH subband block unit / 1HL subband)
                      Code amount in blocks)
A code amount ratio obtained by summing up the block code amount ratio calculated for all blocks, evaluating that the larger the code amount ratio is, the larger the amount of motion is, and selecting a frame with a small amount of motion. This is a featured moving image processing apparatus.

The invention of claim 5
  A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  Based on the information in the header in the code of each frame, which describes the information about the code amount of 1LH subband, the code amount of 1HL subband, the code amount of 2LH subband, and the code amount of 2HL subband. Means for calculating the code amount of 1LH subband, the code amount of 1HL subband, the code amount of 2LH subband, and the code amount of 2HL subband in units of blocks smaller than the subband, and the means , Using the code amount of the block unit corresponding to the positional relationship,
  Block code amount ratio = [(1LH subband block code amount / 1HL subband
      Code amount in block units) / (code amount in block units of 2LH subbands / 2HL)
      Subband block code amount)]
A code amount ratio obtained by summing up the block code amount ratio calculated for all blocks, evaluating that the larger the code amount ratio is, the larger the amount of motion is, and selecting a frame with a small amount of motion. This is a featured moving image processing apparatus.

According to a sixth aspect of the present invention, there is provided the moving image processing apparatus according to the second or third aspect , wherein the code amount is corrected in accordance with a difference in truncation amount between subbands.

The invention according to claim 7 is the moving image processing apparatus according to claim 2 or 3 , wherein the code amount is corrected in accordance with a difference in truncation amount between subbands and a difference in truncation amount between frames. A moving image processing apparatus.

According to an eighth aspect of the present invention, there is provided the moving image processing apparatus according to the fourth or fifth aspect of the invention, wherein the code amount is corrected in accordance with a difference in truncation amount between blocks.

According to a ninth aspect of the present invention, in the moving image processing apparatus according to the fourth or fifth aspect of the present invention, the code amount is corrected in accordance with a difference in a truncation amount between blocks and a difference in a truncation amount between frames. An image processing apparatus.

According to a tenth aspect of the present invention, in the moving image processing apparatus according to any one of the second to ninth aspects, the code amount is corrected in accordance with a difference in the number of linear quantization steps between subbands. A moving image processing apparatus.

According to an eleventh aspect of the present invention, in the moving image processing apparatus according to any one of the second to ninth aspects of the present invention, the code amount is determined by the difference in the number of linear quantization steps between subbands and the number of linear quantization steps between frames. A moving image processing apparatus is characterized in that correction is performed according to a difference.

The invention of claim 12 is the moving image processing apparatus according to any one of claims 1 to 11 , wherein the calculated code amount is a code amount of a luminance component. .

According to a thirteenth aspect of the present invention, in the moving image processing apparatus according to any one of the first to twelfth aspects of the present invention, the means for selecting the frame selects a frame whose evaluated motion amount is smaller than a predetermined value. Is a moving image processing apparatus.

According to a fourteenth aspect of the present invention, in the moving image processing apparatus according to any one of the first to twelfth aspects, the means for selecting the frame selects a predetermined number of frames in ascending order of the evaluated motion amount. This is a featured moving image processing apparatus.

A fifteenth aspect of the present invention is the moving image processing apparatus according to the thirteenth or fourteenth aspect of the present invention, wherein the means for selecting the frame excludes the first frame of the moving image from the selection target. is there.

A sixteenth aspect of the present invention is the moving image processing apparatus according to the thirteenth, fourteenth, or fifteenth aspects of the present invention, further comprising means for deleting the code of the frame selected by the means for selecting the frame from the moving image. is there.

The invention according to claim 17 is a program that causes a computer to function as each means of the moving image processing apparatus according to any one of claims 1 to 16 .

The invention of claim 18 is a computer-readable information recording medium on which the program of the invention of claim 17 is recorded.

  The invention according to each of the above claims will be described in order below.

  First, “comb” in frame-based encoding of interlaced images will be described with reference to FIG. (A) in FIG. 1 is an image of the (n) field, (b) is an image of the (n + 1) field after 1/60 second, and (c) is a composite of the images of the two fields (interlaced image). Examples of frame images obtained in this way are shown. In frame-based encoding, a frame image obtained by combining two fields is encoded as it is.

  Now, when the subject moves to the right as shown in the drawing between two yields, the left and right edge portions of the subject are shifted in a comb shape by a plurality of pixels for each scanning line on the frame image composed of two fields. (D) is an enlargement of the comb-shaped edge portion, and the length L of the lateral edge corresponds to the amount of movement of the subject between two fields (within the frame).

  That is, the greater the movement of the subject, the greater the amount of lateral edge of the frame image. When shooting with a video camera, the movement of the subject is overwhelmingly large in the horizontal direction, so the edge amount of the horizontal edge as seen at the edge of the comb is the amount of movement of the frame. It is reasonable to use it as an indicator.

  On the other hand, the length of the vertical edge of the comb-shaped edge portion generated by the lateral movement of the subject is substantially constant regardless of the amount of movement of the subject. Therefore, it is also reasonable to use the ratio of the horizontal edge amount and the vertical edge amount (= horizontal edge amount / vertical edge amount) as an index of the frame motion amount.

  When an encoding method using frequency transform (wavelet transform or the like) is used for frame-based encoding, generally, the amount of horizontal edge and the amount of vertical edge are reflected in the code amount in the frame code. The As will be described later, in JPEG 2000 or the like, such a code amount can be obtained from header information in the code.

  Based on this consideration, the invention of claim 1 calculates, for each frame, a code amount that reflects a lateral edge amount based on header information of a code of each frame in a moving image obtained by frame-based encoding of an interlaced image. The frame motion amount is evaluated based on the calculated code amount, and a frame with a small motion amount is selected. According to a second aspect of the present invention, in a moving image encoded using frame-based encoding, a code amount reflecting a horizontal edge amount based on header information of a code of each frame and a code reflecting a vertical direction Calculate the amount, evaluate the motion amount of the frame based on the ratio of the code amount (= code amount reflecting the horizontal edge amount / code amount reflecting the vertical edge amount), and select a frame with a small motion amount It is what.

  Now, in an encoding method (for example, JPEG2000) that encodes a frequency conversion coefficient for each subband (frequency band), the edge amount in the horizontal direction and the edge amount in the vertical direction are each reflected in the code amount of a specific subband. .

In view of this, the invention of claim 1 or 2 reflects the code amount of the 1LH subband reflecting the amount of horizontal edge, or the code amount and vertical edge amount of 1LH subband reflecting the amount of horizontal edge. The code amount of 1HL subband is to be calculated.

  In the case of an encoding method that performs multi-resolution subband division (for example, JPEG2000), the edge component is remarkably reflected in the subband of the highest resolution level (the subband with the least number of frequency conversions applied), and the resolution level is lowered. The degree to which the edge component is reflected decreases, and the edge component is hardly reflected below a certain resolution level. In particular, a horizontal edge component with one line interval appearing at the comb-shaped edge portion due to movement of the subject is reflected in a specific subband at the highest resolution level, and is not substantially reflected in a subband at a lower resolution level.

  In addition, in a frame in which a small subject moves, when viewed in subband units, the change in the code amount reflecting the lateral edge amount (by the comb shape) due to the movement of the subject may not be so large.

The invention according to claim 4 or 5 more accurately evaluates the amount of motion of a small subject by calculating a code amount that reflects a horizontal edge amount or an edge amount in each vertical and horizontal direction in units of blocks smaller than subbands. It is something that can be done.

The present invention assumes the case where two-dimensional wavelet transform is used for frame-based encoding. A representative example of such an encoding method is JPEG2000. In a Motion-JPEG2000 moving image, each frame is encoded by JPEG2000. Here, an outline of JPEG2000 will be described.

  FIG. 2 shows a basic flow of JPEG2000 compression (encoding) / decompression (decoding) processing.

  First, compression (encoding) processing will be described. For example, a color image composed of three RGB components is divided into one or more non-overlapping tiles for each component, and processing is performed for each tile of each component. First, for each tile, a DC level shift and a component conversion (color conversion) to luminance / color difference components are performed, and then a two-dimensional wavelet transform (discrete wavelet transform) is performed for each tile of each component. ) Is made. The wavelet coefficients are linearly quantized as necessary for each subband, and then entropy-coded in units of bitplanes (more precisely, one bitplane has three subbitplanes (encoding Pass) and encoded). Then, unnecessary codes are discarded (truncated), packets are generated by collecting necessary codes, packets are arranged in a predetermined order, and necessary tags and tag information are added. It is formed.

  In JPEG2000, a reversible wavelet transform called 5 × 3 transform and an irreversible wavelet transform called 9 × 7 transform can be used. Linear quantization of wavelet coefficients is applicable only when 9 × 7 wavelet transform is used. When linear quantization is applied, a bit plane composed of coefficients after linear quantization of wavelet coefficients is entropy encoded. When linear quantization is not performed, unnecessary bit-plane codes are discarded or even necessary bit-planes are encoded (hereinafter referred to as truncation). When 5 × 3 wavelet transform is used, linear quantization cannot be applied, and the specification is such that the code is discarded by truncation.

  The decompression (decoding) process is just the reverse of the compression process. The codestream is decomposed into codestreams for each tile of each component, entropy-decoded in bit plane units, the decoded wavelet coefficients are dequantized, and then subjected to two-dimensional inverse wavelet transform, and then inverse component transform (Reverse color conversion) and reverse DC level shift are performed to restore the original RGB pixel values.

  A forward transform formula and an inverse transform formula of a wavelet transform called a 5 × 3 transform and a wavelet transform called a 9 × 7 transform are shown below.

<5x3 conversion>
(Forward conversion)
C (2i + 1) = P (2i + 1) −floor ((P (2i) + P (2i + 2)) / 2) [step1]
C (2i) = P (2i) + | _ (C (2i-1) + C (2i + 1) +2) / 4_ | [step2]
(Inverse transformation)
P (2i) = C (2i) −floor ((C (2i-1) + C (2i + 1) +2) / 4) [step1]
P (2i + 1) = C (2i + 1) + floor ((P (2i) + P (2i + 2)) / 2) [step2]
Here, floor (x) represents a floor function of x (a function that replaces the real number x with an integer that does not exceed x and is closest to x).

<9x7 conversion>
(Forward conversion)
C (2n + 1) = P (2n + 1) + α * (P (2n) + P (2n + 2)) [step1]
C (2n) = P (2n) + β * (C (2n-1) + C (2n + 1)) [step2]
C (2n + 1) = C (2n + 1) + γ * (C (2n) + C (2n + 2)) [step3]
C (2n) = C (2n) + δ * (C (2n-1) + C (2n + 1)) [step4]
C (2n + 1) = K * C (2n + 1) [step5]
C (2n) = (1 / K) * C (2n) [step6]
(Inverse transformation)
P (2n) = K * C (2n) [step1]
P (2n + 1) = (1 / K) * C (2n + 1) [step2]
P (2n) = X (2n) -δ * (P (2n-1) + P (2n + 1)) [step3]
P (2n + 1) = P (2n + 1) -γ * (P (2n) + P (2n + 2)) [step4]
P (2n) = P (2n) -β * (P (2n-1) + P (2n + 2)) [step5]
P (2n) = P (2n + 1) -α * (P (2n) + P (2n + 2)) [step6]
However, α = -1.586134342059924
β = -0.052980118572961
γ = 0.882911075530934
δ = 0.443506852043971
K = 1.230174104914001

  A process of performing wavelet transform called 5 × 3 transform in two dimensions (vertical direction and horizontal direction) on a monochrome image of 16 × 16 pixels will be described with reference to FIGS.

  As shown in FIG. 3, the XY coordinates are taken, and the pixel value of a pixel whose Y coordinate is y is expressed as P (y) (0 ≦ y ≦ 15) for a certain X coordinate. In JPEG2000, a coefficient C (2i + 1) is obtained by first applying a high-pass filter in the vertical direction (Y-coordinate direction) around an odd-numbered pixel (y = 2i + 1). Next, a low-pass filter is applied to the pixels whose Y coordinates are even (y = 2i) to obtain a coefficient C (2i) (this is performed for all X coordinates). The step 1 expression in the 5 × 3 conversion forward conversion expression represents a high-pass filter, and the step 2 expression represents a low-pass filter.

  Note that there may be no adjacent pixel at the edge of the image with respect to the central pixel. In this case, the pixel value is compensated by mirroring.

  For the sake of simplicity, if the coefficient obtained by the high-pass filter is denoted by H and the coefficient obtained by the low-pass filter is denoted by L, the image in FIG. 3 is arranged as shown in FIG. Is converted to

  Subsequently, a high-pass filter is applied to the coefficient array shown in FIG. 3 in the horizontal direction centering on an odd-numbered (x = 2i + 1) coefficient, and then an even-numbered (x = 2i) coefficient. (This is performed for all the Y coordinates. In this case, P (2i) and the like in the conversion formula are read as those representing coefficient values.)

  For simplicity, LL is obtained by applying a low-pass filter centered on the L coefficient, HL is obtained by applying a high-pass filter centered on the L coefficient, and a low-pass filter is applied centering on the H coefficient. 4 is converted into a coefficient array as shown in FIG. 5, if the coefficient obtained is expressed as LH and the coefficient obtained by applying a high-pass filter around the H coefficient as HH. Here, the coefficient group to which the same symbol is attached is called a subband.

  With the above, one wavelet transform (one decomposition (decomposition)) is completed, and only the LL coefficients are collected (collected for each subband as shown in FIG. 6 and only the LL subband is taken out). An “image” having half the resolution of the image is obtained (in this way, classifying the coefficients for each subband is called deinterleaving, and arranging in the state shown in FIG. 5 is called interleaving).

In the second wavelet transform, the LL subband may be regarded as an original image and the same transformation as described above may be performed. When the second wavelet transform is performed and the coefficients are deinterleaved, a schematic diagram 7 is obtained.
Here, in FIGS. 6 and 7, the prefixes 1 and 2 of the coefficient indicate how many times the wavelet transform has been obtained, and are called the decomposition level. In the wavelet transform, this decomposition level corresponds to the frequency band. Further, FIG. 8 shows the definition of the resolution level that is almost opposite to the composition level.

  In such inverse transformation of 5 × 3 wavelet transform, an inverse low-pass operation is first performed on the interleaved coefficient array as shown in FIG. 5 centering on a coefficient whose X coordinate is an even number (x = 2i). Apply a filter, and then apply an inverse high-pass filter centered on a coefficient whose X coordinate is an odd number (x = 2i + 1) (this is performed for all y). The step 1 expression in the 5 × 3 inverse conversion expression represents an inverse low-pass filter, and the step 2 expression represents an inverse high-pass filter. As in the case of forward conversion, there may be no adjacent coefficient for the central coefficient at the edge of the image. In this case, the coefficient value is appropriately compensated by mirroring.

  As a result, the coefficient array in FIG. 5 is converted (inversely converted) into a coefficient array as shown in FIG. Next, in the vertical direction, apply an inverse low-pass filter centered on the coefficient whose Y coordinate is an even number (y = 2i), and then apply an inverse high-pass filter centered on the coefficient whose Y coordinate is an odd number (y = 2i + 1). If this is done (for all X coordinates), one wavelet inverse transformation is completed, and the image of FIG. 3 is returned (reconstructed). If wavelet transformation is performed a plurality of times, FIG. 3 is regarded as an LL subband, and similar inverse transformation may be repeated using other coefficients such as HL.

  As described above, in the 5 × 3 wavelet transform, one low-pass filter output (low-pass coefficient) is obtained using five pixels, and one high-pass filter output (high-pass coefficient) is obtained using three pixels. It is a conversion. On the other hand, the 9 × 7 wavelet transform is a conversion in which 9 pixels are used to obtain one low-pass filter output (low-pass coefficient), and 7 pixels are used to obtain one high-pass filter output (high-pass coefficient). is there. As described above, the main difference between the 5 × 3 conversion and the 9 × 7 conversion is the difference in the filter range, and the low pass filter is applied to the center of the even position and the high pass filter is applied to the center of the odd position. . Therefore, FIGS. 4 to 7 apply to the 9 × 7 conversion as well.

  FIG. 9 shows the relationship among tiles, subbands, precincts, and code blocks. In JPEG2000, a subband is divided into precincts. The precinct is a subband divided into rectangles (of a size that can be specified by the user), and the precincts of the three subbands HL, LH, and HH form a group of three (however, the LL subbands). The precinct is divided into a single unit.), And roughly represents a position in the image. The precinct can be the same size as the subband. A code block is obtained by further dividing the precinct into rectangles (of a size that can be specified by the user).

  Although the sub-band coefficients are bit-plane encoded as described above, more specifically, the bit-plane encoding is performed in units of code blocks.

  A packet is a collection of a part of the code extracted from all the code blocks included in the precinct (for example, a collection of codes from the MSB of all the code blocks to the third bit plane). . The contents of the packet may be “empty” in terms of sign.

  When packets of all precincts (= all code blocks = all subbands) are collected, a part of the code of the entire image (for example, the code of the wave plane coefficient of the entire image from the MSB to the third bit plane) You can, but this is called a layer. Since the layers are roughly part of the code of the bit plane of the entire image, the image quality increases as the number of layers to be decoded increases. A layer is a unit of image quality. If all the layers are collected, it becomes the sign of all the bit planes throughout the image.

  FIG. 10 shows an example of the layer configuration when the number of wavelet transform layers (the number of decomposition levels) = 2 and the precinct size = subband size. Packets included in some of the layers are illustrated in FIG. A packet surrounded by a thick line in FIG. 11 is a packet.

  In this example, the precinct size = subband size, and the code block having the same size as the size of the printinct is adopted. Therefore, the subband at the decomposition level 2 is divided into four code blocks. The level 1 subband is divided into 9 code blocks. Since the packet is a unit of precinct, when precinct = subband, the packet extends across the HL to HH subbands.

  Here, the packet is “a collection of code blocks extracted and collected”, and unnecessary codes need not be generated as packets. For example, the code of the lower bit plane as included in the layer 9 shown in FIG. 10 is usually discarded (truncated).

  The order of the packets on the code stream is called the progression order. In JPEG2000, the LRCP, RLCP, PRCL, PCRL, and CPRL are combined according to the combination of resolution (R), precinct (P), component (C), and layer (L). Five types of progressive order are defined.

For example, for LRCP:
for (layer) {
for (resolution) {
for (component) {
for (Precinct) {
Encoding: Place packet
When decoding: Interpret packets}
}
}
}
Packets are arranged (when encoded) and interpreted (when decoded) in the order determined by the nested for loop. In other progressive orders as well, the same nested for loop determines the arrangement order during packet encoding and the interpretation order during decoding.

  A packet consists of a code that is the main body and a packet header. The packet header describes the code length and the like, but the layer number and resolution level are not described in the packet header. At the time of decoding, a for loop as described above is formed from the progressive order specified by the COD marker in the main header of the code stream, and the resolution of which layer of which bucket is determined by which for loop the packet is handled in. Will be determined. The number of layers, the number of resolutions, and the number of precincts can be read from the COD marker in the main header, and the number of components can be read from the SIZ marker in the main header (the number of precincts can be calculated because the precinct size is known from the COD marker). After that, the number of packets can be counted as long as the packet breaks can be identified. Since the length of the code included in the packet is written in the packet header, the break can be counted.

  As described above, in the case of JPEG2000 codes, the code lengths of code blocks included in a packet can be obtained by reading the packet headers of all packets based on the progressive order. If the sum of the code lengths of the code blocks of all packets included in the subband is taken for each subband, the code amount for each subband can be obtained without decoding. This is one of the specific examples of calculating the code amount from the header information without decoding, as described in relation to claims 1 and 2.

  Now, among the subbands of the two-dimensional wavelet transform, the horizontal edge component, that is, the high frequency component in the vertical direction is most greatly reflected in the coefficients of the 1LH subband (decomposition level 1, vertical high frequency, horizontal low frequency), On the other hand, the vertical edge component, that is, the high frequency component in the horizontal direction is most greatly reflected in the coefficient of 1HL subband (decomposition level 1, vertical low frequency, horizontal high frequency). Accordingly, the amount of edge in the horizontal direction is most greatly reflected in the code amount of the entropy code of the 1LH subband, and the amount of edge in the vertical direction is most reflected in the code amount of the entropy code of 1HL subband.

In view of the above, the invention of claim 2 is the ratio of the code amount of the 1LH subband to the code amount of the 1HL subband, that is, the code amount ratio when using, for example, JPEG2000 using two-dimensional wavelet transform for frame-based encoding. = Ru der intended to assess the amount of movement of the frame by 1LH subband coding amount / 1HL subband coding amount.

Although it is difficult to expect an evaluation accuracy equivalent to the case where the code amount ratio of the 1LH and 1HL subbands is used, it is also possible to use only the code amount of the 1LH subband for the motion amount evaluation instead of the code amount ratio. . Such an aspect of evaluating the amount of motion is included in the invention of claim 1 .

  The amount of movement of the subject, that is, the “comb-shaped” lateral edge amount is reflected in the 1LH subband code amount, but the lateral edge amount of the subject other than the comb shape is also reflected in the 1LH subband code amount. Therefore, for example, in a frame including a subject having a long horizontal edge, even if the subject moving speed is slow, the code amount ratio (= 1LH subband code amount / 1HL subband code amount) is a large value. May take. On the other hand, the code amount of the 2LH subband, which is one level lower in resolution than the 1LH subband, is almost unaffected by the horizontal edges of the “comb” every other line, and is exclusively influenced by the high-frequency components in the vertical direction other than the comb. Therefore, it takes a stable value regardless of the movement of the subject.

Therefore, the ratio of 1LH subband code amount / 1HL subband code amount is A,
Consider the ratio A / B, where B is the ratio of 2LH subband code amount / 2HL subband code amount. When the subject moves at high speed, the value of the numerator A increases because the amount of comb-shaped lateral edges increases, but the value of the denominator B does not increase, so the value of the ratio A / B increases. When the same subject is stationary or moving at a low speed, the value of numerator A decreases, but the value of denominator B does not change, so the value of ratio A / B decreases. The same applies to the case where the subject has a long edge in the horizontal direction. Therefore, when the ratio A / B is used, that is, a value obtained by normalizing the code amount ratio of 1LH and 1HL with the code amount ratio of 2LH and 1HL, for example, a subject having a long edge in the horizontal direction can be processed at high speed. The case of moving and the case of moving at a low speed can be more reliably distinguished.

In view of such consideration, the invention of claim 3 is the code amount ratio corresponding to the ratio A / B, that is, the code amount ratio = (1LH subband code amount / 1HL subband code amount) / (2LH subband).
Code amount / 2HL subband code amount)
Based on, Ru der intended to assess the amount of movement of the frame more reliably.

  If the moving subject is smaller than the image size, the increase in the code amount ratio in subband units is small even if the subject moves at high speed. This is because the amount of horizontal edge of the entire frame is reflected in the code amount of the 1LH subband. If a similar code amount ratio is calculated in units of blocks smaller than the subband, the amount of movement of the subject is reflected sharply in the code amount ratio of the block corresponding to the position of the subject.

Based on such considerations, the inventions of claims 4 and 5 try to calculate a code amount ratio that more reliably reflects the movement of a small subject, and to evaluate the motion amount of the frame based on the calculated code amount ratio. As a block which is a code amount calculation unit in the fourth and fifth aspects of the invention, a JPEG 2000 precinct or one or a plurality of code blocks can be used.

The code amount ratio in the invention of claim 4 can be expressed as follows.
Code amount ratio =
Σ (code amount of block i of 1 LH subband / code amount of block j of 1HL subband)
However, the coefficients of block i and block j are derived from substantially the same pixel position, and the sum is taken for all 1HL subband blocks having at least 1HL subband blocks derived from substantially the same pixel position.

The code amount ratio in the invention of claim 5 can be expressed as follows.
Code amount ratio =
Σ (code amount of block 1 of 1LH subband / code amount of block j of 1HL subband) / (code amount of block k of 2LH subband / code amount of block 1 of 2HL subband)
However, the coefficients of block i and block j are derived from substantially the same pixel position, and the coefficients of block k and block l are also derived from the substantially same pixel position. The coefficients of block i and block k include coefficients derived from substantially the same pixel position, and the coefficients of block j and block l also include coefficients derived from substantially the same pixel position. The sum is taken for at least all 1HL subband blocks having 1HL subband blocks derived from approximately the same pixel location.

It is obvious that the invention of the superordinate concept including the inventions of claims 4 and 5 is the invention of claims 2 and 3 .

  As described above, in JPEG 2000, not all bit planes are necessarily encoded. As described in connection with FIG. 1 and FIG. 10, it is common that truncation of unnecessary lower bitplane codes or coefficients is performed during the encoding process.

  For each frame of a moving image, there is no problem if all the bit planes are coded, but otherwise, the number of truncated bit planes (truncation amount) may vary depending on the subband. You have to be careful. For example, the truncation amount is not always the same between the 1HL subband and the 1LH subband. Therefore, it is desirable to use a code amount for calculating the code amount ratio, in which the influence of the difference in the truncation amount due to the subband is corrected.

  For example, if the 1HL subband is truncated for 3 bit planes and the 1LH subband is truncated for 2 bitplanes, the least significant bit plane of the code of the 1LH subband (the bit plane that was not truncated) The code corresponding to the least significant bit plane) should be excluded from the code amount calculation, and the code amount should be the same truncation amount as that of the 1HL subband. Alternatively, the code of the least significant bit plane of the 1HL subband is regarded as being equal to the code of the most significant bit plane of the truncated bit planes of the 1HL subband, and the 1HL subband is calculated when calculating the code amount. It is also possible to add the code of the least significant bit plane it has twice. Such truncation amount adjustment can of course be performed in units of sub-bit planes.

According to the sixth aspect of the invention, the code amount of the 1LH subband and the code amount of the 1HL subband are corrected according to the difference in the truncation amount between the subbands, and the motion amount is more accurately evaluated. is there. Such correction for the code amount may be performed when the code amount is calculated or when the code amount ratio is calculated.

  Even within the same subband, the truncation amount is not always uniform in all blocks. When the code amount ratio of the block unit is used for the motion amount evaluation, the difference in the truncation amount between the blocks should be considered. For example, if the nth block has been truncated by 3 bit planes and the n + 1 block has been truncated by 2 bit planes, the least significant bit plane of the code of the n + 1 block (not truncated) The code for the lowest bit plane of the bit planes) should be excluded from the code amount calculation, and the code amount at the same truncation amount as that of the nth block should be obtained. Alternatively, the code of the least significant bit plane that the nth block has is regarded as substantially the same as the code of the most significant bit plane of the truncated bit planes that the nth block does not have, and the nth block It is also possible to add the code of the least significant bit plane of the block twice. Alternatively, the code of the least significant bit plane of the (n + 1) th block is regarded as substantially the same as the code of the most significant bit plane of the truncated bit planes not possessed by the (n) th block, and is adjacent to the (n + 1) th block. It is also possible to substitute the code amount of, or use the code amount of neighboring blocks. Of course, such a truncation amount can be adjusted in units of sub-bit planes.

The invention of claim 8 intends to more accurately evaluate the motion amount by correcting the code amount in units of blocks in accordance with the difference in the truncation amount between the blocks. Such correction for the code amount may be performed when the code amount is calculated or when the code amount ratio is calculated.

  Now, even if the number of truncated bit planes in each subband is uniform, if each subband is subjected to different linear quantization, the difference needs to be considered. The number of quantization steps at the time of linear quantization is not necessarily the same between the 1HL subband and the 1LH subband. For example, if 1HL subband is linearly quantized with 8 steps and 1LH subband with 4 steps, the number of bitplanes in 1LH subband should be one more than the number of bitplanes in 1HL subband. (8/4 = 2 to the first power, which corresponds to one bit plane), the least significant bit plane of the code of the 1LH subband (the least significant bit plane of the non-truncated bit plane) The code for bit plane) should be excluded from the code amount calculation. Alternatively, the code of the least significant bit plane included in the 1HL subband is regarded as substantially equal to the code of the most significant bit plane of the truncated bit planes not included in the 1HL subband, and the 1HL subband has at the time of code amount calculation. It is also possible to add the code of the least significant bit plane twice.

  Such adjustment of the linear quantization amount can also be performed in units of sub-bit planes. This is because the number of sub-bit planes n corresponds to the ratio of the number of steps of linear quantization being 2 to the power of n / 3 (one bit plane = three sub-bit planes = 2 to the first power). That is, when the ratio of the number of quantization steps between subbands is X, it corresponds to the difference in the number of 3 log (X) subbit planes (the bottom of the log is 2). It is only necessary to adjust the code amount by the amount.

The invention of claim 10 intends to more accurately evaluate the amount of motion by correcting the amount of code in accordance with the difference in the number of linear quantization steps between subbands. Such correction for the code amount may be performed when the code amount is calculated or when the code amount ratio is calculated.

  So far, corrections for differences in the amount of truncation or the number of quantization steps have been considered only within individual frames. However, if there is a difference in the amount of truncation or the number of quantization steps between the frames of the moving image, it is desirable to consider the difference. In particular, when ordering the motion amount evaluated by the code amount ratio for each frame in ascending order and selecting a certain number of frames from the smaller motion amount, when calculating the code amount for obtaining the code amount ratio, In addition, it is desirable to perform correction to eliminate the influence of the truncation amount or the difference in the number of quantization steps between frames. For this purpose, for example, the amount of code related to the amount of truncation or the number of quantization steps in each frame may be corrected on the basis of the amount of truncation or the number of quantization steps in the first frame of the moving image.

According to the seventh , ninth , and eleventh aspects of the present invention, by correcting the code amount so as to eliminate the influence of the truncation amount between the frames or the difference in the number of linear quantization steps, the motion amount is more accurately determined. It is something to be evaluated.

In the case of a color moving image, the code consists of the code of the luminance component and the code of the color difference component, but since the code amount of the color difference component has a large fluctuation range due to the difference in the color of the subject, the luminance as the code amount reflecting the edge amount It is generally appropriate to use the code amount of the component. This is considered in the invention of claim 12 .

When thinning out a frame of a moving image, if a frame with a large amount of motion is thinned out, the unnatural motion of the moving image at the time of decoding becomes prominent. The invention of claim 16 is intended to reduce the code amount of a moving image without thinning the motion of the moving image at the time of decoding by thinning out a frame with a small amount of motion from the moving image. .

  As a method of selecting the thinned frame, there are a method (1) for selecting a frame whose motion amount is smaller than the reference regardless of the number of frames, and a method (2) for selecting a predetermined number of frames in order from the frame having the smallest motion amount. . The former selection method (1) can greatly reduce the code amount of a moving image having many frames with a smaller amount of motion than the reference, and is effective when priority is given to code amount reduction. There is also a concern that the number of thinned-out frames is excessive for moving images with a small total number. The latter selection method (2) has an advantage that it is possible to easily avoid the disadvantage that the number of thinned frames is excessive. However, in a moving image having a large amount of motion, a frame having a relatively large amount of motion is thinned out. There is a fear.

The invention of claim 13 assumes a case where a thinned frame is selected by the selection method (1). The invention of claim 14 assumes a case in which a thinned frame is selected by the selection method (2).

A selection method (3) that combines the selection method (1) and the selection method (2) is also possible. That is, this is a method in which a predetermined number of frames are selected in ascending order of motion amount, from among frames whose motion amount is below the reference. In the image processing apparatus according to any one of claims 1 to 12, a frame may be selected by the selection method (3), and the image processing apparatus and method having such a configuration are also included in the present invention.

When a moving image is displayed as an icon as a still image, the first frame is often used for the display, and therefore the first frame is preferably excluded from the thinning target. According to the fifteenth aspect of the invention, since the first frame of the moving image is excluded from the selection target, it is possible to avoid thinning out the first frame.

In other words, in the moving picture coding apparatus, when the frame is executed up to the entropy coding process for each frame, the amount of movement of the frame is evaluated in the same manner as in the invention of claims 1 to 16 , and the amount of movement is determined. It is also possible to thin out frames with a small amount of motion during the encoding process by aborting the process for frames evaluated as small.

Here, the description of JPEG2000 will be supplemented.
The conversion formula for the DC level shift of JPEG2000 is as follows.
Forward conversion
I (x, y) ← I (x, y) −2 ^ Ssiz (i)
Reverse transformation
I (x, y) ← I (x, y) + 2 ^ Ssiz (i)
Here, Ssiz (i) is the bit depth of each component i of the original image (i0, 1, 2 for RGB images). When this DC level shift is a positive number such as an RGB signal value, the forward conversion subtracts half of the dynamic range of the signal from each signal value, and the inverse conversion uses the signal dynamics for each signal value. A level shift that adds half the range is performed. However, this level shift is not applied to signed integers such as Cb and Cr signals of YCbCr signals.

  JPEG2000 defines reversible conversion (RCT) and irreversible conversion (ICT) as color conversion (component conversion).

RCT forward and inverse transforms are expressed by the following equations.
Forward conversion
Y0 (x, y) = floor ((I0 (x, y) + 2 * (I1 (x, y) + I2 (x, y)) / 4)
Y1 (x, y) = I2 (x, y) -I1 (x, y)
Y2 (x, y) = I0 (x, y) -I1 (x, y)
Reverse transformation
I1 (x, y) = Y0 (x, y) -floor ((Y2 (x, y) + Y1 (x, y)) / 4)
I0 (x, y) = Y2 (x, y) + I1 (x, y)
I2 (x, y) = Y1 (x, y) + I1 (x, y)
In the equation, I represents an original signal, and Y represents a signal after conversion. In the case of the RGB signal, 0 = R, 1 = G, 2 = B in the I signal, and 0 = Y, 1 = Cb, 2 = Cr in the Y signal.

The forward conversion and the reverse conversion of ICT are expressed by the following equations.
Forward conversion
Y0 (x, y) = 0.299 * I0 (x, y) + 0.587 * I1 (x, y) + 0.144 * I2 (x, y)
Y1 (x, y) =-0.16875 * I0 (x, y) -0.33126 * I1 (x, y) + 0.5 * I2 (x, y)
Y2 (x, y) = 0.5 * I0 (x, y) -0.41869 * I1 (x, y) -0.08131 * I2 (x, y)
Reverse transformation
I0 (x, y) = Y0 (x, y) + 1.402 * Y2 (x, y)
I1 (x, y) = Y0 (x, y) -0.34413 * Y1 (x, y) -0.71414 * Y2 (x, y)
I2 (x, y) = Y0 (x, y) + 1.772 * Y1 (x, y)
In the equation, I represents an original signal, and Y represents a signal after conversion. In the case of the RGB signal, 0 = R, 1 = G, 2 = B in the I signal, and 0 = Y, 1 = Cb, 2 = Cr in the Y signal.

As described above, when the 9 × 7 wavelet transform is selected in JPEG2000, the wavelet coefficients can be linearly (scalar) quantized for each subband. The quantization formula is as follows.
qb (u, v) = sign (ab (u, v)) * floor (| ab (u, v) | / Δb)
Where ab (u, v) is the coefficient in subband b
qb (u, v) is a coefficient in subband b Δb is the number of quantization steps in subband b The number of quantization steps Δb is expressed by the following equation.
Δb = 2 ^ (Rb-εb * floor (1 + μb / 2 ^ 11))
Where Rb is the dynamic range in subband b εb is the quantization exponent in subband b μb is the quantization mantissa in subband b Exponent εb and mantissa μb are the QCD markers in the main header or tile part header in the codestream Or it is defined by the QCC marker.

As is clear from the above description, (1) according to the inventions of claims 1 to 16 , the frame motion amount is evaluated and a frame with a small motion amount is selected without decoding the moving image code. Furthermore, it is possible to delete a frame with a small amount of motion and reduce the amount of code of the moving image without making the motion of the moving image unnatural when decoded. (2) According to the inventions of claims 4 and 5 , it is possible to more accurately evaluate the motion amount of the frame including the small moving object. (3) According to the invention of claim 3, the amount of motion of a frame including a subject having a long lateral edge can be more accurately evaluated. (4) According to the inventions of claims 6 to 11 , even if there is a difference in the number of quantization steps and the truncation amount between subbands or blocks in the frame, and also between frames, the frame motion amount is accurately evaluated. Thus, a frame with a small amount of motion can be selected. (5) According to the invention of claim 13 , it is possible to effectively reduce the code amount of a moving image having many frames with a small amount of motion. (6) According to the invention of claim 14 , it is possible to prevent excessive frame thinning in a moving image having many frames with small motion. (7) When the top frame is subsampled moving the head frame of the icon display and decimation icon display before the first frame of the image are different after the thinning, but easily mistaken as different moving image, claim 15 According to this invention, even when the amount of movement of the first frame is small, it is possible to prevent the thinning-out, and thus it is possible to avoid such an inconvenience at the time of icon display. (8) According to the inventions of claims 17 and 18, the inventions of claims 1 to 16 can be easily implemented using a computer, and the like can be obtained.

  FIG. 12 is a block diagram for explaining an embodiment of the present invention. In FIG. 12, reference numeral 100 denotes a moving image file to be processed, and reference numeral 101 denotes code amount calculation means for calculating a code amount reflecting an edge amount in a specific direction from the code of each frame of the moving image. A frame selection unit 102 evaluates the motion amount of each frame based on the code amount calculated by the code amount calculation unit 101, and selects a frame having a small motion amount. Reference numeral 103 denotes a processing unit such as frame deletion, which generates a new moving image file 104 by performing processing such as deleting the code of the frame selected by the frame selecting unit 102 from the moving image file 100.

  According to a preferred embodiment of the present invention, the moving image files 100 and 104 are Motion-JPEG2000 moving image files. Motion-JPEG2000 moving images are composed of a series of independently encoded frames (I) as schematically shown in FIG. The frame rate is 30 frames / second. Each frame is encoded by JPEG2000.

  In Motion-JPEG2000, as schematically shown in FIG. 14, field-based coding for performing processing after wavelet transform by separating a frame into odd and even fields and a field as schematically shown in FIG. 15. Any of the frame-based encoding that executes the processing after the wavelet transform without separating the image is possible. However, what is targeted by the present invention is a moving image by frame-based encoding.

  As schematically shown in FIG. 16, the Motion-JPEG2000 moving image file includes a “movie data” (abbreviation mdat) in which a code stream of a frame is stored, and a “movie resource” (abbreviation movov) in which an attribute of mdat is stored. ). The components of the Motion-JPEG2000 file are called boxes, and many boxes form a hierarchical structure. Such a hierarchical structure is shown in FIG. The boxes located on the left side of the table of FIG. 17 are the upper layers, and therefore “moov” and “mdat” are the uppermost layers.

In the moving image file 104 processed by the processing unit 103 such as frame deletion, the code stream of a part of the frame is deleted from “mdat”, and the content of “moov” is updated accordingly. The content of “moov” that is updated when a frame is deleted is a lower box of “sample-table box (stbl)” related to the number of frames and the frame interval.
“Time-to-Sample Box (stts)” and “Sample-to-Chunk Box (stsc)”. However, if the number of chunks is 2 or more, it is also necessary to update “Chunk-Offset Box (stco)” in “stbl”. A chunk is a group of scenes. Here, the description will be continued assuming that the number of chunks is 1.

According to the Motion-JPEG2000 standard, "stts"
aligned (8) class TimeToSampleBox
extends FullBox ('stts', version = 0,0) {
unsigned int (32) entry-count;
int i;
for (i = 1; i ≦ entry-count; i ++) {
unsigned int (32) sample-count;
int (32) sample-delta;
}
}
It is defined as

  That is, “stts” is a table as shown in FIG. entry-count is the number of rows in the table. sample-count and sample-dalta are the components of the table, sample-count is the number of frames in each row (however, the frame interval for the same row is the same), sample-delta is the time interval of each row (in units of 1/30 seconds) Yes).

Also, “stsc”
aligned (8) class SampleToChunkBox
extends FullBox ('stsc', version = 0,0) {
unsigned int (32) entry-count;
for (i = 1; i ≦ entry-count; i ++) {
unsigned int (32) first-chunk;
unsigned int (32) samples-per-chunk;
unsigned int (32) sample-description-index;
}
}
It is defined as

That is, “stsc” is a table as shown in FIG. entry-count is the number of rows in the table.
First-chunk, samples-per-chunk, and sample-description-index are components of this table. The first-chunk is a chunk number that uses the first-chunk table (here, 1 is a chunk). samples-per-chunk is the number of frames included in the chunk corresponding to each row of the table, and sample-description-index is the ID of the frame included in the chunk corresponding to each row (here, 1 is 1).

  According to the exemplary embodiment of the present invention, the functions of the respective units 101, 102, 103 of the moving image processing apparatus according to the present invention shown in FIG. 12, in other words, the processing of the moving image processing method according to the present invention. The process corresponding to each means in the procedure is realized by one or more programs using a computer. An example of such an embodiment will be briefly described with reference to FIG. In FIG. 18, reference numeral 200 denotes a CPU, 201 denotes a RAM, and 202 denotes a hard disk device, which are connected to each other via a bus 203.

  The outline of the processing flow is as follows. First, the content of the moving image file recorded in the hard disk device 202 is read into the RAM 201 by an instruction from the CPU 200 (1). The CPU 200 calculates a code amount used for motion amount evaluation for each frame in “mdat” of the moving image file on the RAM 201, evaluates the motion amount of the frame based on the code amount, and selects a frame having a small motion amount. (2). Next, the CPU 201 deletes the code of the selected frame from “mdat” of the moving image file on the RAM 201 and updates the content of “moov” accordingly (3). Then, according to the instruction of the CPU 200, the moving image file on the RAM 201 is transferred to the hard disk device 201 and stored as a moving image file different from the original moving image file (4).

  Examples of the present invention will be described below.

The moving image file processed in the embodiment described here is a Motion-JPEG2000 moving image file, and 9 × 7 wavelet transform is used for encoding each frame, and the luminance / color difference value of RGB pixel value is YCbCr. And converted to. Also, the number of chunks of moving images is 1. Further, a value obtained by dividing the ratio of the code amount of the 1LH and 1HL subbands by the ratio of the code amount of the 2LH and 2HL subbands, that is, the code amount ratio in the invention of claim 3 is used for evaluating the motion amount of the frame. To do. In addition, with respect to the code amount of each 1LH, 1HL, 2LH, and 2HL subbands from which the code amount ratio is calculated, the difference in the number of quantization steps between the subbands in the frame and between the first frame and the quantization Correction is performed to absorb the influence of the difference in the number of steps. Also, the first frame is not selected as a deletion frame.

  FIG. 21 is a flowchart showing the overall flow of processing. The entire process will be briefly described with reference to this flowchart and FIG.

  First, in step 1, the code amount calculation means 101 calculates the code amounts of the 1LH, 1HL, 2LH, and 2HL subbands of the luminance component of each frame (excluding the first frame in this embodiment), and at this time, the code amount Correction is also performed. In step 2, the frame selection means 102 calculates the code amount ratio for evaluating the motion amount of the frame using the obtained code amount. However, as will be described later, it is efficient to execute the processing of step 1 and step 2 as a group of processing.

In step 3, the frame selection unit 102 selects a frame with a small amount of motion (deletion frame) evaluated based on the calculated code amount ratio. As the selection method, the method of the invention of claim 13 or claim 14 can be selected. Further, in this embodiment, as defined in the invention of claim 15 , the first frame is not selected as the deletion frame.

  In step 4, the frame deletion processing unit 103 performs processing for deleting the code stream of all the components of the selected frame. In step 5, the frame deletion processing unit 103 performs a necessary update process of “moov” accompanying the frame deletion. As a result, a new moving image file in which a frame with a small amount of motion is deleted is generated, and in step 6, the processing for saving the moving image file in the internal or external storage device of the image processing apparatus is performed by the frame deletion processing means 103. , Complete a series of processing.

FIG. 22 shows a processing flow in the case where step 1 and step 2 are executed as a group of processes. Step 18 in the figure is a processing step corresponding to step 2.

  First, in step 11, the address and the number of frames from “moov” to “mdad” are read. Here, description will be made assuming that one chunk and 10 frames.

  At step 12, a table having the number of entries equal to the number of frames is created. As shown in FIG. 23, this table is a table for storing a frame number, a code amount ratio, and a deletion / save flag for each frame.

  In step 13, the header information of the code stream of the first frame in “mdat” is referred to, and the quantization step number and truncation amount of the luminance component 1LH, 1HL, 2LH, and 2HL subbands are acquired and held. In this embodiment, since the code amount is corrected in units of bit play, the number of bit planes is used as the truncation amount. However, it is also possible to perform code amount correction in units of sub-bit planes as described above, and in this case, the number of truncated sub-bit planes is acquired as the truncation amount.

  In step 14, the code stream of the next frame is referred to, and the number of quantization steps of the 1LH, 1HL, 2LH, and 2HL subbands is acquired and held from the main header, and in the next step 15, each step is performed based on the packet header of the frame. The sub-band code amount and truncation amount are obtained and held. Inter-frame conversion processing for correcting the difference in the number of quantization steps from the first frame and the difference from the truncation amount is performed on the code amount of each subband in step 16, and then in step 17, the subband of the frame is processed. Intra-frame conversion processing is performed to correct the difference in the number of quantization steps and the difference in the amount of truncation. These conversion processes will be described later.

  In step 18, the code amount ratio (abbreviated as 1LH / 1HL / 2LH / 2HL) is calculated using the code amount after the conversion process, and the result is written in the corresponding entry of the table (FIG. 23).

  The above steps 14 to 18 are repeatedly executed for each frame up to the last frame.

  Next, the frame selection process based on the code amount ratio (step 3 in FIG. 21) will be described. In this embodiment, a method (1) for selecting frames (excluding the first frame) whose code amount ratio is smaller than a predetermined threshold and a method (2) for selecting a predetermined number of frames in ascending order of the code amount ratio are selected. Can do.

  First, the former selection method (1) will be described with reference to the flowchart of FIG. In step 31, a save flag is written in the first entry of the table (FIG. 23). In step 32, the code amount ratio is read from the next entry in the table, and in step 33, the code amount ratio is compared with a predetermined threshold value TH. When the code amount is smaller than the threshold value TH, a deletion flag is written in the entry (that is, the frame corresponding to the entry is selected as the deletion frame). If the code amount ratio is equal to or greater than the threshold value, a save flag is written in the entry. The same process is repeated until the last entry in the table.

  Next, the latter selection method (2) will be described with reference to the flowchart of FIG. In step 41, a save flag is written in the first entry of the table (FIG. 23). In step 42, the code amount ratio from the second entry to the last entry of the table is read and sorted in ascending order of value. That is, the table entries are ordered in ascending order of the code amount ratio. Then, in step 43, a predetermined number of entries are selected from the top entries in this order, a deletion flag is written in each entry, and a save flag is written in each remaining entry.

  In step 4 of FIG. 21, the code stream of the frame corresponding to the entry in which the deletion flag is recorded is deleted with reference to the flag recorded in each entry of the table (FIG. 23).

  FIG. 26 is a flowchart of the “moov” update process (step 5 in FIG. 21). In step 51, “time-to-sample box (stts)” is updated, and in step 52, “sample-to-chunk box (stsc)” is updated. In this embodiment, since the number of chunks is 1, the update is completed as described above. However, when there are a plurality of chunks, an update process (step 53) of “chunk-offset box (stco)” is also required.

The method described with reference to FIG. 24 is used as a method for selecting a deleted frame, and FIGS. 27 to 29 show examples of processing results when the threshold value TH for determining the code amount ratio is 1. FIG. 27 shows the contents of the table (FIG. 23) after processing. FIG. 28 shows before and after updating “stts”, and FIG. 29 shows before and after updating “stsc”. Since frame numbers 2 to 4 and 6 are deleted, sample-delta (frame interval), which is uniform value 1, takes three kinds of values, and entry-count is also 3. Along with this, sample
count is also updated. Since 4 frames are deleted, samples-per-chunk is reduced to 10-4 = 6.

  The method described with reference to FIG. 25 is used as a method for selecting a deletion frame, and FIGS. 30 to 32 show examples of processing results when the number of deletion frames is five. FIG. 30 shows the contents of the table (FIG. 23) after processing. FIG. 31 shows before and after updating “stts”, and FIG. 32 shows before and after updating “stsc”. Since 5 frames are deleted, samples-per-chunk is reduced to 10-5 = 5.

  Next, the code amount inter-frame conversion process (step 16 in FIG. 22) will be described. This conversion process includes a conversion process related to the number of quantization steps and a conversion process related to the amount of truncation.

  First, the inter-frame conversion process regarding the number of quantization steps will be described. FIG. 33 is a flowchart of the conversion process for the code amounts of the 1LH and 1HL subbands. Similar conversion processing is performed for the 2LH and 2HL subbands.

  Referring to FIG. 33, in step 101, the larger quantization step number of the held first frame 1HL and 1LH subbands is selected and set as X. The code amount is corrected based on this X. That is, in step 102, the number of quantization steps of the 1HL subband of the processing target frame held is set as B. In step 103, X and B are compared and determined. If X−B> 0, the first frame is more quantized. Therefore, in step 104, the code amount of the 1HL subband of the processing target frame On the other hand, correction is performed by subtracting log (X-B) bit planes from the lower order (however, integers are rounded off). If X−B <0, it means that the processing target frame is more quantized. Therefore, in step 105, the code amount of the least significant bit plane is set with respect to the code amount of the 1HL subband of the processing target frame. Correction to (log (B−X) +1) times is performed. The bottom of the log is 2 (the same applies hereinafter). When X−B = 0, the code amount is not corrected. The corrected code amount is again held as the code amount. Next, in step 106, the number of quantization steps of the 1LH subband of the processing target frame held is set as B. In step 107, X and B are compared and determined. If X−B> 0, the first frame is more quantized, and in step 108, the code amount of the 1LH subband of the processing target frame is determined. On the other hand, correction is performed by subtracting log (X-B) bit planes from the lower order (however, integers are rounded off). If X−B <0, it means that the processing target frame is more quantized. Therefore, in step 109, the code amount of the least significant bit plane is set with respect to the code amount of the 1LH subband of the processing target frame. Correction to (log (B−X) +1) times is performed. When X−B = 0, the code amount is not corrected. The corrected code amount is again held as the code amount.

  Next, an inter-frame conversion process for correcting the difference in truncation amount will be described. FIG. 34 is a flowchart of this conversion processing for the 1LH and 1HL subbands. Similar conversion processing is performed for the 2LH and 2HL subbands.

  Referring to FIG. 34, in step 111, the larger truncation amount of the 1LH and 1HL subband truncation amounts (number of bit planes) of the held first frame is selected and set as X. The code amount is corrected based on this X. That is, in step 112, the truncation amount of the 1HL subband of the processing target frame held is set as B. In step 113, X and B are compared and determined. If X−B> 0, the first frame is more truncated, and in step 114, the code amount of the 1HL subband of the processing target frame is set. On the other hand, correction is performed by subtracting (X-B) bit planes from the lower order. If X−B <0, the processing target frame is more truncated, and in step 115, the code amount of the least significant bit plane is set to (1) the code amount of the 1HL subband of the processing target frame ( B-X + 1) correction is performed. When X−B = 0, the code amount is not corrected. The corrected code amount is again held as the code amount. Next, in step 116, the truncation amount of the 1LH subband of the processing target frame held is set as B. In step 117, X and B are compared and determined. If X−B> 0, the first frame is more truncated, and in step 118, the code amount of the 1LH subband of the processing target frame is set. On the other hand, correction is performed by subtracting (X-B) bit planes from the lower order. If X−B <0, the processing target frame is more truncated, and in step 115, the code amount of the least significant bit plane is set to (1LH subband code amount of the processing target frame) ( B-X + 1) correction is performed. When X−B = 0, the code amount is not corrected. The corrected code amount is again held as the code amount.

  Next, the code amount intra-frame conversion process (step 17 in FIG. 22) will be described. This conversion process includes a conversion process related to the number of quantization steps and a conversion process related to the amount of truncation.

  First, the intra-frame conversion process regarding the number of quantization steps will be described. FIG. 35 is a flowchart of the conversion process for the code amounts of the 1LH and 1HL subbands. Similar conversion processing is performed for the 2LH and 2HL subbands.

  Referring to FIG. 35, in step 121, the number of quantization steps of the 1HL subband of the held processing target frame is set to A, and the number of quantization steps of the 1LH subband is set to B. In step 122, A and B are compared and determined. If AB> 0, 1HL is more quantized, and therefore, in step 123, the code amount from the lower order is set for the code amount of 1LH subband. Correction is performed by subtracting (A-B) (but rounded off) bit planes. If A-B <0, 1LH is more quantized, so in step 124, log (BA) sheets from the lower order for the code amount of 1HL subband (however, rounded off to an integer) )) Is subtracted from the bit plane. When A−B = 0, the code amount is not corrected. The corrected code amount is again held as the code amount.

  In the processing described above, the code amount correction method is adapted to the number of bit planes of the subband having the larger quantization step number. However, the code amount correction is adapted to the number of bit planes of the subband having the smaller quantization step number. It is also possible to use a method, and a processing flow in the case of the method is shown in FIG.

Referring to FIG. 36, in step 131, the quantization step number of the 1HL subband of the held processing target frame is set to A, and the quantization step number of the 1LH subband is set to B. In step 132, A and B are compared and determined, and if A−B> 0, 1HL is more quantized. Therefore, in step 133, the least significant code is assigned to the code amount of 1HL subband. Correction for multiplying the bit plane code amount by (log (A−B) +1) is performed. If A-B <0, 1LH is more quantized, so in step 134, the code amount of the least significant bit plane is set to (log (B− A) +1) multiplication is performed. When A−B = 0, the code amount is not corrected. The same applies to the correction for the code amounts of the 2HL and 2LH subbands.

  Next, the intra-frame conversion process regarding the truncation amount will be described. FIG. 37 is a flowchart of the conversion process for the code amounts of the 1LH and 1HL subbands. Similar conversion processing is performed for the 2LH and 2HL subbands.

  Referring to FIG. 37, in step 141, the truncation amount of the 1HL subband of the held processing target frame is set to A, and the truncation amount of the 1LH subband is set to B. In step 142, A and B are compared and determined. If AB> 0, 1HL is more truncated, and in step 143, the code amount of 1LH subband is compared to the lower order (A -B) Correction for subtracting bit planes. In the case of A−B <0, 1LH is more quantized. Therefore, in step 144, (B−A) bit planes from the lower order are subtracted from the code amount of 1HL subband. Make corrections. When A−B = 0, the code amount is not corrected. The corrected code amount is again held as the code amount.

  In the above-described processing, the code amount correction method is adapted to the number of bit planes of the subband having the larger truncation amount. However, the code amount correction method is adapted to the number of bit planes of the subband having the smaller truncation amount. FIG. 38 shows a processing flow in the case of this method.

  Referring to FIG. 38, in step 151, the truncation amount of the 1HL subband of the held processing target frame is set to A, and the truncation amount of the 1LH subband is set to B. In step 152, A and B are compared and determined. If A−B> 0, 1HL is more truncated, and in step 153, the least significant bit plane for the code amount of 1HL subband is obtained. Is corrected by multiplying the code amount of (A−B + 1). If A−B <0, it means that the 1LH subband is more quantized. Therefore, in step 154, the code amount of the least significant bit plane is set to (B−A + 1) with respect to the code amount of 1LH subband. ) Perform double correction. When A−B = 0, the code amount is not corrected. Similar correction is performed for the code amounts of the 2HL and 2LH subbands.

  When it is not necessary to calculate the code amount ratio strictly, one or both of the intra-frame conversion process and the inter-frame conversion process can be omitted. When selecting a certain number of frames to be deleted in accordance with the order of the code amount ratio of frames, it is desirable to perform inter-frame conversion processing because the relative magnitude relationship of the code amount ratio between frames is important.

  In the embodiment described above, the amount of motion of the frame is evaluated using the ratio of the code amount of the entire subband. However, the frame is calculated using the code amount ratio obtained by summing the ratio of the code amount of the block unit smaller than the subband. A description will be supplemented regarding the mode of evaluating the amount of movement of the.

  Here, it is assumed that Motion-JPEG2000 moving images are used and precincts are used as blocks. When the precincts are numbered as shown in FIG. 39 for the HL and LH subbands at decomposition levels 1 and 2, a precinct unit (block unit) using the precinct code amount corresponding to the positional relationship as shown in FIG. ) Is calculated.

  The code amount ratio 0 is obtained by dividing the code amount ratio of the precinct 0 of the 1LH and 1HL subbands by the ratio of the code amount of the precinct 0 of the 2LH and 2HL subbands. The code amount ratio 1 is obtained by dividing the code amount ratio of the precinct 1 of the 1LH and 1HL subbands by the ratio of the code amount of the precinct 0 of the 2LH and 2HL subbands. The code ratio of other subband units can be obtained in the same manner.

  By calculating the sum of the code amount ratios in units of precincts obtained in this way, it is possible to obtain a code amount ratio that more accurately reflects the movement of the subject for each block (precinct in this case). Processing such as selection of a deletion frame using the code amount ratio may be the same as in the above-described embodiment. Also, the interframe conversion process and the intraframe conversion process for the code amount may be performed in the same manner in units of precincts. Note that the same code amount ratio can be obtained using the code block as a unit of the block, and a smaller subject motion can be reflected in the code amount ratio.

  The embodiment of the present invention has been described above by taking the Motion-JPEG 2000 moving image as an example. However, the present invention can also be applied to a moving image encoded by an encoding method using wavelet transform or other frequency conversion. It is clear.

It is a figure for explanation of "comb shape" by a subject's movement. It is a block diagram for demonstrating JPEG2000. It is a figure which shows the example and coordinate system of an original image. It is a figure which shows the arrangement | sequence of the coefficient after filtering to a perpendicular direction. It is a figure which shows the arrangement | sequence of the coefficient after filtering to a horizontal direction. It is a figure which shows the arrangement | sequence of the coefficient which carried out the deinterleaving. It is a figure which shows the arrangement | sequence of the coefficient which carried out the deinterleaving after two conversions. It is a figure which shows the subband and resolution level after 3 times of conversion. It is a figure which shows the relationship between an image, a tile, a subband, a precinct, and a code block. It is a figure which shows the example of a layer structure. It is a figure which shows the example of a palette. It is a block diagram for demonstrating the moving image processing apparatus which concerns on this invention. It is a figure which shows the picture structure of the moving image of Motion-JPEG2000. It is explanatory drawing of field-based encoding. It is explanatory drawing of frame-based encoding. It is a figure which shows the basic structure of the moving image file of Motion-JPEG2000. It is a figure which shows the hierarchical structure of the moving image file of Motion-JPEG2000. It is a block diagram for demonstrating the form which implements this invention by a program. It is explanatory drawing of Time-to-Sample Box. It is explanatory drawing of Sample-to-Chunk Box. It is a flowchart which shows the flow of the whole process in one Example. It is a flowchart which shows the process from calculation of code amount to calculation of code amount ratio. It is explanatory drawing of the table produced for a process. It is a flowchart which shows an example of the frame selection process based on code amount ratio. It is a flowchart which shows another example of the frame selection process based on code amount ratio. It is a flowchart of the update process of moov accompanying frame deletion. It is a figure which shows the content of the table after a process. It is a figure which shows the content before and after the update of Time-to-Sample Box. It is a figure which shows the content before and after the update of Sample-to-Chunk Box. It is a figure which shows the content of the table after a process. It is a figure which shows the content before and after the update of Time-to-Sample Box. It is a figure which shows the content before and after the update of Sample-to-Chunk Box. It is a flowchart of the conversion process between frames regarding the number of quantization steps. It is a flowchart of the conversion process between frames regarding the amount of truncation. It is a flowchart of the intra-frame conversion process regarding the number of quantization steps. It is a flowchart which shows another example of the intra-frame conversion process regarding the number of quantization steps. It is a flowchart of the intra-frame conversion process regarding the truncation amount. It is a flowchart which shows another example of the intra-frame conversion process regarding a truncation amount. It is a figure which shows a precinct and its number. It is a figure which shows the precinct used for calculation of the code amount of a block unit.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Moving image file before processing 101 Code amount calculation means 102 Frame selection means 103 Processing means such as frame deletion 104 Moving image file after processing

Claims (18)

A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  A means for calculating the code amount of the 1LH subband based on information of a portion of the header of the code of each frame in which information relating to the code amount of the 1LH subband is described, and the code amount calculated by the means And a means for selecting a frame with a small amount of motion by evaluating that the amount of motion is larger as the size of the moving image processing device is larger. A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  Based on the information in the header of the code of each frame and the information about the code amount of the 1LH subband and the code amount of the 1HL subband, the code amount of the 1LH subband and the code amount of the 1HL subband Respectively, and a code amount calculated by the means,
  Code amount ratio = (Code amount of 1LH subband) / (Code amount of 1HL subband)
And a means for selecting a frame with a small amount of motion by evaluating that the amount of motion is larger as the code amount ratio calculated by (2) is larger. A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  Based on the information in the header in the code of each frame, which describes the information about the code amount of 1LH subband, the code amount of 1HL subband, the code amount of 2LH subband, and the code amount of 2HL subband. 1LH subband code amount, 1HL subband code amount, 2LH subband code amount and 2HL subband code amount respectively, and using the code amount calculated by the means,
  Code quantity ratio = [(1LH subband code quantity / 1HL subband code quantity) / (2LH subband code quantity
                Code amount / 2HL subband code amount)]
And a means for selecting a frame with a small amount of motion by evaluating that the amount of motion is larger as the code amount ratio calculated by (2) is larger. A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  1LH subband code in units of blocks smaller than the subband based on the header information in the code of each frame and the information of the portion describing the information about the code amount of the 1LH subband and the code amount of the 1HL subband. A unit for calculating the amount and the code amount of the 1HL subband, and the code amount of the block unit corresponding to the positional relationship calculated by the unit,
  Block code amount ratio = (Code amount of 1LH subband block unit / 1HL subband)
                      Code amount in blocks)
A code amount ratio obtained by summing up the block code amount ratio calculated for all blocks, evaluating that the larger the code amount ratio is, the larger the amount of motion is, and selecting a frame with a small amount of motion. A moving image processing apparatus. A moving image processing apparatus that processes a moving image obtained by frame-based encoding an interlaced image independently for each frame,
  The frame-based encoding uses a two-dimensional wavelet transform as a frequency transform, and encodes a wavelet coefficient for each subband.
  Based on the information in the header in the code of each frame, which describes the information about the code amount of 1LH subband, the code amount of 1HL subband, the code amount of 2LH subband, and the code amount of 2HL subband. Means for calculating the code amount of 1LH subband, the code amount of 1HL subband, the code amount of 2LH subband, and the code amount of 2HL subband in units of blocks smaller than the subband, and the means , Using the code amount of the block unit corresponding to the positional relationship,
  Block code amount ratio = [(1LH subband block code amount / 1HL subband
      Code amount in block units) / (code amount in block units of 2LH subbands / 2HL)
      Subband block code amount)]
A code amount ratio obtained by summing up the block code amount ratio calculated for all blocks, evaluating that the larger the code amount ratio is, the larger the amount of motion is, and selecting a frame with a small amount of motion. A moving image processing apparatus. The moving image processing apparatus according to claim 2 or 3 ,
A moving image processing apparatus characterized in that a code amount is corrected in accordance with a difference in truncation amount between subbands. The moving image processing apparatus according to claim 2 or 3 ,
A moving image processing apparatus for correcting a code amount according to a difference in truncation amount between subbands and a difference in truncation amount between frames. The moving image processing apparatus according to claim 4 or 5 ,
A moving image processing apparatus for correcting a code amount according to a difference in a truncation amount between blocks. The moving image processing apparatus according to claim 4 or 5 ,
A moving image processing apparatus that corrects a code amount in accordance with a truncation amount difference between blocks and a truncation amount difference between frames. The moving image processing apparatus according to any one of claims 2 to 9 ,
A moving image processing apparatus that corrects a code amount according to a difference in the number of linear quantization steps between subbands. The moving image processing apparatus according to any one of claims 2 to 9 ,
A moving picture processing apparatus that corrects a code amount according to a difference in the number of linear quantization steps between subbands and a difference in the number of linear quantization steps between frames. The moving image processing apparatus according to any one of claims 1 to 11 ,
A moving image processing apparatus, wherein the calculated code amount is a code amount of a luminance component. The moving image processing apparatus according to any one of claims 1 to 12 ,
The moving image processing apparatus according to claim 1, wherein the means for selecting the frame selects a frame having an estimated motion amount smaller than a predetermined value. The moving image processing apparatus according to any one of claims 1 to 12 ,
The moving picture processing apparatus according to claim 1, wherein the means for selecting the frames selects a predetermined number of frames in ascending order of the evaluated motion amount. The moving image processing apparatus according to claim 13 or 14 ,
The moving image processing apparatus characterized in that the means for selecting the frame excludes the first frame of the moving image from the selection target. 16. The moving image processing apparatus according to claim 13, 14 or 15 , further comprising means for deleting the code of the frame selected by the means for selecting the frame from the moving image. A program that causes a computer to function as each unit of the moving image processing apparatus according to any one of claims 1 to 16.
A computer-readable information recording medium on which the program according to claim 17 is recorded.

Images