JP2005102125A - Image decoding method and image decoding apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently decode all or some frames of encoded moving image data based on the processing performance of a moving image decoding apparatus, and obtain high playback image quality with less visual disturbance. <P>SOLUTION: The moving image decoding method for decoding of encoded moving image data includes: a step S1901 of extracting encoded image data of a sample frame from each frame, decoding the data in a prescribed decode unit and measuring time related to the decoding; a step S1902 of determining the decoded number of decoding units so that the measured time is a prescribed time or below; a step S1904 of decoding encoded image data of each frame according to the given number of the decoding units; a step S1905 of measuring the time required for decoding of each frame in decoding each frame; a step 1907 of accumulating a difference between the prescribed time and the time measured in the step S1905 every time each frame is decoded and updating the number of decoding units determined in the step S1092 when the accumulated value reaches the prescribed value or over; and the S1904 decoding the encoded image data of each frame according to the number of decoding units determined in the S1902 or the number of decoding units updated in the step S1907. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for inputting moving image data in which an image of each frame is encoded and decoding the encoded image data of each frame.

  In general, encoding methods of moving image data can be broadly classified into those that use the correlation between frames and those that do not use the correlation between frames. Each method has advantages and disadvantages, and which method is suitable depends on the application to be used. For example, Motion JPEG is an example of an encoding method that uses each frame of moving image data as a single still image and encodes it independently, and does not use correlation between frames. Encoding independently for each frame makes it easy to edit moving images such as video segmentation, concatenation, and partial rewriting, and selects and decodes the number of decoding frames according to the processing capability of the decoding side. There is an advantage that it is possible.

  2. Description of the Related Art In recent years, attention has been paid to a method of encoding each frame by combining wavelet transform and bit-plane encoding in an encoding method for encoding moving image data independently for each frame. In such a moving image coding method, decoding by changing the spatial resolution in stages using the subband decomposition mechanism in the wavelet transform is possible, and by changing the number of decoding bit planes There is a great feature such that the pixel accuracy can be changed in stages.

  JPEG2000 (ISO / IEC 15444), which is an image coding method that is being standardized by ISO / IEC JTC1 / SC29 / WG1, is also configured by a combination of wavelet transform and bit-plane coding. Part 3 of the standard defines the file format when applied to encoding each frame of a moving image under the name Motion JPEG2000.

  Such a moving image coding system represented by Motion JPEG2000 has advantages such as decoding resolution and flexibility of decoding pixel accuracy as described above, but has a high load of encoding / decoding processing by bit plane encoding. There is a drawback. In particular, when video recorded by a dedicated moving image recording apparatus is played back by a personal computer, there arises a problem that depending on the performance of the computer, all data cannot be decoded and displayed in real time.

Conventionally, there has been a method for determining a desired decoding processing time for decoding a frame, allocating a decoding processing time to an encoding processing unit, and decoding in units of bit planes using the allocated decoding processing time. It is disclosed (for example, see Patent Document 1).
Japanese Patent Laid-Open No. 2001-112004

  However, according to the above method, in the video decoding device that terminates the decoding process at a predetermined time, the point at which the decoding process is aborted (the number of decoded bit planes) is likely to change for each frame. There is a problem that a time change of the shape occurs, and this becomes a visual disturbance factor as flicker.

  The present invention has been made in view of the above problems, and decodes all or part of moving image encoded data efficiently according to the processing capability to obtain a good reproduction image quality with less visual interference. The purpose is to provide technology.

  In order to achieve the object of the present invention, for example, an image decoding method of the present invention comprises the following arrangement.

That is, an image decoding method for inputting moving image data in which an image of each frame is encoded and decoding the encoded image data of each frame,
A sample frame decoding step of extracting encoded image data of a sample frame from each frame and decoding it in a predetermined decoding unit;
A first measuring step of measuring a time required for decoding the encoded image data of the sample frame;
A determining step for determining the number of decoding units to be decoded so that the time measured in the first measuring step is equal to or less than a predetermined time;
A decoding step of decoding the encoded image data of each frame according to a given number of decoding units;
A second measuring step of measuring a time required for decoding each frame when decoding each frame in the decoding step;
The difference between the predetermined time and the time measured in the second measuring step is accumulated every time each frame is decoded, and is determined in the determining step when the accumulated value exceeds a predetermined value. An update step of updating the number of decoded units,
In the decoding step, the encoded image data of each frame is decoded according to the number of decoding units determined in the determining step or the number of decoding units updated in the updating step.

  In order to achieve the object of the present invention, for example, an image decoding apparatus of the present invention comprises the following arrangement.

That is, an image decoding apparatus that inputs moving image data in which an image of each frame is encoded and decodes the encoded image data of each frame,
Sample frame decoding means for extracting encoded image data of a sample frame from each frame and decoding it in a predetermined decoding unit;
First measuring means for measuring a time for decoding the encoded image data of the sample frame;
Determining means for determining the number of decoding units to be decoded so that the time measured by the first measuring means is a predetermined time or less;
Decoding means for decoding the encoded image data of each frame according to the given number of decoding units;
Second measuring means for measuring a time required for decoding each frame when the decoding means decodes each frame;
The difference between the predetermined time and the time measured by the second measuring means is accumulated every time each frame is decoded, and the determining means determines when the accumulated value exceeds a predetermined value. Update means for updating the number of decoded decoding units,
The decoding means decodes the encoded image data of each frame according to the number of decoding units determined by the determining means or the number of decoding units updated by the updating means.

  According to the configuration of the present invention, it is possible to efficiently decode all or part of moving image encoded data according to the processing capability, and obtain a good reproduction image quality with little visual interference.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

  Before describing a preferred embodiment of the image decoding device of the present invention, first, moving image encoded data to be decoded in the image decoding device of the present invention will be described.

  FIG. 1 is a diagram illustrating a functional configuration of a moving image encoding apparatus that generates moving image encoded data to be decoded by an image decoding apparatus used in the following embodiment.

  This moving image encoding apparatus encodes each frame constituting a moving image independently by using an encoding method in which wavelet transform and bit plane encoding are combined. The flow of the encoding process in the moving image encoding apparatus will be briefly described with reference to FIG. As shown in FIG. 1, the moving image coding apparatus includes a moving image data input unit 201, a discrete wavelet transform unit 202, a coefficient quantization unit 203, a bit plane coding unit 204, a code string forming unit 205, and a secondary storage. A device 206 and a signal line 207 are provided.

  Next, the operation of each component of the moving picture encoding apparatus shown in FIG. 1 will be described. Note that the moving image handled here is 30 frames per second, and the image of each frame is monochrome moving image data in which the luminance value of each pixel is 8 bits. Therefore, description will be made assuming that such moving image data is taken into a moving image encoding device for 4 seconds (120 frames in total) and encoded. That is, in this moving image encoding apparatus, moving image data of 30 frames per second input from the moving image data input unit 201 is encoded in units of frames, and finally the encoded data is stored in the secondary storage device 206. To do.

  First, moving image data for 4 seconds is input from the moving image data input unit 201 at 30 frames per second. The moving image data input unit 201 is an imaging part such as a digital camera, and can be realized by an imaging device such as a CCD and various image adjustment circuits such as gamma correction and shading correction. The moving image data input unit 201 sends the input moving image data to the discrete wavelet transform unit 202 frame by frame. In the following description, for the sake of convenience, each frame data is given a number from 1 in the order of input, and each frame is identified by a number such as frame 1, frame 2,. Also, the horizontal pixel position (coordinates) in each frame is x, the vertical pixel position is y, and the pixel value at the pixel position (x, y) is represented by P (x, y).

  One frame of image data input from the moving image data input unit 201 is appropriately stored in an internal buffer (not shown) by the discrete wavelet transform unit 202, and two-dimensional discrete wavelet transform is performed. The two-dimensional discrete wavelet transform is realized by applying a one-dimensional discrete wavelet transform in the horizontal and vertical directions. FIG. 2 is a diagram illustrating a process of generating four subbands by performing two-dimensional discrete wavelet transform on one frame of image data.

  FIG. 2A shows image data to be encoded. First, a one-dimensional discrete wavelet transform is applied to the encoding target image as shown in the figure in the vertical direction, and a low frequency subband L and a high frequency subband H are obtained as shown in FIG. Decompose.

  Next, by applying a horizontal one-dimensional discrete wavelet transform to each subband, the subbands are decomposed into four subbands LL, HL, LH, and HH as shown in FIG.

  In the discrete wavelet transform unit 202 of the moving picture coding apparatus, the two-dimensional discrete wavelet transform is repeatedly applied to the subband LL obtained by the above-described two-dimensional discrete wavelet transform. As a result, the encoding target image can be decomposed into seven subbands LL, LH1, HL1, HH1, LH2, HL2, and HH2. FIG. 3 is a diagram showing seven subbands obtained by two two-dimensional discrete wavelet transforms.

  In the moving picture coding apparatus, a coefficient in each subband is represented as C (Sb, x, y). Here, Sb represents the type of subband, and is one of LL, LH1, HL1, HH1, LH2, HL2, and HH2. Further, (x, y) represents the coefficient position (coordinates) in the horizontal direction and the vertical direction when the coefficient position of the upper left corner in each subband is (0, 0).

  This moving image encoding apparatus includes two methods as one-dimensional discrete wavelet transform for N one-dimensional signals x (n) (where n is an integer from 0 to N−1) in the discrete wavelet transform unit 202. One is conversion by the integer type 5 × 3 filter shown in the equations (1) and (2), and the other is conversion by the real number type 5 × 3 filter shown in the equations (3) and (4).

h (n) = x (2n + 1)
−floor {(x (2n) + x (2n + 2)) / 2} (1)
l (n) = x (2n)
+ Floor {(h (n-1) + h (n) +2) / 4} (2)
h (n) = x (2n + 1)
-(X (2n) + x (2n + 2)) / 2 (3)
l (n) = x (2n)
+ (H (n-1) + h (n)) / 4 (4)
Further, h (n) represents a high frequency subband coefficient, l (n) represents a low frequency subband coefficient, and floor {R} represents a maximum integer value not exceeding the real number R. It should be noted that x (n) at both ends (n <0 and n> N−1) of the one-dimensional signal x (n) required in the calculations of the expressions (1), (2) and (3), (4). Is obtained from the value of the one-dimensional signal x (n) (n = 0 to N−1) by a known method.

  Whether an integer type 5 × 3 filter or a real number type 5 × 3 filter is applied can be specified in units of frames by a filter selection signal input from the outside of the apparatus via the signal line 207. For example, when the filter selection signal input from the signal line 207 is “0”, the target frame is decomposed by an integer type 5 × 3 filter, and when the filter selection signal is “1”, the target frame is a real type 5 × 3. For example, it is decomposed by a filter.

  In the coefficient quantization unit 203, the coefficient C (S, x, y) of each subband generated by the discrete wavelet transform unit 202 is quantized using a quantization step delta (S) determined for each subband. To do. Here, when the quantized coefficient value is expressed as Q (S, x, y), the quantization processing performed by the coefficient quantization unit 203 can be expressed by Expression (5).

Q (S, x, y) = sign {C (S, x, y)}
× floor {| C (S, x, y) | / delta (S)} (5)
Here, sign {I} is a function representing the sign of the integer I, and returns 1 when I is positive and -1 when negative. Further, floor {R} represents the maximum integer value not exceeding the real number R. However, the above-described quantization processing is applied only when a real type 5 × 3 filter is selected and used in the discrete wavelet transform unit 202, and an integer type 5 × 3 filter is obtained by a filter selection signal input from the signal line 207. Is selected, the coefficient C (S, x, y) is output as a quantized coefficient value. That is, in this case, Q (S, x, y) = C (S, x, y).

  The bit plane encoding unit 204 encodes the coefficient value Q (S, x, y) quantized by the coefficient quantization unit 203. Various methods have been proposed as coding methods such as a method of facilitating random access by dividing the coefficients of each subband into blocks and coding them separately, but here, for the sake of simplicity of explanation. Encode in subband units.

  The encoding of the quantized coefficient value Q (S, x, y) (hereinafter simply referred to as “coefficient value”) of each subband is performed by using the coefficient value Q (S, x, y) in the subband. The absolute value is expressed by a natural binary number, and binary arithmetic coding is performed by giving priority to the bit plane direction from the upper digit to the lower digit. In the following description, the n-th bit from the bottom in the case where the coefficient value Q (S, x, y) of each subband is expressed in a natural binary notation is expressed as Qn (x, y). Note that a variable n representing a binary digit is referred to as a bit plane number, and the LSB (least significant bit) of the bit plane number n is the 0th digit.

FIG. 4 is a flowchart for explaining a processing procedure for encoding the subband S by the bit plane encoding unit 204. As shown in FIG. 4, first, the absolute value of the coefficient in the subband S to be encoded is checked to obtain its maximum value Mabs (S) (step S601). Next, the number of digits N _BP (S) required when expressing Mabs (S) in binary number to represent the absolute value of the coefficient in the subband is obtained using equation (6) (step S602). .

N _BP (S) = ceil {log2 (Mabs (S + 1))} (6)
However, ceil {R} represents a minimum integer value greater than or equal to the real number R.

Next, the effective digit number N _BP (S) is substituted into the bit plane number n (step S603). Then, 1 is subtracted from the bit plane number n to obtain n−1 and is substituted for n (step S604).

  Further, the n-th bit plane is encoded using a binary arithmetic code (step S605). When each bit in the bit plane is encoded, it is classified into several states (contexts) from the encoded information, and encoded by different appearance probability prediction models. In the moving picture coding apparatus, MQ-Coder is used as an arithmetic code to be used. For the procedure for encoding a binary symbol generated in a certain state (context) S using this MQ-Coder, or for the initialization procedure and termination procedure for arithmetic coding processing, the international standard ISO for still images Since it is described in detail in the / IEC 15444-1 recommendation, the description thereof is omitted here.

In the moving picture encoding apparatus, the arithmetic encoder is initialized at the start of encoding of each bit plane, and the termination process of the arithmetic encoder is performed at the end. Also, immediately after “1” encoded at the beginning of each coefficient, the sign of the coefficient is represented by 0 and 1, and is arithmetically encoded. Here, 0 is set for positive and 1 is set for negative. For example, when the coefficient is −5 and the number of significant digits N _BP (S) of the subband S to which this coefficient belongs is 6, the absolute value of the coefficient is represented by a binary number 00101, and the upper digit is encoded by encoding each bit plane. Are encoded from 1 to the lower digits. When the second bit plane is encoded (in this case, the fourth digit from the top), the first “1” is encoded, and immediately thereafter, the positive / negative code “1” is arithmetically encoded.

  Next, it is determined whether or not the bit plane number n is 0 (step S606). As a result, when n is 0, that is, when the LSB plane is encoded in step S605, the subband encoding process is terminated. In other cases, the process proceeds to step S604.

  Through the processing described above, all the coefficients of the subband S can be encoded, and a code string CS (S, n) corresponding to each bit plane n can be generated. The generated code string CS (S, n) is sent to the code string forming unit 205 and temporarily stored in a buffer (not shown) in the code string forming unit 205.

  In the code sequence forming unit 205, when the bit plane encoding unit 204 finishes encoding all the subband coefficients and stores all the code sequences in the internal buffer, the code sequence stored in the internal buffer in a predetermined order. Is read. Then, necessary additional information is inserted to form a code string corresponding to one frame and output to the secondary storage device 206.

  The final code string generated by the code string forming unit 205 includes a header and encoded data layered in three levels, level 0, level 1, and level 2. The header requires the number of pixels in the horizontal and vertical directions of the image, the number of application times of the two-dimensional discrete wavelet transform, the information specifying the selected filter, the quantization step delta (S) of each subband, etc. The additional information is stored.

The encoded data of level 0 is composed of a code string of CS (LL, N _BP (LL) -1) to CS (LL, 0) obtained by encoding the coefficients of the subband LL.

Level 1 is obtained by encoding CS (LH1, N _BP (LH1) -1) to CS (LH1, 0), CS (HL1, N) obtained by encoding the coefficients of the subbands LH1, HL1, and HH1. _BP (HL1) -1) from CS consisting of _{(HL1,0), CS (HH1,} N BP (HH1) -1) with CS (HH1,0).

Further, level 2 is obtained by encoding CS (LH2, N _BP (LH2) -1) to CS (LH2, 0), CS (HL2, N) obtained by encoding the coefficients of the LH2, HL2, and HH2 subbands. _BP (HL2) -1) from CS consisting of _{(HL2,0), CS (HH2,} N BP (HH2) -1) with CS (HH2,0).

  FIG. 5 is a diagram illustrating a detailed structure of a code string corresponding to one frame data generated in the code string forming unit 205.

  The code string configured in this way can obtain a restored image having the original 1/4 resolution by decoding the header and the level 0 encoded data on the decoding side. In addition, a restored image having an original resolution of 1/2 can be obtained by decoding by adding the encoded data of level 1. Further, when decoding is performed by adding up to level 2 encoded data, the image can be decoded by gradually increasing the resolution so that a restored image of the original resolution can be obtained.

  On the other hand, when only a few high-order bit planes of bit-level encoded data of each level are decoded, a rough decoded image is gradually increased. The conversion coefficient of each subband can be restored with increased accuracy, and the decoded image quality can be improved.

  The secondary storage device 206 is a storage device such as a hard disk or a memory, for example, and stores therein the code string generated by the code string forming unit 205. In the secondary storage device 206, the code sequences of the frames output from the code sequence forming unit 205 are connected to form encoded data of moving image data.

  A preferred embodiment of the image decoding apparatus of the present invention for decoding the encoded data of each frame encoded by the above moving image encoding apparatus will be described below.

[First Embodiment]
FIG. 14 is a diagram showing a basic configuration of an image decoding apparatus according to this embodiment.

  Reference numeral 1401 denotes a CPU that controls the entire apparatus using programs and data stored in the RAM 1402 and ROM 1403 and executes image decoding processing described later.

  A RAM 1402 has an area for storing programs and data downloaded from the external device via the external storage device 1407, the storage medium drive 1408, or the I / F 1409, and when the CPU 1401 executes various processes. A work area is also provided.

  Reference numeral 1403 denotes a ROM which stores a boot program, a setting program for the apparatus, and data.

  Reference numerals 1404 and 1405 denote a keyboard and a mouse, respectively. Various instructions can be input to the CPU 1401.

  A display device 1406 includes a CRT, a liquid crystal screen, and the like, and can display information such as images and characters.

  Reference numeral 1407 denotes an external storage device, which is a large-capacity information storage device such as a hard disk drive device, where an OS, a program for image decoding processing to be described later, and encoded data of a moving image having a decoding target image as each frame These programs and data are loaded into a predetermined area on the RAM 1402 under the control of the CPU 1401.

  A storage medium drive 1408 reads programs and data recorded on a storage medium such as a CD-ROM or DVD-ROM and outputs them to the RAM 1402 or the external storage device 1407. It should be noted that a program for image decoding processing to be described later, moving image data having an image to be decoded as each frame, or the like may be recorded on this storage medium. In this case, the storage medium drive 1408 is controlled by the CPU 1401. Under control, these programs and data are loaded into a predetermined area on the RAM 1402.

  Reference numeral 1409 denotes an I / F, which connects an external device to the present apparatus through the I / F 1409 and enables data communication between the present apparatus and the external apparatus. For example, the moving picture encoding apparatus can be connected to the I / F 1409, and the encoded data of the moving picture generated by the moving picture encoding apparatus can be input to the RAM 1402 or the external storage device 1407 of the present apparatus.

  A bus 1410 connects the above-described units.

  FIG. 9 is a block diagram showing a functional configuration of the image decoding apparatus according to the present embodiment. The same parts as those in FIG. As shown in FIG. 9, the image decoding apparatus according to the present embodiment includes a code string reading unit 101, a bit plane decoding unit 102, an inverse discrete wavelet transform unit 104, a moving image data output unit 105, a decoding processing time measurement unit 106, A non-decoding subband determination unit 107. Although the configuration shown in FIG. 9 may be realized by hardware, in the present embodiment, each unit shown in FIG. 9 is realized by a program that causes a computer to realize the function of each unit, and this program is stored in the external storage described above. It is assumed that the RAM 1402 is loaded from an external device via the device 1407, the storage medium drive 1408, or the I / F 1409.

  In the figure, reference numeral 205 denotes the above-mentioned secondary storage device, from which the encoded data of the moving image is input to this device. The secondary storage device 206 is, for example, the external storage device 1407 or the storage device. Assume that the media drive 1408 or an external device connected to the present apparatus via the I / F 1409 is loaded with the encoded data of the moving image into the RAM 1402 from either of them. Note that the encoded data of the moving image is generated by the moving image encoding device as described above.

  Hereinafter, processing performed by the image decoding apparatus according to the present embodiment will be described with reference to FIG.

  The moving image encoded data to be decoded by the image decoding device according to the present embodiment is generated by the above-described moving image encoding device. In generating the moving image encoded data, all frames are generated. Use an integer type 5x3 filter. That is, a signal for selecting an integer type 5 × 3 filter is input from the signal line 207 of the above-described moving image encoding apparatus to encode moving image data.

  First, the image decoding apparatus according to the present embodiment extracts and decodes encoded image data of an arbitrary frame as a sample from moving image encoded data to be decoded, and measures the time required for decoding processing of each subband. To estimate the decoding processing time.

  In general, decoding of moving image encoded data is performed in units of frames in the moving image encoded data. The code string reading unit 101 illustrated in FIG. 9 reads the encoded data of the frame of interest from the moving image encoded data stored in the secondary storage device 206 and stores the encoded data in the RAM 1402. Reading is basically performed in the order of frame 1 and frame 2, but in this embodiment, in order to measure the time required for the decoding processing of each subband and estimate the decoding processing time, At the start, first, encoded data of an arbitrary frame is read as a sample and stored in the RAM 1402. In the following description, a frame to be decoded (hereinafter referred to as a sample frame) for estimation of the decoding processing time is assumed to be the 60th frame for the sake of convenience. However, the present invention is not limited to this.

  The bit plane decoding unit 102 reads the encoded data stored in the RAM 1402 in the subband order, and decodes the quantized transform coefficient data Q (S, x, y). The processing in the bit plane decoding unit 102 is the reverse of the processing performed by the bit plane encoding unit 204 shown in FIG.

  That is, the bit plane encoding unit 204 binary-encodes each bit of the absolute value of the coefficient from the upper bit plane to the lower bit plane using a predetermined context. On the other hand, the bit plane decoding unit 102 similarly decodes binary arithmetic encoded data from the upper bit plane to the lower bit plane using the same context as that used for encoding, and restores each bit of the coefficient. In addition, with respect to the sign of the coefficient, arithmetic codes are decoded using the same context at the same timing as when encoding.

  However, at this time, data F (S) indicating whether to decode for each subband is input from the non-decoding subband determination unit 107, and the bit plane decoding unit 102 refers to this data F (S), Each bit plane is decoded for the subband S with F (S) = 1, and the subband S with F (S) = 0 is not decoded. The F (S) data is generated by the non-decoding subband determination unit 107, and this generation will be described later.

  Note that for a sample frame that is first decoded by the image decoding apparatus according to the present embodiment, F (S) = 1 is set for all subbands S, and all subbands are decoded. It is assumed that F (S) data for the sample frame is loaded in advance in the RAM 1402.

  In the inverse discrete wavelet transform unit 104, inverse transform of the wavelet transform process performed by the discrete wavelet transform unit 202 in FIG. 1 is performed to restore the frame data. In the present embodiment, as described above, moving image encoded data generated using an integer type 5 × 3 filter in all frames is a decoding target, and thus inverse transformation corresponding to equations (1) and (2) is performed. .

  The moving image data output unit 105 outputs the restored image data output from the inverse discrete wavelet transform unit 104 to an external device or the display device 1460 via the I / F 1409. However, since the sample frame data that is first decoded by the image decoding apparatus according to the present embodiment is decoded for estimation of the decoding processing time, the decoded frame data is discarded and is not output.

  When reproducing and displaying a moving image, each frame data is displayed at a predetermined time. Since the output from the inverse discrete wavelet transform 104 depends on the time required for the decoding process, it does not synchronize with the time to be displayed. For this reason, it is necessary to store the decoded frame data in a buffer and adjust the display time. For example, the moving image data output unit 105 outputs the restored image data to an external device via the I / F 1409, and performs buffer storage processing for adjustment with the display time of the moving image decoding device of the first embodiment. It may be performed externally or may be performed inside the moving image data output unit 105.

  As a specific example of the moving image data output unit 105, a case where a display is connected to the moving image data output unit 105 to display a moving image will be described. FIG. 15 is a diagram showing one form of the moving image data output unit 105 and connection to a display.

  In the figure, 1601 is a buffer capable of storing a plurality of frame data, 1602 is a display interface, and 1603 is a display. A display 1603 corresponds to the display device 1406 in FIG. 14, and a connection line between the display interface 1602 and the display 1406 corresponds to the bus 1410. The buffer 1601 stores the restored image data output from the inverse discrete wavelet transform unit 104 at variable time intervals in order. When storing restored image data, the frame number of the restored image data to be stored and the address to be stored are held so that they can be sequentially retrieved later. The display interface 1602 sequentially retrieves the restored image data from the buffer 1601 at a certain time interval (for example, every 1/30 seconds for a moving image of 30 frames per second) according to the frame rate of the moving image data. To display. The extracted frame data is erased from the buffer 1601. The buffer 1601 plays a role of adjusting the difference between the input time of the restored image data from the inverse discrete wavelet transform unit 104 and the time interval of data extraction by the display interface.

  In the moving picture decoding apparatus according to the present embodiment, control is performed so that the average decoding processing time of one frame becomes the target decoding time T by processing described later. However, since the time taken to restore the data of each frame varies, the number of frame data stored in the buffer 1601 changes. In order to prevent frame data from being prepared in the buffer at the display time, playback and display are started after a predetermined time from the start of video decoding, or playback is started after a predetermined number of frames are stored in the buffer. Adjust the playback start timing. If the buffer capacity is limited, it is necessary to control the frame data decoding process to be stopped when a predetermined number or more of frame data is stored in the buffer.

  The decoding processing time measuring unit 106 measures the time (decoding time Dt (SI)) for decoding the coefficient of each subband of the sample frame by the bit plane decoding unit 102. Here, SI is an index value that identifies a subband. That is, the decoding processing time measuring unit 106 measures the decoding time Dt (SI) for each decoding unit when the decoding unit of the decoding processing by the bit plane decoding unit 102 is “subband”. Based on this measurement, the decoding processing time measuring unit 106 constructs a table as shown in FIG. 10 and stores the data of this table in the RAM 1402.

  FIG. 10 is a diagram illustrating a configuration example of a table representing a relationship between the subband index SI and the coefficient decoding time (decoding time required) of the subband S. The actual measurement time is stored in the portion from Dt (0) to Dt (6). With reference to this table, for example, the coefficient decoding time of the subband HH1 (SI = 3) can be specified as Dt (3).

  Also, the decoding processing time measuring unit 106 starts from the time when the code string reading unit 101 starts reading the encoded image data for one frame from the secondary storage device 206, and the restored image data of this frame is the moving image data output unit 105. The time Dt required to be output from the first frame is measured as “the time required for decoding the encoded image data of one frame” and output to the non-decoding subband determination unit 107. In this embodiment, the decoding processing time measuring unit 106 first measures a time Dt related to decoding of the encoded image data of the sample frame.

  The decoding processing time measuring unit 106 measures the time Dt except for the first frame in the moving image and outputs the time Dt to the non-decoding subband determining unit 107.

  The non-decoding subband determination unit 107 stores in advance in the RAM 1402 the upper limit value of the time required for the decoding processing time Dt of the sample frame output from the decoding processing time measurement unit 106 to restore and output one frame of image data. A process of determining a subband not to be decoded among the subbands constituting one frame of encoded image data is performed so as to be equal to or less than the stored value. More specifically, it is determined whether to decode each subband.

  Here, in the RAM 1402, in advance of this processing, a variable M indicating the number of non-decoding subbands, data of a decoding flag F (S) indicating whether or not each subband is to be decoded, and It is assumed that a variable T representing an upper limit value (target decoding processing time) required for restoring and outputting one frame of image data and a variable ΔT representing a time difference are held.

  Then, a subband not to be decoded is determined by processing to be described later, and data of a decoding flag F (S) reflecting this is generated.

  FIG. 11 is a diagram illustrating a configuration example of a table representing the decoding flag F (S) for each subband S. The figure shows a table when the number of non-decoding subbands M = 3, and M subbands are set as non-decoding subbands in order from the top in the arrangement of the subbands in the figure. In processing to be described later, the table shown in FIG. 11 is created, and the data of the created table is stored in the RAM 1402.

  FIG. 12 is a flowchart showing a flow of decoding processing of moving image encoded data by the image decoding apparatus according to the present embodiment. Note that the program according to the figure is loaded in the RAM 1402, and the CPU 1401 executes this, so that the image decoding apparatus according to the present embodiment can perform processing according to the flowchart shown in FIG. Hereinafter, the image decoding process performed by the image decoding apparatus according to the present embodiment will be described in more detail with reference to the flowchart shown in FIG.

  First, the encoded image data of the sample frame read from the secondary storage device 206 by the code string reading unit 101 is decoded by the bit-plane decoding unit 102 and the inverse discrete wavelet transform unit 104, and the sample is sampled by the decoding processing time measurement unit 106. Decoding process Dt (SI) of each subband constituting the encoded image data of the frame and a decoding process which is a time required to decode and output all the subbands constituting the encoded image data of the sample frame Time Dt is measured (step S1900).

  Next, based on each time obtained in step S1900, the non-decoding subband determining unit 107 determines the number M of non-decoding subbands (step S1901). When the actual decoding processing time Dt is Dt ≦ T with respect to the target decoding processing time T, the decoding time of one frame is within the target decoding processing time even if all the subbands constituting each frame are decoded. Therefore, M = 0 (the number of non-decoding subbands is 0, that is, all subbands are decoded) is set.

  On the other hand, when Dt> T, all subbands constituting one frame are decoded, and the time required to restore and output one frame of image data does not fall within the target decoding processing time. In view of this, a process of sequentially including the low-resolution subbands in the non-decoding subbands among the subbands constituting one frame is performed. In this way, since the number of subbands to be decoded can be reduced, the time required to restore and output one frame of image data can be kept within the target decoding processing time.

  Specifically, the number M of decoding subbands is reduced in the order of subband numbers 0 to 6 until Dt ≦ T. That is, the minimum M satisfying the following expression is obtained, and this is set as the number M of non-decoding subbands (step S1901).

With the above processing, the number M of non-decoding subbands that will be able to be decoded and output within the target decoding processing time is determined. The above process determines the number of non-decoding subbands, that is, the subbands that are not decoded. In other words, this is equivalent to determining the number of subbands to be decoded.
Next, the encoded image data of each frame is sequentially decoded from the beginning of each frame (here, frame 1), but before that, the time difference ΔT is set to 0 (step S1902). .

  Next, the non-decoding subband determination unit 107 determines a decoding flag F (S) based on the number M of non-decoding subbands determined in step S1901 (step S1903). The processing in step S1903 will be described using the table of FIG. 11 as an example. For example, if M = 3 is determined in step S1901, F (S) corresponding to S = HH2, LH2, HL2, ie, three F ( S) becomes 0, and other F (S) becomes 1. The table data generated in this way is stored in the RAM 1402 as described above.

  Subsequently, the code string reading unit 101 reads out the encoded image data of each frame from the secondary storage device 206 and outputs the encoded image data to the subsequent bit plane decoding unit 102. The bit plane decoding unit 102 stores the data in the F (S) table. Referring to FIG. 4, in the encoded image data of each frame input from the code string reading unit 101, only the bit plane in the subband to be decoded is decoded, and the inverse discrete wavelet transform unit 104 is further used using the decoding result. Performs inverse discrete wavelet transform, and the decoding result of each frame, that is, the image data of each frame is output to the moving image data output unit 105 (step S1904).

  The decoding processing time measurement unit 106 measures a time related to a series of processing in step S1904 (time required to restore and output one frame of image data performed in step S1904) Dt, and performs non-decoding sub The band determination unit 107 is notified (step S1905).

  The non-decoding subband determination unit 107 calculates a difference between the target decoding time T for one frame and the actual decoding processing time Dt, and adds the difference to the held time difference ΔT (step S1906). In this time difference Δt, every time one frame of encoded image data is decoded, the “difference between the target decoding time T of one frame and the actual decoding processing time Dt” is accumulated.

  Next, the number M of non-decoding subbands is updated according to the value of ΔT (step S1907). That is, if ΔT is larger than the value Dt (M−1) × n obtained in advance and stored in the RAM 1402, the value held by the variable M, that is, 1 is subtracted from the number of non-decoding subbands, and the value of the variable M Make it smaller.

  n is an arbitrary number determined in advance. The larger the value, the less the change in the number of decoding subbands. ΔT is larger than a predetermined threshold value (Dt (M−1) × n in the present embodiment) because the total of the target decoding processing time T (T × G when the number of decoding processing so far is G) Is the case where the total sum of the decoding times Dt actually taken (Dt × G, where G is the number of decoding processes so far) is small. It can be determined that there is room.

  Therefore, in order to improve the decoding image quality, the value of the number M of non-decoding subbands is decreased by one (in other words, the number of subbands to be decoded is increased by one). Note that the number by which the number of non-decoding subbands is subtracted is not limited to “1”, and a number greater than or equal to “2” may be used depending on the threshold value.

  Here, Dt (M−1) × n used as a threshold value in the present embodiment is a measure of the required decoding time of a subband that will be newly decoded by reducing the number M of non-decoding subbands by one. It is.

  On the other hand, if ΔT is smaller than the value Lq (Lq <0) obtained in advance and stored in the RAM 1402, 1 is added to M to increase the value.

  The reason why ΔT becomes smaller than a predetermined threshold value (Lq in the present embodiment) is actually the sum of the target decoding processing time T (T × G where the number of decoding processing so far is G). This is a case where the sum total of the decoding times Dt (Dt × G when the number of decoding processes so far is set to G) is large. In this case, it can be determined that there is a delay in the entire processing.

  Therefore, the number of non-decoding subbands is increased in order to shorten the decoding time of one frame. However, the range of the value of M is from 0 to 7, and is 0 when it is smaller than 0 by the above update process, and 7 when it is larger than 7.

  The processing from step S1904 to step S1908 is performed for all the frames of the moving image (or the number of frames specified by the keyboard 1402, the mouse 1405, etc.). If all frames are not processed, the process returns to step S1904 to repeat the subsequent processes.

  Through the above process, the decoding process can be performed by determining the number of subbands as a decoding unit.

  From the above description, the image decoding apparatus according to the present embodiment estimates the required subband decoding time in advance, and calculates the non-determining time from the decoding required time, the time required for decoding one frame, and the target decoding time. By determining the number M of decoding subbands, it is possible to control the decoding processing time while suppressing the visual defect of the reproduced image as much as possible.

[Second Embodiment]
In the first embodiment, in order to keep the decoding time of one frame within a predetermined time, the time required for decoding for each subband is estimated, and the number of non-decoding subbands is determined. That is, the decoding unit is a subband. In the present embodiment, in order to achieve the same object, this decoding unit is set as a “bit plane”, parameters for controlling the decoding image quality in stages (hereinafter referred to as Q factor) are set, and decoding of each Q factor is performed. Estimate the required time and determine the number of non-decoding bitplanes. In the image decoding apparatus according to the present embodiment, 10 stages of Q factors from 0 to 9 are set, and the number of non-decoding bit planes of each subband is determined in advance for each Q factor. An example of the Q factor will be described later.

  In the image decoding apparatus according to the present embodiment, encoded image data that has undergone subband decomposition using a real-type 5 × 3 filter is set as a decoding target. That is, in the above-described moving image encoding apparatus, a signal for selecting a real type 5 × 3 filter is input from the signal line 207, and each frame of the moving image is encoded. Also, the quantization step Delta (S) of each subband used when encoding the image of each frame is the same for all frames.

  The basic configuration of the image decoding apparatus according to this embodiment is the same as that of the first embodiment, that is, the same as the configuration shown in FIG.

  FIG. 7 is a block diagram illustrating a functional configuration of the image decoding apparatus according to the present embodiment. The same parts as those in FIGS. 1 and 9 are given the same numbers. As shown in FIG. 7, the image decoding apparatus according to the present embodiment includes a code string reading unit 904, a bit plane decoding unit 102, a coefficient inverse quantization unit 901, an inverse discrete wavelet transform unit 902, a moving image data output unit 105, A decoding processing time measuring unit 106 and a non-decoding bit plane determining unit 903 are provided. Although the configuration shown in FIG. 7 may be realized by hardware, in the present embodiment, each unit shown in FIG. 7 is realized by a program that causes a computer to realize the function of each unit, and this program is stored in the above external storage. It is assumed that the RAM 1402 is loaded from an external device via the device 1407, the storage medium drive 1408, or the I / F 1409.

  In the figure, reference numeral 205 denotes the above-mentioned secondary storage device, from which the encoded data of the moving image is input to this device. The secondary storage device 206 is, for example, the external storage device 1407 or the storage device. Assume that the media drive 1408 or an external device connected to the present apparatus via the I / F 1409 is loaded with the encoded data of the moving image into the RAM 1402 from either of them. Note that the encoded data of the moving image is generated by the moving image encoding device as described above.

  Hereinafter, processing performed by the image decoding apparatus according to the present embodiment will be described with reference to FIG.

  First, the image decoding apparatus according to the present embodiment extracts encoded image data of an arbitrary frame from moving image encoded data to be decoded as a sample, decodes it with Q factors of various values, and outputs each Q The decoding time corresponding to the factor value is measured.

  In general, decoding of moving image encoded data is performed in units of frames in the moving image encoded data. The code string reading unit 904 illustrated in FIG. 7 reads the encoded data of the frame of interest from the moving image encoded data stored in the secondary storage device 206 and stores the encoded data in the RAM 1402. Reading is basically performed in the order of frame 1 and frame 2, but in this embodiment, before performing the decoding process of each frame, the encoded image data of the frame as the sample is subjected to various values of Q. In order to perform decoding with a factor and measure the decoding time according to the value of each Q factor, encoded data of an arbitrary frame is read as a sample and stored in the RAM 1402. In the following description, this frame (hereinafter referred to as a sample frame) is referred to as the 60th frame for convenience, but is not limited to this.

  Further, when reading the encoded image data of the frame, the code string reading unit 904 refers to the header of the encoded image data, and from here the quantization step delta (S) at the time of encoding each subband is performed. Reading and storing in the RAM 1402 are also performed.

  The bit plane decoding unit 102 reads the encoded data stored in the RAM 1402 in the subband order, performs the same processing as in the first embodiment, and obtains the quantized transform coefficient data Q (S, x, y). Decrypt. The processing in the bit plane decoding unit 102 is the reverse of the processing performed by the bit plane encoding unit 204 shown in FIG.

  Here, in the first embodiment, the non-decoding subband determination unit 107 determines data F (S) indicating whether or not to decode for each subband, and the bitplane decoding unit 102 determines the bitplane for each frame. When performing the decoding process, decoding / non-decoding is switched for each subband according to F (S).

  In the present embodiment, the non-decoding bit plane determination unit 903 determines “the number of non-decoding bit planes ND (S)” indicating how many lower-order bit planes are not decoded for each subband. ND (S) is data indicating that the decoding process is not performed on the bit planes corresponding to the lower ND (S) bits of the subband S.

  Therefore, the bit plane decoding unit 102 refers to the data ND (S) when performing the bit plane decoding process for each frame, thereby decoding the bit planes corresponding to the lower ND (S) bits of the subband S. Can be avoided. Processing for obtaining such ND (S) will be described later.

  Note that for the sample frame that is first decoded by the image decoding apparatus according to the present embodiment, ND (S) = 0 is set for all subbands S, and bitplanes of all subbands are decoded. It is assumed that ND (S) data for the sample frame is loaded in advance in the RAM 1402.

  In the coefficient inverse quantization unit 901, the quantization step delta (S) determined for each subband and the quantized coefficient value Q (S, x, y) decoded by the bit plane decoding unit 102 are used to The subband coefficient C (S, x, y) is restored.

  In the inverse discrete wavelet transform unit 902, the inverse transform of the wavelet transform process performed by the discrete wavelet transform unit 202 in FIG. 1 is performed to restore the frame data. In the present embodiment, as described above, moving image encoded data generated using a real number type 5 × 3 filter in all frames is a decoding target, and thus inverse transformation corresponding to equations (3) and (4) is performed. .

  The moving image data output unit 105 outputs the restored image data output from the inverse discrete wavelet transform unit 902 to an external device or the display device 1460 via the I / F 1409. However, since the sample frame data that is initially decoded a plurality of times by the image decoding apparatus according to the present embodiment is decoded for estimation of the decoding processing time, the decoded frame data is discarded and is not output.

  The decoding processing time measuring unit 106 starts the bit string decoding unit 102, the coefficient inverse quantization unit 901, the inverse discrete wavelet from the time when the code string reading unit 904 starts reading one frame of code or image data from the secondary storage device 206. The conversion unit 902 decodes the encoded image data of one frame and outputs the image data to the moving image data output unit 105 by changing the Q factor value from 0 to 9, respectively. In this case, the decoding time for one frame is measured. That is, the decoding time for one frame when the Q factor value is 0, the decoding time for one frame when the Q factor value is 1, and 1 when the Q factor value is 9. The decoding time for the frame is obtained.

  In this embodiment, since the sample frame is first decoded, the decoding processing time measuring unit 106 decodes the sample frame when the Q factor value is 0, and decodes the sample frame when the Q factor value is 1. The decoding time of the sample frame when the value of the time,..., Q factor is 9 is obtained.

  FIG. 8 is a diagram showing the decoding time Dt ′ (Q) of the sample frame with respect to each Q factor value. The actual measurement time is stored in the portion from Dt ′ (0) to Dt ′ (9). The decoding processing time measuring unit 106 performs processing for obtaining each decoding time, thereby generating data of the decoding time Dt ′ (Q) of the sample frame obtained for each Q factor value, and stores the data in the RAM 1402.

  Then, the decoding processing time measurement unit 106 obtains the decoding time data from Dt ′ (0) to Dt ′ (9) shown in FIG. Processing for obtaining an increment Dt (Q) of the decoding processing time with respect to the change is performed.

  Specifically, Dt (Q) is obtained according to the following equation (7).

Dt (Q) = Dt ′ (Q + 1) −Dt ′ (Q) (7)
However, 0 ≦ Q ≦ 8.

  Then, the decoding processing time measurement unit 106 stores the obtained data of Dt (0) to Dt (8) in the RAM 1402.

  Also, the decoding processing time measuring unit 106 starts from the time when the code string reading unit 101 starts reading the encoded image data for one frame from the secondary storage device 206, and the restored image data of this frame is the moving image data output unit 105. Is calculated as “time related to decoding of encoded image data of one frame”, and this is equivalent to the above Dt ′ (0). The data is output to the determination unit 107.

  The decoding processing time measuring unit 106 measures the time Dt (that is, Dt ′ (Q)) except for the first frame in the moving image, and outputs it to the non-decoding subband determining unit 107.

  The non-decoding subband determination unit 107 stores in advance in the RAM 1402 the upper limit value of the time required for the decoding processing time Dt of the sample frame output from the decoding processing time measurement unit 106 to restore and output one frame of image data. A process of determining a bit plane not to be decoded among the bit planes of each subband constituting one frame of encoded image data is performed so as to be equal to or less than the stored value.

  Here, in the RAM 1402, the variable Q used as an index in the previous stage of this processing, the variable T representing the target decoding processing time that is the upper limit value of the decoding time of one frame, the variable ΔT representing the time difference, and the Q factor It is assumed that data of a table in which the number of non-decoding bit planes of each subband corresponding to the value is registered is held.

  FIG. 6 is a diagram illustrating a configuration example of a table in which the number of non-decoding bit planes of each subband is registered according to the Q factor value. The table shown in the figure is created in advance and stored in the RAM 1402. Referring to the table of FIG. 9, for example, when “4” is used as the value of the Q factor, ND (S) = 4 for subbands S = HH2, HL2 (LH2) and HH1, that is, the lower 4 bits. The bit plane is not decoded, ND (S) = 3 for subband S = HL1 (LH1), that is, the bit plane for the lower 3 bits is not decoded, and the above for subband S = LL. It can be seen that ND (S) = 1, that is, the bit plane for the lower one bit is not decoded.

  Therefore, in the process described below, the value of the Q factor used in the decoding process is obtained, and the number of non-decoding bit planes for each subband is obtained according to the obtained Q factor value.

  FIG. 13 is a flowchart showing a flow of decoding processing of moving image encoded data by the image decoding apparatus according to the present embodiment. Note that the program according to the figure is loaded in the RAM 1402, and the CPU 1401 executes this, so that the image decoding apparatus according to the present embodiment can perform processing according to the flowchart shown in FIG. Hereinafter, the image wind process performed by the image decoding apparatus according to the present embodiment will be described in more detail with reference to the flowchart shown in FIG.

  First, the coded image data of the sample frame is read from the secondary storage device 206 by the code string reading unit 904 (step S2001). Next, the decoding processing time measuring unit 106 initializes the value Q by substituting the value 9 (step S2002). Then, the decoding processing time measuring unit 106 performs the processing from step S2003 to step S2006 described below until the value of the variable Q becomes negative.

  First, the coded image data of the sample frame read by the code string reading unit 904 in step S2001 is decoded by the bit plane decoding unit 102, the coefficient inverse quantization unit 901, and the inverse discrete wavelet transform unit 902 (step S2003). This decoding process is a decoding process when the value of the Q factor (that is, the value held by the variable Q) is 9.

  The decoding processing time measuring unit 106 measures the time required for the processing in step S2003 (step S2004). Since the value of the variable Q is “9” at this time, the time measured in step S2004 corresponds to the above-described Dt ′ (9). Next, the decoding processing time measuring unit 106 stores the data of Dt ′ (9) measured in step S2004 in the RAM 1402 (step S2005).

  Next, the decoding processing time measuring unit 106 performs processing for substituting the result obtained by subtracting 1 from the value held by the variable Q into the variable Q (step S2006). After the process of step S2006, the value held by the variable Q is “8”.

  Next, the decoding processing time measurement unit 106 determines whether or not the value held by the variable Q is negative (step S2007). If the value is not negative, the processing returns to step S2003, and the above processing is repeated. Here, since the value held by the variable Q is “8” and 8> 0, the process returns to step S2003.

  In step S2003, decoding processing is performed for Q = 8, and in step S2004, the decoding time is measured. Since the value of the variable Q is “8” at this time, the time measured in step S2004 corresponds to the above-described Dt ′ (8).

  In step S2005, a process for obtaining an increment Dt (Q) of the decoding process time with respect to a change in the value of the Q factor is performed according to the equation (7). The processing for obtaining this increment is performed as follows according to the above equation (7).

Dt (8) = Dt ′ (9) −Dt ′ (8)
That is, Dt (8) is obtained using Dt ′ (8) obtained in step S2004 this time and Dt ′ (9) obtained in step S2004 last time. This is the same for all cases where Q <9. In general, Dt (Q) is obtained by using Dt ′ (Q) obtained in the current step S2004 and Dt ′ (Q + 1) obtained in the previous step S2004.

  That is, in step S2005, when Q = 9, the process of obtaining the increment Dt (Q) is not performed, but the process of storing the data of Dt ′ (9) in the RAM 1402 is performed. The increment Dt (Q) is obtained according to (7).

  The processes from step S2003 to step S2006 are performed until Q <0. When Q <0, that is, Dt (0) to Dt (8) are obtained. The non-decoding bit plane determination unit 903 uses these Dt (0) to Dt (8) to obtain the Q factor value used to decode each frame of the moving image data (step S2208).

  When Dt ′ (0) obtained when Q = 0 in step S2004 with respect to the target decoding processing time T, if Dt ′ (0) ≦ T, all subbands constituting each frame are all included. Even if the bit plane is decoded, the decoding time of one frame is within the target decoding processing time, so that Q = 0 (the number of non-decoding bit planes is 0, that is, all bit planes are decoded) is set.

  On the other hand, when Dt ′ (0)> T, the time required to restore and output one frame of image data is the target when all bit planes of all subbands constituting one frame are decoded. It does not fit within the decoding process time. Therefore, a Q factor value is calculated so that the time required to restore and output one frame of image data is within the target decoding processing time. With reference to the table shown in FIG. By decoding other than the non-decoding bit plane defined by the value, the time required to restore and output one frame of image data can be kept within the target decoding processing time.

  Specifically, the minimum Q satisfying the following expression is obtained, and this is set as the value of the Q factor to be used below.

With the above processing, the Q factor value for decoding and outputting each frame within the target decoding processing time can be obtained.
Next, the encoded image data of each frame is sequentially decoded from the beginning of each frame (here, frame 1), but before that, the time difference ΔT is set to 0 (step S2209). .

  Next, the non-decoding subband determination unit 903 determines the number of non-decoding bit planes based on the value of the Q factor obtained in step S2208 (step S2210). As described above, this is processing performed by referring to the data in the table illustrated in FIG. 6 and acquiring the number of non-decoding bit planes in each subband according to the Q value obtained in step S2208.

  Note that the processing in step S2210 determines the number of non-decoding bit planes, that is, the number of bit planes not to be decoded. In other words, this is equivalent to determining the number of bit planes to be decoded. Note that the determined number of non-decoding bit planes for each subband is stored in the RAM 1402 as the ND (S) data.

  Subsequently, the code string reading unit 101 reads out the encoded image data of each frame from the secondary storage device 206 and outputs the encoded image data to the subsequent bit plane decoding unit 102. The bit plane decoding unit 102 converts the ND (S) data into Referring to the encoded image data of each frame input from code string reading unit 101, only the bit plane to be decoded in each subband is decoded, and further, the inverse discrete wavelet transform unit 104 is used using the decoding result. Performs inverse discrete wavelet transform, and the decoding result of each frame, that is, the image data of each frame is output to the moving image data output unit 105 (step S2211).

  The decoding processing time measuring unit 106 measures the time related to the series of processing in step S2211 (the time required to restore and output one frame of image data performed in step S2211) Dt, and the non-decoding sub The band determination unit 903 is notified (step S2212).

  The non-decoding subband determination unit 903 obtains the difference between the target decoding time T for one frame and the actual decoding processing time Dt and adds it to the held time difference ΔT (step S2213). In this time difference Δt, every time one frame of encoded image data is decoded, the “difference between the target decoding time T of one frame and the actual decoding processing time Dt” is accumulated.

  Next, the value of the Q factor (the value held by the variable Q) is updated according to the value of ΔT (step S2214). That is, if ΔT is larger than the value Dt (Q−1) × n obtained in advance and stored in the RAM 1402, 1 is subtracted from the value held by the variable Q, and the value of the variable Q is reduced.

  n is an arbitrary number determined in advance, and the larger the value, the less likely the change of the Q factor value occurs. ΔT is larger than a predetermined threshold value (Dt (Q−1) × n in the present embodiment) because the total of the target decoding processing time T (T × G when the number of decoding processing so far is set to G) Is the case where the total sum of the decoding times Dt actually taken (Dt × G, where G is the number of decoding processes so far) is small. It can be determined that there is room.

  Therefore, in order to improve the decoded image quality, the value of the Q factor is reduced by one (in other words, the number of non-decoded bit planes is reduced based on the table of FIG. 6). Note that the number by which the Q factor value is subtracted is not limited to “1”, and a number greater than or equal to “2” may be used depending on the threshold value.

  Conversely, if ΔT is smaller than the value Lq (Lq <0) obtained in advance and stored in the RAM 1402, 1 is added to the value held by Q to increase the value.

  The reason why ΔT becomes smaller than a predetermined threshold value (Lq in the present embodiment) is actually the sum of the target decoding processing time T (T × G where the number of decoding processing so far is G). This is a case where the sum total of the decoding times Dt (Dt × G when the number of decoding processes so far is set to G) is large. In this case, it can be determined that there is a delay in the entire processing.

  Accordingly, in order to shorten the decoding time of one frame, the value of the Q factor is increased, and as a result, the number of non-decoding bit planes is increased. However, the range of the value of Q is from 0 to 9, and is 0 when it is smaller than 0 by the above update process, and 9 when it is larger than 9.

  The above processing from step S2210 to step S2214 is performed for all the frames of the moving image (or the number of frames specified by the keyboard 1402, the mouse 1405, etc.). If all frames are not processed, the process returns to step S2210 to repeat the subsequent processes.

  Through the above process, the decoding process can be performed by determining the number of bit planes as decoding units.

  As described above, the image decoding apparatus according to the present embodiment performs decoding by changing the image quality parameter for the sample frame, estimates the required decoding time by changing each image quality parameter, and the required decoding time and one frame decoding process By determining this image quality parameter from the difference between the time required for the above and the target decoding time, it is possible to control the decoding processing time while suppressing the visual defect of the reproduced image as much as possible.

<Modification>
For example, in the first and second embodiments, the bit plane encoding is performed in units of subbands. However, the subbands may be divided into blocks and the bit plane encoding may be performed for each block. One bit plane may be encoded by a plurality of passes.

  As a decoding unit, a subband is used in the first embodiment and a bit plane is used in the second embodiment. However, other than this, a code block may be used, and the decoding unit is limited to the first and second embodiments. It is not a thing.

  In the first and second embodiments, the example in which MQ-Coder is used as the binary arithmetic coding method has been described. However, the present invention is not limited to this. For example, other than MQ-Coder, such as QM-Coder An arithmetic encoding method may be applied, or any other binary encoding method may be applied as long as the method is suitable for encoding a multi-context information source.

  Further, the filter for subband decomposition is not limited to the first and second embodiments, and other filters such as a real type 9 × 7 filter may be used. Further, the number of applications is not limited to the first and second embodiments. In the first and second embodiments, the same number of one-dimensional discrete wavelet transforms are applied in the horizontal direction and the vertical direction.

  Furthermore, the structure of the moving image encoded data is not limited to the first and second embodiments, and the order of the code string, the storage form of the additional information, and the like may be changed. For example, it is suitable when JPEG2000 defined in ISO / IEC 15444-1 is used as the encoding method of frame data. It is good as data.

  In the first and second embodiments, the processing time is estimated by decoding an arbitrary frame prior to the decoding process. However, the decoding processing time may be estimated using a plurality of frames as samples. It is also possible to adopt a configuration in which the estimation of the decoding process time is updated during the decoding process.

[Other Embodiments]
Note that the present invention can be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), but a device (for example, a copier, a facsimile machine, etc.) composed of a single device. You may apply to.

  Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or CPU) of the system or apparatus. Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a figure which shows the function structure of the moving image encoding apparatus which produces | generates the moving image encoded data used as the decoding object by the image decoding apparatus used by embodiment of this invention. It is a figure explaining the process which performs two-dimensional discrete wavelet transformation with respect to the image data of 1 frame, and produces | generates four subbands. It is a figure which shows seven subbands obtained by two times of two-dimensional discrete wavelet transforms. 12 is a flowchart for explaining a processing procedure for encoding subband S by bit plane encoding section 204. It is a figure which shows the detailed structure of the code sequence corresponding to one frame data produced | generated in the code sequence formation part. It is a figure which shows the structural example of the table in which the number of non-decoding bit planes of each subband according to the value of Q factor was registered. It is a block diagram which shows the function structure of the image decoding apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows the decoding time Dt '(Q) of the sample frame with respect to the value of each Q factor. It is a block diagram which shows the function structure of the image decoding apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the structural example of the table showing the relationship between the subband index SI and the decoding time (decoding required time) of the subband S. It is a figure which shows the structural example of the table showing the decoding flag F (S) with respect to each subband S. It is a flowchart which shows the flow of the decoding process of the moving image coding data by the image decoding apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the flow of the decoding process of the moving image coding data by the image decoding apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows the basic composition of the image decoding apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows one form of the moving image data output part 105, and the connection to a display.

Claims (12)

An image decoding method for inputting moving image data in which an image of each frame is encoded and decoding the encoded image data of each frame,
A sample frame decoding step of extracting encoded image data of a sample frame from each frame and decoding it in a predetermined decoding unit;
A first measuring step of measuring a time required for decoding the encoded image data of the sample frame;
A determining step for determining the number of decoding units to be decoded so that the time measured in the first measuring step is equal to or less than a predetermined time;
A decoding step of decoding the encoded image data of each frame according to a given number of decoding units;
A second measuring step of measuring a time required for decoding each frame when decoding each frame in the decoding step;
The difference between the predetermined time and the time measured in the second measuring step is accumulated every time each frame is decoded, and is determined in the determining step when the accumulated value exceeds a predetermined value. An update step of updating the number of decoded units,
In the decoding step, the encoded image data of each frame is decoded according to the number of decoding units determined in the determining step or the number of decoding units updated in the updating step. Furthermore, when decoding the encoded image data of the sample frame in the sample frame decoding step, a third measurement step of measuring the time required for decoding for each decoding unit,
In the determination step,
Decoding required for decoding each decoding unit measured in the third measurement step from the time measured in the first measurement step when the time measured in the first measurement step is greater than the predetermined time. Based on the number of times that the time measured in the third measurement step is subtracted from the time measured in the first measurement step when the time is sequentially subtracted and the result of the subtraction becomes equal to or less than the predetermined value, 1 The number of decoding units to be decoded is determined by determining the number included in decoding units that are not decoded when all decoding units constituting a frame are arranged in a predetermined order and counted from one side. The image decoding method according to claim 1. 2. The image decoding method according to claim 1, wherein in the determining step, a decoding unit to be decoded / not to be decoded is determined from among decoding units constituting one frame. 2. The image decoding method according to claim 1, wherein, in the update step, the number of decoding units determined in the determination step is increased when the cumulative value is equal to or greater than a predetermined value. Furthermore, data of a table in which a plurality of sets of parameter values related to image quality and the number of decoding units to be decoded are registered is stored in a memory,
In the sample frame decoding step, the decoding processing of the encoded image data of the sample frame using the parameters is performed for each of the parameters having different values, with the values of the parameters being different.
In the first measurement step, the decoding time by the sample frame decoding step is measured for each parameter of the respective values,
In the determination step,
When the first time measured in the first measurement step when the sample frame is decoded in the sample frame decoding step using the parameter having a predetermined value is larger than the predetermined time,
In each of the decoding times, the difference between adjacent times is sequentially subtracted from the first time, and the value of the parameter is based on the number of subtractions when the subtraction result is equal to or less than the predetermined value. The image decoding method according to claim 1, further comprising: determining a number of decoding units to be decoded corresponding to the determined parameter value by referring to the table. In the update step, when the cumulative value is equal to or greater than a predetermined value, the value of the parameter determined in the determination step is increased, and further, the number of decoding units is determined by referring to the data in the table. The image decoding method according to claim 5, wherein the image decoding method is increased. The image decoding method according to claim 5 or 6, wherein the parameter relating to the image quality is a Q factor. The image decoding method according to any one of claims 1 to 7, wherein the decoding unit includes a bit plane or a subband. Further, the image of each frame decoded in the decoding step is held for a predetermined number of frames, and is displayed at a predetermined time interval corresponding to the frame rate for a display device that displays at a predetermined frame rate. The image decoding method according to claim 1, further comprising a step of sequentially outputting images to the display device. An image decoding device for inputting moving image data in which an image of each frame is encoded and decoding the encoded image data of each frame,
Sample frame decoding means for extracting encoded image data of a sample frame from each frame and decoding it in a predetermined decoding unit;
First measuring means for measuring a time for decoding the encoded image data of the sample frame;
Determining means for determining the number of decoding units to be decoded so that the time measured by the first measuring means is a predetermined time or less;
Decoding means for decoding the encoded image data of each frame according to the given number of decoding units;
Second measuring means for measuring a time required for decoding each frame when the decoding means decodes each frame;
The difference between the predetermined time and the time measured by the second measuring means is accumulated every time each frame is decoded, and the determining means determines when the accumulated value exceeds a predetermined value. Update means for updating the number of decoded decoding units,
The image decoding apparatus characterized in that the decoding means decodes encoded image data of each frame according to the number of decoding units determined by the determining means or the number of decoding units updated by the updating means. A program causing a computer to execute the image decoding method according to any one of claims 1 to 9. A computer-readable storage medium storing the program according to claim 11.

Images