JP3092613B2 - recoding media - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture coding and decoding method in which inter-frame prediction is performed and luminance or color intensity is expressed as a quantized numerical value, and a moving picture coding apparatus and decoding. The present invention relates to a gasifier.

[0002]

2. Description of the Related Art In highly efficient coding of moving images, it is known that inter-frame prediction (motion compensation) utilizing similarity between temporally adjacent frames has a great effect on information compression. The motion compensation system which is the mainstream of the current image coding technology is H.264, which is an international standard of the moving image coding system. 263, half-pixel precision block matching adopted in MPEG1 and MPEG2. In this method, an image to be encoded is divided into a large number of blocks, and a motion vector is obtained for each block in the horizontal and vertical directions using a half length of a distance between adjacent pixels as a minimum unit.

[0005] This processing is expressed as follows using mathematical expressions. A sample value (sample value of intensity of luminance or color difference) at a coordinate (x, y) of a predicted image P of a frame to be coded (current frame) is represented by P (x, y),
A sample value at the coordinates (x, y) of the reference image R (a decoded image of a frame which is temporally close to P and has already been encoded) is defined as R (x, y). Further, it is assumed that x and y are integers, and that a pixel exists at a point where the coordinate value is an integer in P and R. It is also assumed that the pixel sample values are quantized as non-negative integers. At this time, the relationship between P and R is

[0004]

(Equation 1)

[0005] Here, assuming that the image is divided into N blocks, Bi represents a pixel included in the i-th block of the image, and (ui, vi) represents a motion vector of the i-th block.

When the values of ui and vi are not integers, it is necessary to find the intensity value at a point where no pixel actually exists in the reference image. In this case, bilinear interpolation using four peripheral pixels is often used. When this interpolation method is described by a mathematical expression, d is a positive integer, 0 ≦ p,
Assuming that q <d, R (x + p / d, y + q / d) is

[0007]

(Equation 2)

## EQU1 ## However, "//" is a type of division, which is characterized by rounding the result of normal division (division by real number operation) to a nearby integer.

FIG. Configuration Example 10 of Encoder of H.263
Indicates 0. H. H.263 adopts a hybrid coding method (inter-frame / intra-frame adaptive coding method) combining block matching and DCT (discrete cosine transform) as a coding method.

A subtractor 102 calculates a difference between an input image (original image of the current frame) 101 and an output image 113 (described later) of an inter-frame / intra-frame coding changeover switch 119, and outputs an error image 103. This error image is DC
After being converted into DCT coefficients by the T transformer 104, they are quantized by the quantizer 105 to become quantized DCT coefficients 106.
The quantized DCT count is output to the communication path as transmission information, and is also used in the encoder to synthesize an inter-frame predicted image.

The procedure for synthesizing a predicted image will be described below. The above-described quantized DCT coefficient 106 passes through an inverse quantizer 108 and an inverse DCT transformer 109 to become a decoded error image 110 (the same image as the error image reproduced on the receiving side). The output image 113 (described later) of the inter-frame / intra-frame coding changeover switch 119 is added to the adder 111, and the decoded image 112 of the current frame (the same image as the decoded image of the current frame reproduced on the receiving side) is added. Get. This image is temporarily stored in the frame memory 114 and is delayed by a time corresponding to one frame. Therefore, at present, the frame memory 114 outputs the decoded image 115 of the previous frame. The decoded image of the previous frame and the input image 101 of the current frame are input to the block matching unit 116,
Block matching processing is performed.

In block matching, an image is divided into a plurality of blocks, and a portion most similar to the original image of the current frame is extracted from the decoded image of the previous frame for each block, whereby the predicted image 117 of the current frame is synthesized. You.
At this time, a process of detecting how much each block has moved between the previous frame and the current frame (motion estimation process)
Need to do. The motion vector for each block detected by the motion estimation processing is the motion vector information 12.
It is transmitted to the receiving side as 0.

The receiving side can independently synthesize the same predicted image as that obtained on the transmitting side from the motion vector information and the decoded image of the previous frame. Predicted image 117
Is input to the inter-frame / intra-frame coding switch 119 together with the “0” signal 118. This switch switches between inter-frame coding and intra-frame coding by selecting one of the two inputs. When the predicted image 117 is selected (FIG. 2 shows this case), inter-frame encoding is performed. On the other hand, "0"
If a signal is selected, the input image is
Since the encoded data is output to the communication channel, intra-frame encoding is performed. In order for the receiving side to correctly obtain a decoded image, it is necessary to know whether inter-frame coding or intra-frame coding has been performed on the transmitting side. Therefore, the identification flag 121 is output to the communication path. The final H. The H.263 coded bit stream 123 is obtained by multiplexing the information of the quantized DCT coefficient, the motion vector, and the intra-frame / inter-frame identification flag by the multiplexer 122.

FIG. 2 shows an example of the configuration of a decoder 200 for receiving an encoded bit stream output from the encoder of FIG. The received H. The 263 bit stream 217 is separated into a quantized DCT coefficient 201, motion vector information 202, and an intra-frame / inter-frame identification flag 203 by a separator 216. The quantized DCT coefficient 201 is calculated by the inverse quantizer 20.
4 and an error image 206 decoded through the inverse DCT transformer 205. This error image is added to the output image 215 of the inter-frame / intra-frame encoding changeover switch 214 by the adder 207 and output as a decoded image 208. The interframe / intraframe coding changeover switch switches the output according to the interframe / intraframe coding identification flag 203. A predicted image 212 used for performing inter-frame encoding is synthesized by a predicted image synthesis unit 211. Here, a process of moving the position for each block in accordance with the received motion vector information 202 is performed on the decoded image 210 of the previous frame stored in the frame memory 209. On the other hand, in the case of intra-frame coding, the inter-frame / intra-frame coding changeover switch outputs the “0” signal 213 as it is.

[0015]

Problems to be Solved by the Invention The image encoded by H.263 includes one luminance plane (Y plane) having luminance information and two color difference planes (U plane and V plane) having color information (also referred to as color difference information). At this time, if the image has 2 m pixels in the horizontal direction and 2 n pixels in the vertical direction (m and n are positive integers), the Y plane has 2 m pixels in the horizontal direction and 2 n pixels in the vertical direction. , U and V planes are characterized by having m pixels in the horizontal direction and n pixels in the straight line direction. The reason why the resolution of the color difference plane is low is that human vision is characterized by being relatively insensitive to spatial changes in color difference. When such an image is used as an input, 26
In FIG. 3, a code called a macro block is used as a unit. FIG. 3 shows the configuration of a macro block. A macro block is composed of three blocks, a Y block, a U block, and a V block. The size of a Y block 301 having luminance value information is 16 × 16 pixels, and a U block 302 having chrominance information.
And the size of the V block 303 is 8 × 8 pixels.

H. At 263, block matching with half-pixel accuracy is applied to each macroblock. Therefore, assuming that the estimated motion vector is (u, v),
u and v are each obtained using half of the distance between pixels, that is, 1/2 as the minimum unit. FIG. 4 shows a state of the interpolation processing of the intensity values (hereinafter, the “luminance value” and the intensity value of the color difference are collectively referred to as “intensity value”) at this time. H. 2
In 63, when the interpolation of Equation 2 is performed, the result of the division is rounded to the nearest integer, and when the result of the division is a value obtained by adding 0.5 to the integer, a process of rounding up the value away from 0 is performed. Done.

That is, in FIG.
The intensity values of 2, 403 and 404 are La, Lb and L, respectively.
Assuming that c and Ld (La, Lb, Lc and Ld are non-negative integers), positions 405 and 40 at which the intensity values are to be obtained by interpolation.
The intensity values Ia, Ib, Ic, Id of 6, 407, 408 are (I
a, Ib, Ic, and Id are non-negative integers), and are represented by the following formulas.

[0018]

(Equation 3)

However, "[]" represents a process of truncating a decimal part.

At this time, it is considered to calculate an expected value of an error generated by a process of rounding the result of the division to an integer value. The probability that the position at which the intensity value is to be obtained by interpolation is the position 405, 406, 407, or 408 in FIG. At this time, the error in obtaining the intensity value Ia at the position 405 is clearly zero. In addition, the error in finding the intensity value Ib at the position 406 is 0 when La + Lb is an even number, and is 1/2 when La + Lb is an odd number because rounding is performed. Assuming that both the probability that La + Lb becomes an even number and the probability that La + Lb becomes an odd number are ２, the expected value of the error is 0 · 1/2 +
・ · ＝= ４. Intensity value I at position 407
When obtaining c, the expected value of the error is 1/4 as in the case of Ib.
Becomes When calculating the intensity value Ic at the position 408, La +
When Lb + Lc + Ld is divided by 4, too much is 0, 1, 2,
When the error is 3, the errors are 0,-／, 1/2,
If the probabilities that become 1/4 and the excess becomes 0 to 3 are assumed to be equal probabilities, the expected value of the error is 0 · 1 / 4−1 / 4 ·
１ ／ + ／ 1/2 ・ +1/4 ・ １ ／ = 1/8. As described above, if the probabilities that the intensity values at the positions 405 to 408 are calculated are equal probabilities, the expected value of the final error is 0 · １ ／ +＋ １ · ・ · ＋ １ + ／.
1/4 + 1/8 1/4 = 5/32. this is,
This means that every time motion compensation is performed by one block matching, a 5/32 error occurs in the pixel intensity value.

In general, in the case of low-rate coding, it is not possible to secure a sufficient number of bits for coding an inter-frame prediction error, and therefore, the quantization step size of DCT coefficients tends to be large. . Therefore, it is difficult to correct the error generated by the motion compensation by the error coding. In such a case, if the inter-frame coding is continued without performing the intra-frame coding, the above-described error may accumulate, which may have a bad influence such as a reddish reproduction image.

As described above, the number of pixels in the color difference plane is half in both the vertical and horizontal directions. Accordingly, for the U block and the V block, values obtained by dividing the horizontal and vertical components of the motion vector of the Y block by 2 are used. At this time, since the horizontal and vertical components u and v of the motion vector of the original Y block are values that are integral multiples of １ ／, when normal division is performed,
As the motion vector, a value that is an integral multiple of 1/4 will appear. However, since the interpolation of the intensity value when the coordinate value is an integral multiple of 1/4 becomes complicated, H.264 / H. In 263, the motion vectors of the U block and the V block are also rounded to half-pixel accuracy. The rounding method at this time is as follows.

Now, it is assumed that u / 2 = r + s / 4.
At this time, r and s are integers, and s takes a value of 0 or more and 3 or less. When s is 0 or 2, since u / 2 is an integral multiple of 1/2, there is no need to perform rounding. However, when s is 1 or 3, an operation of rounding this to 2 is performed. This is because, by increasing the probability that s becomes 2, the number of times the intensity value is interpolated is increased, and the motion compensation processing has a filtering effect.

If the probability that the value of s before the rounding takes a value of 0 to 3 is 1/4, the probability that s becomes 0 or 2 after the rounding is 1/4, respectively.
And 3/4. The above is the discussion about the horizontal component u of the motion vector, but the same discussion can be applied to the vertical component v.

Therefore, in the U block and the V block, the probability that the intensity value at the position 401 is obtained is 1
The probability that the intensity value at the position of ４/4 ・ = 1/16, 402 and 403 is obtained is 1１ ／ 3/4 = 3/1.
The probability that the intensity value at the position of 6, 404 is obtained is 3/4.
3/4 = 9/16. Using this, the expected value of the error in the intensity value is calculated by the same method as above, and
6 + ／ · 3/16 + ／ · 3/16 + ／ · 8.9
/ 16 = 21/128, which causes a problem of accumulation of errors when intra-frame encoding is continued as in the case of the Y block described above.

In the moving picture coding and decoding method in which the inter-frame prediction is performed and the luminance or color intensity is expressed as a quantized numerical value, the error in quantizing the luminance or the color intensity in the inter-frame prediction is determined. May accumulate.
An object of the present invention is to improve the image quality of a reproduced image by preventing the accumulation of the error.

[0027]

[MEANS FOR SOLVING THE PROBLEMS]
By performing an operation to cancel the generated error, accumulation of the error is prevented.

[0028]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First, consideration will be given to the case where accumulation of rounding errors described in "Prior Art" occurs.

FIG. 5 shows MPEG1, MPEG2 and H.264. 26
3 shows an example of a moving image encoded by an encoding method capable of performing both bidirectional prediction and unidirectional prediction such as No. 3. The image 501 is a frame encoded by intra-frame encoding, and is called an I frame. On the other hand, the images 503, 505, 507, and 509 are called P frames, and are encoded by unidirectional inter-frame encoding using the immediately preceding I or P frame as a reference image. Therefore, for example, when encoding the image 505, the image 503
Is performed as a reference image. Image 50
2, 504, 506, and 508 are called B frames, and bidirectional inter-frame prediction using the immediately preceding and succeeding I or P frames is performed. The B frame also has a feature that another frame is not used as a reference image when performing inter-frame prediction.

First, since motion compensation is not performed in an I frame, a rounding error caused by motion compensation does not occur. On the other hand, in the P frame, motion compensation is performed, and the P frame is used as a reference image of another P or B frame, which causes accumulation of a rounding error. On the other hand, although the effect of accumulation of rounding errors appears in the B frame because motion compensation is performed, it does not cause accumulation of rounding errors because it is not used as a reference image. For this reason, by preventing accumulation of rounding errors in the P frame, it is possible to mitigate the adverse effects of the rounding errors in the entire moving image. In addition, H. In H.263, there is a frame called a PB frame that collectively encodes a P frame and a B frame (for example, frames 503 and 5).
04 can be collectively encoded as a PB frame), and the same discussion can be applied if the combined two frames are considered as separate objects. In other words, if measures are taken against rounding errors in a portion corresponding to the P frame in the PB frame, accumulation of errors can be prevented.

The rounding error is set to 0 when the value obtained by adding 0.5 to an integer value as a result of normal division (division in which the operation result is a real number) is obtained when the intensity value is interpolated. This is caused by rounding up in the direction away from. For example, when an operation of dividing by 4 is performed to obtain an interpolated intensity value, if the value is too small and if the value is 3, the absolute values of the generated errors are equal and the signs are opposite. Work to cancel each other out when calculating the expected value of (the more general case, when divided by a positive integer d ', the case where too much t and the case where d'-t cancels out) . However, if the number is too large, that is, if the result of the ordinary division is a value obtained by adding 0.5 to an integer, this cannot be canceled, leading to accumulation of errors.

Therefore, as a result of the ordinary division, when a value obtained by adding 0.5 to an integer comes out, both a rounding method for rounding up and a rounding method for rounding down can be selected, and these can be combined well. It is considered that the generated error is canceled. Hereinafter, a method of rounding the result of normal division to the nearest integer and rounding up the value obtained by adding 0.5 to an integer away from 0 is referred to as “plus rounding”. A rounding method of rounding the result of normal division to the nearest integer and rounding off a value obtained by adding 0.5 to an integer close to 0 is referred to as “minus rounding”. Equation 3 shows the processing in the case where plus rounding is performed in block matching with half-pixel accuracy. However, in the case where negative rounding is performed, this can be rewritten as follows.

[0033]

(Equation 4)

Now, motion compensation for performing plus rounding at the time of interpolation of intensity values in synthesis of a predicted image is performed using motion compensation using plus rounding, and motion compensation performing minus rounding is performed using minus rounding. Compensation. Also, a P frame to which block matching with half-pixel accuracy is performed and to which motion compensation using plus rounding is applied is called a P + frame, and a P frame to which motion compensation using minus rounding is applied is called a P-frame. (In this case, all H.263 P frames are P + frames). The expected value of the rounding error in the P-frame has the same absolute value as that of the P + frame, and has the opposite sign. Therefore, if the P + frame and the P- frame alternately appear on the time axis, accumulation of rounding errors can be prevented.

In the example of FIG. 5, this processing can be realized if the frames 503 and 507 are P + frames and the frames 505 and 509 are P-frames. Also, P
+ Frame and P- frame alternately occur in B
This means that when performing bidirectional prediction in a frame, one P + frame and one P-frame are used as reference images one by one. Generally, in a B frame, a predicted image in the forward direction (for example, a predicted image synthesized using the frame 503 as a reference image when encoding the frame 504 in FIG. 5) and a predicted image in the reverse direction (for example, the frame in FIG. 5) 5
When encoding 04, the average of the predicted image synthesized using the frame 505 as a reference image can often be used as the predicted image. Therefore, here the P + frame and P
-Averaging images synthesized from frames is effective in canceling the effect of errors.

As described above, the rounding process in the B frame does not cause accumulation of errors. Therefore, no problem occurs even if the same rounding method is applied to all B frames. For example, even if all of the B frames 502, 504, 506, and 508 in FIG. 5 perform motion compensation based on positive rounding, this does not cause any particular deterioration in image quality. In order to simplify the decoding process of the B frame, it is desirable to use only one type of rounding method for the B frame.

FIG. 16 shows a block matching section 160 of an image encoder corresponding to the above-described plurality of rounding methods.
An example of 0 is shown. Like numbers in other figures refer to like parts. By replacing the block matching unit 116 in FIG. 1 with 1600, a plurality of rounding methods can be supported. In the motion estimator 1601, motion estimation processing is performed between the input image 101 and the decoded image 112 of the previous frame. As a result, the motion information 120 is output. This motion information is used when the predicted image synthesizer 1603 synthesizes the predicted image.

The rounding method determiner 1602 determines whether the rounding method used in the frame currently being encoded is positive rounding or negative rounding. Information 1604 about the determined rounding method is
It is input to the predicted image synthesizer 1603. The predicted image synthesizer synthesizes and outputs the predicted image 117 based on the rounding method specified by 1604. The block matching unit 116 in FIG.
There is no portion corresponding to 2, 1604, and the predicted image is synthesized only by positive rounding. Also, the rounding method 1605 determined from the block matching unit is output,
This information may be further multiplexed and incorporated into a transmission bit stream for transmission.

FIG. 17 shows an example of a predicted image synthesizing unit 1700 of an image decoder corresponding to a plurality of rounding methods. Like numbers in other figures refer to like parts. By replacing the predicted image synthesis unit 211 of FIG.
It is possible to support a plurality of rounding methods. The rounding method determiner 1701 determines a rounding method to be applied to a predicted image synthesis process when decoding.

In order to perform correct decoding, the rounding method determined here must be the same as the rounding method applied at the time of encoding. For example,
Odd-numbered P counting from the last encoded I-frame
In principle, positive rounding is applied to frames, and negative rounding is applied to even-numbered P frames. A rounding method determiner on the encoding side (for example, 160 in FIG. 16)
If both 2) and the rounding method determiner 1701 on the decoding side follow this principle, correct decoding can be performed. The predicted image synthesizer 1703 synthesizes a predicted image from the information 1702 on the rounding method determined in this way, the decoded image 210 of the previous frame, and the motion information 202. The predicted image 212 is output and used for synthesizing the decoded image.

It should be noted that a case where information about the rounding method is incorporated in the bit stream (a case where information about the rounding method 1605 is output by the encoder in FIG. 16) can be considered. In this case, the rounding method determiner 1701 is not used, and information 1704 on the rounding method extracted from the coded bit stream is used.
Is input to the prediction image synthesizer 1703.

According to the present invention, in addition to the conventional image coding apparatus and image decoding apparatus using the dedicated circuits and dedicated chips shown in FIGS. 1 and 2, a software image coding apparatus using a general-purpose processor and software The present invention can also be applied to an image decoding device. 6 and 7 show the software image encoding device 600 and the software image decoding device 700.
Here is an example. In the software encoder 600, first, the input image 601 is stored in the input frame memory 602,
The general-purpose processor 603 reads the information from here and performs an encoding process. A program for driving the general-purpose processor is read from a storage device 608 such as a hard disk or a floppy disk and stored in the program memory 604. In addition, the general-purpose processor performs the encoding process using the processing memory 605. The encoded information output from the general-purpose processor is temporarily stored in an output buffer 606 and then encoded bit stream 607.
Is output as

FIG. 8 shows an example of a flowchart of the encoding software (computer-readable recording medium) operating on the software encoder shown in FIG. First, the process starts at 801, and 0 is substituted for a variable N at 802. Subsequently, when the value of N is 100 in 803 and 804, 0 is substituted. N is a frame number counter, which is incremented by one every time processing of one frame is completed, and is allowed to take a value of 0 to 99 when encoding is performed. When the value of N is 0, the frame being encoded is an I frame, when it is odd, it is a P + frame, and when it is an even number other than 0, it is a P-frame. The upper limit of the value of N being 99 means that one I frame is encoded after 99 P frames (P + or P− frame) are encoded.

As described above, by always including one I frame in several frames, (a) due to a mismatch between the processing of the encoder and the decoder (for example, a mismatch of the calculation result of the DCT) It is possible to obtain effects such as preventing accumulation of errors and (b) reducing the amount of processing (random access) for obtaining a reproduced image of an arbitrary frame from encoded data. The optimal value of N varies depending on the performance of the encoder and the environment in which the encoder is used. In this example, 100
, But this does not mean that the value of N must be 100.

The processing for determining the encoding mode and the rounding method for each frame is performed in 805, and FIG. 9 shows an example of a flowchart showing the details of the processing. First, 90
At 1, it is determined whether or not N is 0. If it is 0, 'I' is output to the output buffer as identification information of the prediction mode at 902, and the frame on which the encoding process is to be performed is an I frame. . Here, “output to the output buffer” means that the data is stored in the output buffer and then output from the encoding device to the outside as a part of the encoded bit stream. If N is not 0,
At 903, 'P' is output as prediction mode identification information. If N is not 0, it is further determined 904 whether N is odd or even. 90 if N is odd
In step 5, "+" is output as identification information of the rounding method,
The frame to be subjected to the encoding process is a P + frame. On the other hand, if N is an even number, “−” is output as identification information of the rounding method in 906, and the frame on which the encoding process is to be performed is a P-frame.

Returning to FIG. After determining the encoding mode at 805, the input image is stored in the frame memory A at 806. Note that the frame memory A described here is:
The memory area of the software encoder (for example, 6 in FIG. 6)
05 is secured in the memory 05). At 807, it is determined whether the frame currently being encoded is an I frame. If it is not an I frame, a motion estimation / motion compensation process is performed at 808.

FIG. 10 shows an example of a flowchart showing details of the processing in 808. First, at 1001, frame memories A and B (as written at the end of this paragraph,
A motion estimation process is performed for each block between the images stored in the frame memory B (the decoded image of the previous frame is stored), and a motion vector of each block is obtained. Is output to Subsequently, at 1002, it is determined whether or not the current frame is a P + frame. When the current frame is a P + frame, a predicted image is synthesized by using positive rounding at 1003, and the predicted image is stored in the frame memory C. On the other hand, if the current frame is a P-frame, a predicted image is synthesized using negative rounding at 1004, and the predicted image is stored in the frame memory C. In step 1005, a difference image between the frame memories A and C is obtained, and the difference image is stored in the frame memory A.

Here, returning to FIG. Immediately before the start of the processing in 809, the frame memory A stores an input image when the current frame is an I frame and an input image when the current frame is a P frame (P + or P− frame). A difference image of the image is stored. 8
At 09, DCT is applied to the image stored in the frame memory A, and the DCT coefficient calculated here is quantized and output to the output buffer. Then, at 810, the quantized DCT coefficients are inversely quantized, inverse DCT is applied, and the resulting image is stored in frame memory B. Subsequently, in step 811, it is determined again whether or not the current frame is an I frame. If the current frame is not an I frame, the images in the frame memories B and C are added in step 812, and the result is stored in the frame memory B. . Here, the encoding process for one frame is completed.

The image stored in the frame memory B immediately before the processing of step 813 is performed is a reproduced image (same as that obtained on the decoding side) of the frame whose encoding processing has just been completed. At 813, it is determined whether or not the frame for which encoding has been completed is the last frame. If it is the last frame, the encoding process ends. If it is not the last frame, 1 is added to N in 814, and the process returns to 803 again to start the encoding process of the next frame.

FIG. 7 shows an example of the software decoder 700. The input coded bit stream 701 is temporarily stored in an input buffer 702 and then stored in a general-purpose processor 7.
03 is read. The general-purpose processor performs a decoding process by utilizing a program memory 704 for storing a program read from a storage device 708 such as a hard disk or a floppy disk, and a processing memory 705. The decoded image obtained as a result is temporarily stored in an output frame memory 706 and then output as an output image 707.

FIG. 1 shows an example of a flowchart of the decoding software operating on the software decoder shown in FIG.
It is shown in FIG. The process starts at 1101 and first determines at 1102 whether or not there is input information. If there is no input information, the decoding process ends at 1103. If there is input information, first, at 1104, encoded identification information is input. Note that “input” means reading information stored in an input buffer (for example, 702 in FIG. 7). In 1105, it is determined whether or not the read encoding mode identification information is “I”.
If it is not "I", the identification information of the rounding method is input in 1106, and then the motion compensation processing is performed in 1107.

FIG. 12 shows an example of a flowchart showing details of the processing performed in step 1107. First, 1201
Inputs the motion vector information for each block. Then, it is determined in step 1202 whether or not the identification information of the rounding method read in 1106 is “+”. This is'
If it is + ', the frame currently being decoded is a P + frame. At this time, a predicted image is synthesized at 1203 by positive rounding, and the predicted image is stored in the frame memory D
Is stored in

The frame memory D described here is a memory area of the software decoder (for example, FIG.
(This memory area is secured in the memory 705). On the other hand, if the identification information of the rounding method is not '+', the frame currently being decoded is a P-frame, and a predicted image is synthesized by negative rounding at 1204, and the predicted image is stored in the frame memory D. Is stored. At this time, if some error occurs, P +
The frame is decoded as a P-frame or vice versa
If the − frame is decoded as a P + frame, a predicted image different from the one intended by the encoder will be synthesized in the decoder, and the image quality will be degraded without correct decoding.

Returning now to FIG. In step 1108, a quantized DCT coefficient is input, and an image obtained by applying inverse quantization and inverse DCT thereto is stored in the frame memory E.
At 1109, it is determined again whether the frame currently being decoded is an I frame. If it is not an I frame, the images stored in the frame memories D and E are added at 1110, and the resulting image is stored in the frame memory E. The image stored in the frame memory E immediately before performing the processing of 1111 is a reproduced image.
At 1111, the image stored in the frame memory E is output to an output frame memory (for example, 706 in FIG. 7), and output as it is from the decoder as an output image. Thus, the decoding process for one frame is completed, and the process returns to 1102 again.

When the software image encoder and the software image decoder shown in FIGS. 6 and 7 are caused to execute the program based on the flowcharts shown in FIGS. 8 to 12, when a device using a dedicated circuit and a dedicated chip is used. The same effect as described above can be obtained.

The software encoder 601 shown in FIG.
Storage media recording a bit stream generated by performing the processing shown in the flowcharts of 10 to 10
FIG. 13 shows an example of (recording medium). Digital information is recorded concentrically on a recording disk (eg, magnetic, optical disk, etc.) 1301 on which digital information can be recorded. When a part 1302 of the digital information recorded on this disc is taken out, the encoding mode identification information 1303, 1305, 1 of the encoded frame is obtained.
308, 1311, 1314, identification information 1306, 1309, 1312, 1315 of the rounding method, information 1304, 1307, 131 such as motion vectors and DCT coefficients
0, 1313, and 1316 are recorded. 8 to 10
According to the method shown in FIG.
5, 1308, 1311, 1314 have 'P', 130
6, 1312 is' + ', 1309, 1315 is'
Information meaning-'will be recorded. In this case, for example, 'I' and '+' are 1-bit 0, 'P' and '
If-'is represented by 1 of one bit, the decoder can correctly interpret the recorded information and obtain a reproduced image.
By storing the encoded bit stream in the storage medium in this manner, it is possible to prevent the occurrence of accumulation of rounding errors when the bit stream is read and decoded.

FIG. 15 shows an example of a storage medium on which an encoded bit stream relating to an image sequence including the P + frame, P-frame and B frame shown in FIG. 5 is recorded. As with 1301 in FIG. 13, a recording disk (eg, magnetic, optical disk, etc.) 1501 on which digital information can be recorded has digital information recorded concentrically. When a part 1502 of the digital information recorded on this disc is taken out, the encoding mode identification information 1503, 1505, 15 of the encoded frame is obtained.
08, 1510, 1513, identification information 1 of rounding method
Information 506, 1512, and information 1504, 1507, 1509, 1511, and 1514 such as motion vectors and DCT coefficients are recorded.

At this time, “I”, 150
'P' for 5 and 1510, and '1' for 1508 and 1513
B 'and 1505 record information meaning' + 'and 1511 record information meaning'-'. For example, 'I', 'P', '
B ′ is 2 bits 00, 01, 10, and “+”
If "-" and "-" are represented by 1-bit 0 and 1, respectively, the decoder can correctly interpret the recorded information and obtain a reproduced image.

At this time, information on 501 (I frame) in FIG. 5 is 1503, 1504, and 502 (B frame).
Information about 1508 and 1509, frame 503
(P + frame) information is 1505-1507,
Information about frame 504 (B frame) is 1513
1515 and information about the frame 505 (P-frame) are 1510 to 1512. As described above, in the case of encoding a moving image including a B frame, the order in which information on frames is generally transmitted is different from the order in which information is reproduced.
This is because before decoding a certain B frame, it is necessary to decode the reference images before and after the B frame is used for synthesizing the predicted image. Therefore, although the frame 502 is reproduced before the frame 503, the information on the frame 503 used by the frame 502 as the reference image is transmitted before the information on the frame 502.

As described above, since the B frame is not a person causing accumulation of rounding errors, it is not necessary to apply a plurality of rounding methods as in the case of the P frame. For this reason, in the example shown here, information such as '+' or '-' for specifying the rounding method is not transmitted for the B frame. By doing so, for example, even if only positive rounding is always applied to the B frame, the problem of accumulation of errors does not occur. In this way, by storing the encoded bit stream including the information on the B frame in the storage medium, it is possible to prevent the occurrence of accumulation of rounding errors when the bit stream is read and decoded.

FIG. 14 shows an encoding method based on an encoding method in which a P + frame and a P-frame are mixed as shown in this specification.
A specific example of the decoding device will be described. By incorporating image encoding / decoding software into the personal computer 1401,
It can be used as an image encoding / decoding device. This software is recorded on a storage medium (CD-ROM, floppy disk, hard disk, etc.) 1412 which is a computer-readable recording medium, and is read and used by a personal computer.
Further, by connecting this personal computer to some kind of communication line, it can be used as a video communication terminal.

The decoding method described in this specification can be implemented in a reproducing apparatus 1403 which reads and decodes an encoded bit stream recorded on a storage medium 1402 which is a recording medium. In this case, the reproduced video signal is displayed on the television monitor 1404. Also, 140
The device No. 3 only reads the coded bit stream, and a case where a decoding device is incorporated in the television monitor 1404 is also conceivable.

Recently, digital broadcasting using satellites and terrestrial waves has become a hot topic, but a decoding device can be incorporated in a television receiver 1405 for digital broadcasting.

The cable 140 for cable television
8 or a set-top box 1409 connected to a satellite / terrestrial broadcasting antenna, and a decoding device is mounted on the TV monitor 1410 to reproduce the decoding device. At this time, as in the case of 1404, the encoding device may be incorporated in the television monitor instead of the set-top box.

Reference numerals 1413, 1414, and 1415 show configuration examples of the digital satellite broadcasting system. In the broadcast station 1413, the encoded bit stream of the video information is transmitted to a communication or broadcast satellite 1414 via radio waves. The satellite receiving the signal transmits a radio wave for broadcasting, and the radio wave is received by the home 1415 having the satellite broadcasting receiving equipment.
The encoded bit stream is decoded by a device such as a television receiver or a set-top box and reproduced.

With the possibility of encoding at a low transmission rate, digital video communication by the digital portable terminal 1406 has recently attracted attention.
In the case of a digital portable terminal, in addition to a transmission / reception type terminal having both an encoder and a decoder, three types of mounting formats are considered: a transmission terminal having only an encoder and a reception terminal having only a decoder.

It is also possible to incorporate an encoding device in a camera 1407 for capturing a moving image. In this case, the photographing camera has an encoding device and a recording device for recording the output from the encoding device on a recording medium, and records the encoded bit stream output from the encoding device on the recording medium. Further, the camera only captures the video signal, and a configuration in which the video signal is incorporated in the dedicated encoding device 1411 is also conceivable.

For any of the devices and systems shown in this figure, by implementing the method described in this specification, it is possible to handle image information with higher image quality as compared with the case where the conventional technology is utilized. Becomes possible.

It is apparent that the following modifications are also included in the present invention.

(1) In the above discussion, it was assumed that block matching was used as a motion compensation method. However, according to the present invention, the horizontal and vertical components of the motion vector can take values other than an integral multiple of the sampling interval of the pixels in the horizontal and vertical directions, and the intensity value at a position where no sample value exists is obtained by linear interpolation. The present invention can be applied to all image coding systems and image decoding systems that employ the motion compensation system. For example, Japanese Patent Application No. 08-06057
Global motion compensation described in 2 and Japanese Patent Application No. 08-24
The present invention is also applicable to the warping prediction described in 9601.

(2) In the above discussion, only the case where the horizontal and vertical components of the motion vector take an integral multiple of 1/2 has been discussed. However, generalizing the discussion,
The present invention is applicable to a system in which the horizontal and vertical components of the motion vector take an integral multiple of 1 / d (d is a positive integer and an even number). However, when d becomes large,
Since the divisor (the square of d, see Equation 2) in the division of the next interpolation becomes large, the probability that the result of the ordinary division becomes a value obtained by adding 0.5 to an integer becomes relatively low. Therefore, the absolute value of the expected value of the rounding error calculation when only the positive rounding is performed becomes small, and the adverse effect due to the accumulation of errors becomes less noticeable. Therefore, for example, in a motion compensation method in which the value of d is variable, if d is smaller than a certain value, both positive rounding and negative rounding are used, and if d is equal to or more than the above fixed value, It is also effective to use only plus or minus rounding.

(3) As described in the background art, when DCT is used as an error encoding method, an adverse effect due to accumulation of rounding errors tends to appear when the quantization step size of DCT coefficients is large. Therefore, when the quantization step size of the DCT coefficient is larger than a certain value, both the positive rounding and the negative rounding are used, and the DCT coefficient is used.
When the quantization step size of the T coefficient is equal to or smaller than the above-mentioned fixed value, a method of using only plus or minus rounding is also effective.

(4) In the case where the accumulation of the rounding error occurs in the luminance plane and the case where the accumulation of the rounding error occurs in the chrominance plane, in general, the effect on the reproduced image is more serious when it occurs in the chrominance plane. It is. this is,
This is because, when the color of the image changes overall, the image is more noticeable than when the image becomes slightly brighter or darker as a whole. Therefore, it is also effective to use both the positive rounding and the negative rounding for the color difference signal, and use only the positive or negative rounding for the luminance signal.

Further, in the conventional technique, H. 1 in 263
Although the method for rounding a motion vector with a precision of / 4 pixel to a motion vector with a precision of 1/2 pixel has been described, it is possible to reduce the absolute value of the expected value of the rounding error by slightly modifying this method. It is. The H.264 described in the prior art. In H.263, the value obtained by halving the horizontal or vertical component of the motion vector of the luminance plane is r + s /
Assuming that 4 (r is an integer and s is an integer of 0 or more and less than 4), when s is 1 or 3, an operation of rounding this to 2 is performed. This may be changed so that when s is 1, this is set to 0, and when s is 3, 1 is added to r to perform rounding to make s 0. By doing so, the number of times of calculating the intensity values at the positions 406 to 408 in FIG. 4 relatively decreases (the probability that the horizontal and vertical components of the motion vector become integers increases). Becomes smaller. However, although this method can suppress the magnitude of the generated error, it cannot prevent the accumulation of the error.

(5) There is a method in which the average of the inter-frame predicted images by two types of motion compensation methods is used as the final inter-frame predicted image for the P frame. For example, Japanese Patent Application 8-36
In block 16, block matching assigns one motion vector to a block of 16 pixels in length and width, and block matching divides a block of 16 pixels in length and width into 4 blocks of 8 pixels in width and assigns a motion vector to each block. A method is described in which two types of inter-frame predicted images obtained by the two types of methods are prepared, and an average of the intensity values of these inter-frame predicted images is obtained as a final inter-frame predicted image. . In this method 2
Rounding is also performed when calculating the average value of the types of images. Continuing to perform only positive rounding in this averaging operation creates a new cause of accumulation of rounding errors. In this method, a negative rounding is performed in the averaging operation for a P + frame that performs a positive rounding in block matching, and a positive rounding is performed in the averaging operation for a P− frame. For example, the effect of canceling out the rounding error due to block matching and the rounding error due to averaging within the same frame is obtained. (6) When a method of alternately arranging P + frames and P− frames is used, an encoding device and a decoding device are used. In order to determine whether the currently encoded P frame is a P + frame, which is a P + frame, for example, the following processing may be performed. P currently being encoded or decoded
The number of the P-frame after the most recently encoded or decoded I-frame is counted, and if this is an odd-numbered P-frame, and if it is an even-numbered P-frame, it may be a P-frame. This is called an implicit method). Also, there is a method of writing information identifying whether the P frame currently encoded by the encoding device is a P + frame or a P- frame, for example, in a header portion of frame information (this is an explicit method). ).
This method is more resistant to transmission errors.

The method of writing the information for identifying the P + frame and the P-frame in the header of the frame information has the following advantages. As described in “Prior Art”, past encoding standards (eg, MPEG-1 and MPEG-1)
In -2), only positive rounding is performed in the P frame. Thus, for example, M
A motion estimation / motion compensation device for PEG-1 / 2 (for example,
The portion corresponding to 106 in FIG. 1) is a P + frame and a P− frame.
It is impossible to cope with encoding in which frames are mixed. Now, it is assumed that there is a decoder that supports encoding in which P + frames and P− frames coexist. In this case, if the decoder is based on the above implicit method,
It is difficult to use a motion estimation / motion compensation device for MPEG-1 / 2 to create an encoder that generates a bit stream that can be correctly decoded by a decoder based on this implicit method.

However, if the decoder is based on the explicit method described above, this problem can be solved. An encoder using a motion estimation / motion compensation device for MPEG-1 / 2 only needs to continuously send a P + frame and write identification information indicating this in a header of the frame information. In this way, the decoder based on the explicit method can correctly reproduce the bit stream generated by the encoder.

Of course, in this case, since only the P + frame exists, accumulation of rounding errors is likely to occur. However, if this encoder uses only a small value as the quantization step size of the DCT coefficient (an encoder dedicated to high-rate encoding), accumulation of errors does not become a major problem.

In addition to the problem of compatibility with the past methods, the explicit method further includes (a) an encoder dedicated to high-rate encoding and a rounding error caused by frequently inserting I frames. An encoder that is unlikely to generate only has to implement either the positive or negative rounding method, and can reduce the cost of the device. (B) An encoder that does not easily generate the rounding error is P + Alternatively, since only one of the P-frames needs to be continuously transmitted, it is not necessary to determine whether the frame currently being encoded is a P + frame or a P-frame, thereby simplifying the processing. There are advantages such as.

(7) The present invention can also be applied to a case where filtering involving rounding processing is performed on an inter-frame prediction image. For example, H.264, which is an international standard for video coding, is used. In 261, a low-pass filter (referred to as a loop filter) is applied to a signal in a block whose motion vector is not 0 in the inter-frame prediction image. H. In 263, it is possible to use a filter for smoothing discontinuities (so-called block distortion) occurring at the boundary between blocks. In these filters, a weighted averaging process is performed on the pixel intensity values, and an operation of rounding to an integer is performed on the filtered intensity values. Also in this case, it is possible to prevent accumulation of errors by using positive rounding and negative rounding properly.

(8) In addition to IP + P−P + P−.
+ P + P-P-P + P + ... or IP + P-P-P + P +
Various methods are conceivable for mixing P + frames and P- frames. For example, a random number generator that generates 0 and 1 with a probability of 1/2, respectively, may be used. In any case, generally, the more the P + and P- frames are mixed and the difference between the respective existence probabilities within a certain period of time is smaller, the less the accumulation of rounding errors occurs. When the encoder is allowed to mix arbitrary P + frames and P-frames, the encoder and the decoder are not based on the implicit method shown in (6) but are explicitly described. Must be based on strategic methods. Thus, the explicit method is advantageous in terms of allowing more flexible implementations for the encoder and decoder.

(9) The present invention does not limit the method of obtaining the intensity value at a point where no pixel exists to bilinear interpolation. When the interpolation method of the intensity value is generalized, it can be expressed as the following equation.

[0083]

(Equation 5)

Here, r and s are real numbers, h (r, s) is a function of a real number for interpolation, T (z) is a function of rounding a real number z to an integer, and R (x, y), The definitions of x and y are the same as Equation 4. If T (z) is a function representing plus rounding, motion compensation using plus rounding is performed, and if T (z) is a function representing minus rounding, motion compensation using minus rounding is performed. The present invention can be applied to an interpolation method that can be expressed in the form of Expression 5. For example, h (r, s)

[0085]

(Equation 6)

If the definition is made as follows, bilinear interpolation is performed. However, for example, h (r, s)

[0087]

(Equation 7)

With the definition as described above, an interpolation method different from the bilinear interpolation is performed, but in this case, the present invention can be applied.

(10) The present invention does not limit the error image coding method to DCT. For example, instead of DCT, wavelet transform (for example, M. Antonioni, e.
t. al, "Image Coding Using Wavelet Transform", IEEE
Trans. Image Processing, vol. 1, no.2, April 199
2) and Walsh-Hadamard transformation
Transform) (eg, AN Netravalli and BG Ha
skell, "Digital Pictures", Plenum Press, 1998).

[0090]

According to the present invention, accumulation of rounding errors in an inter-frame prediction image can be suppressed, and the image quality of a reproduced image can be improved.

[Brief description of the drawings]

FIG. FIG. 263 is a diagram illustrating a configuration example of an image encoder of H.263.

FIG. FIG. 263 is a diagram illustrating a configuration example of an image decoder of H.263.

FIG. FIG. 263 is a diagram illustrating a configuration of a macroblock in H.263.

FIG. 4 is a diagram showing a state of an interpolation process of a luminance value in block matching of half-pixel components.

FIG. 5 is a diagram showing a state of an encoded image sequence.

FIG. 6 is a diagram illustrating a configuration example of a software image encoding device.

FIG. 7 is a diagram illustrating a configuration example of a software image decoding device.

FIG. 8 is a diagram illustrating an example of a flowchart of a process in the software image encoding device.

FIG. 9 is a diagram illustrating an example of a flowchart of a coding mode determination process in the software image coding device.

FIG. 10 is a diagram illustrating an example of a flowchart of a motion estimation / motion compensation process in the software image encoding device.

FIG. 11 is a diagram illustrating an example of a flowchart of a process in the software image decoding device.

FIG. 12 is a diagram illustrating an example of a flowchart of a motion compensation process in the software image decoding device.

FIG. 13 is a diagram illustrating an example of a storage medium in which a bit stream encoded by an encoding method that mixes an I frame, a P + frame, and a P− frame is recorded.

FIG. 14 is a diagram illustrating a specific example of an apparatus that uses an encoding method that mixes P + frames and P− frames.

FIG. 15 shows an I frame, a B frame, a P + frame and a P frame.
FIG. 2 is a diagram illustrating an example of a storage medium that records a bit stream encoded by an encoding method that mixes frames.

FIG. 16 is a diagram illustrating an example of a block matching unit included in an apparatus that uses an encoding method that mixes P + frames and P− frames.

FIG. 17 is a diagram illustrating an example of a predicted image combining unit included in a device that decodes a bit stream encoded by an encoding method that mixes P + frames and P− frames.

[Explanation of symbols]

100: image encoder, 101: input image, 102: subtractor, 103: error image, 104: DCT converter, 10
5. DCT coefficient quantizer, 106, 201 ... Quantized DC
T coefficient, 108, 204 ... DCT coefficient inverse quantizer, 10
9, 205: inverse DCT converter, 110, 206: decoding error image, 111, 207: adder, 112: decoded image of current frame, 113, 215: output image of inter-frame / intra-frame coding changeover switch, 114 , 209 ...
Frame memory, 115, 210... Decoded image of previous frame, 116, 1600... Block matching unit, 11
7, 212 ... predicted image of current frame, 118, 213 ...
"0" signal, 119, 214 ... inter-frame / intra-frame coding changeover switch, 120, 202 ... motion vector information, 121, 203 ... inter-frame / intra-frame identification flag, 122 ... multiplexer, 123 ... transmission bit stream , 200 ... image decoder, 208 ... output image, 21
1, 1700: predicted image synthesizing unit, 216: separator, 30
1 ... Y block, 302 ... U block, 303 ... V block, 401-404 ... pixels, 405-408 ... positions for which intensity values are to be obtained by primary interpolation, 501 ... I frames,
503, 505, 507, 509 ... P frame, 50
2, 504, 506, 508: B frame, 600: software image encoder, 602: frame memory for input image, 603, 703: general-purpose processor, 604, 7
04: Program memory, 605, 705: Processing memory, 606: Output buffer, 607, 701: Encoded bit stream, 608, 708: Storage device, 7
00: software image decoder, 702: input buffer, 706: frame memory for output image. 801-81
5, 901 to 906, 1001 to 1005, 1101 to
1111, 1201 to 1204... Processing items in the flowchart, 1301, 1402, 1501.
1302, 1502... Tracks on which digital information is recorded, 1303 to 1316, 1503 to 1514... Digital information, 1401... Personal computer, 1403... Storage media playback device, 1404, 1410.
405: TV broadcast receiver, 1406: Wireless portable terminal,
1407: TV camera, 1408: Cable for cable TV, 1409: Set-top box, 141
DESCRIPTION OF SYMBOLS 1 ... Image coding apparatus, 1412 ... Storage media which recorded software information, 1413 ... Broadcasting station, 1414 ... Communication or broadcast satellite, 1415 ... Household with satellite broadcasting receiving equipment, 1601 ... Motion estimator, 1602, 1701 ... Rounding Method determiner, 1604, 1605, 1702, 1
704: Information on rounding method, 1603, 170
3. Predictive image synthesizer.

Continuation of the front page (56) References JP-A-6-214754 (JP, A) Yuichiro Nakaya, 3 others, "D-11-44 Prevention of Accumulation of Rounding Error in Motion Compensation with Half Pixel Accuracy", 1998 Proceedings of the IEICE General Conference, Information and System 2, March 1998, p. 44 ITU-T Recommendation H. 263, February 1998 (58) Fields investigated (Int. Cl. ⁷ , DB name) H04N ^7/ 24-7/68 H04N 5/232 H04N 11/04 H03M ^7/ 30-7/50 G06F 7/38

Claims (3)

(57) [Claims] 1. A recording medium in which encoded information of the moving image is recorded as the information for a plurality of frames, information about the frame, the input image of the the predicted image of the current frame obtained by the motion compensation current frame Characterized in that information about a difference between the two is recorded, information about a motion vector obtained by the motion compensation, and information for identifying whether the motion compensation is motion compensation by plus rounding or motion compensation by minus rounding. Recording medium. 2. A recording medium on which information about a moving image is recorded as an encoded bit stream, wherein the moving image is encoded by using motion compensation, and the encoded bit stream is used to calculate pixel intensity values in the motion compensation . rounding method or Ma rounding method used in the interpolation is positive
Identify one of the Inas rounding methods
A recording medium having information . 3. The recording medium according to claim 2, wherein the plus rounding method comprises: a first pixel having an intensity La of an image forming the moving image, and a first pixel adjacent to the first pixel in a horizontal direction. A second pixel having an intensity Lb, a third pixel vertically adjacent to the first pixel and having an intensity Lc, and vertically adjacent to the second pixel, and the third pixel And a fourth pixel horizontally adjacent and having an intensity Ld,
An intensity Ib at an intermediate point between the first pixel and the second pixel where no pixel exists, an intensity Ic at an intermediate point between the first pixel and the third pixel, the first, second, and When calculating the intensity Id at a point surrounded by the third and fourth pixels and equidistant from the first, second, third, and fourth pixels, Ib = [(La + Lb + 1) / 2], Ic = [(La + Lc + 1) / 2], Id = [(La + Lb + Lc +
Ld + 2) / 4], and the negative rounding method is as follows: Ib = [(La + Lb) / 2], Ic = [(La + Lc) / 2], Id = [(La + Lb + Lc + Ld +
1) / 4], which is a rounding method.