JP2000224593A - Method and device for interpolating frame and recording medium recording the method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize compensation predictive coding and decoding by a simple interpolating method and its device, by which the interpolating frame of a picture, which is different from the actual one, is not created by plural subjects with different movements. SOLUTION: In the method, at first, an area dividing part 110 compares the inter- movement compensating frame difference absolute value of a decoding object small block with a threshold at every pixel and the areas are divided into a first area with the pixels being smaller than the threshold and a second one with the larger pixels. An interpolating value arithmetic part 116 averages the pixel value of the first area and a corresponding pixel value in a reference block and constitutes the interpolating frame. The interpolating value arithmetic part 117 judges shielding and appearing in the decoding object frame of the second area, one of the corresponding area of the decoding object block or the reference block is set as the reference area and the frame to which a reference area dose not belong is set as a re-location reference frame. Then the pixel value of the re-search reference frame fixed by a half of a second movement vector which is obtained between the reference area and the re-search reference frame is copied on the second area of the interpolating frame so that the interpolating frame is constituted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention aims to increase the number of display frames per time by efficiently interpolating non-coded frames when reproducing a coded moving picture signal. , Frame interpolation method and apparatus.

[0002]

2. Description of the Related Art In the coding of a moving image, motion-compensated predictive coding utilizing high correlation in the time direction is often used. In motion-compensated predictive coding, the encoding target frame is divided into small blocks, and for each encoding target small block, the most similar block is detected from the already decoded frames, and the difference signal is quantized. To be encoded. At this time, a signal indicating how much the similar block is spatially displaced from the encoding target small block is encoded as a motion vector. A motion vector is obtained for each small block.

In such motion-compensated predictive coding, in order to suppress the amount of generated information within a predetermined bit rate, when the compression ratio becomes high, frames are skipped at certain intervals without coding all frames. A frame dropping control is performed to perform encoding. When the encoded information is reproduced by dropping frames, the motion is not smooth and unnatural because the frames are skipped. For this reason, a method of increasing the number of frames by creating frames skipped during reproduction by interpolation from decoded frames has been considered.

FIG. 4 shows a conventional frame interpolation method using motion compensated predictive coding. Here, it is assumed that the frame at the center in time is thinned out. In motion-compensated predictive coding, since a motion vector from a frame already decoded is sent for each small block to be decoded, a skip frame between the temporally decoded frame and the currently decoded frame is sent. Is interpolated using the average value of the small block to be decoded and the similar block on the decoded frame indicated by the motion vector. At this time, the interpolation pixel position is set to a position indicated by making the motion vector half the size, assuming that the motion of the small block is moving in parallel.

FIG. 5 is a block diagram of an image encoding / decoding apparatus implementing a conventional frame interpolation method.

On the encoding side, a motion vector search section 502 performs a motion search on a reference image 510 in a frame memory 511 for an encoding target block 501 of an image signal, and calculates a motion vector 503. The motion compensation unit 506 obtains a reference small block 505 corresponding to the selected motion vector 503, calculates a difference from the coding target block 501 in the subtraction unit 504, and obtains a motion compensation inter-frame difference value 507. Quantizer 508
Is quantized. The quantized motion-compensation inter-frame difference value 509 is inversely quantized by an inverse quantization unit 513,
The addition unit 515 calculates the sum of the restored motion-compensated inter-frame difference value 514 and the reference small block 505, and stores the sum as the local decoded output 512 in the frame memory 511.

On the decoding side, the quantized motion-compensated inter-frame difference value 509 transmitted from the encoding side is inversely quantized by an inverse quantization unit 516, and a motion-compensated inter-frame difference value 518 is calculated. On the other hand, in the motion compensation unit 521,
Using the transmitted motion vector 517, a reference image block 520 on the reference image 523 stored in the frame memory 524 is obtained, and an addition unit 519 calculates the sum with the motion compensation inter-frame difference value 518 to obtain a decoded image block. 522 are obtained. Further, the average value calculation unit 525 calculates the average value of the pixel values of the decoded image block 522 and the reference image block 520, and the vector scaling unit 527 calculates 1
/ 2 scaled motion vector 528
Is stored in the address on the interpolation image frame memory 526 indicated by, and is output as the interpolation image block 529.

[0008]

In the above prior art,
When a plurality of subjects having different motions are included in a small block, an image different from an actual image is generated on the generated interpolation frame.

For example, consider a case where one of the _two regions R ₁ moves so as to hide the other region R ₂ as shown in FIG. Decoding target block C on the decoding target frame in FIG. 6 (3) is on the boundary of the region R ₁ and R _2, the reference block A on the reference frame defined by a motion vector is determined as FIG. 6 (1) Figure 6
The pixel value in the area B ₂ of the interpolation block B in (2) is the average value of the pixel values in the area A ₂ in the reference block and the area C ₂ in the decoding target block, and is the actual area B ′ in the reference frame. There is a problem that the pixel value is completely different from the pixel value.

As a method for solving such a problem, a frame interpolation method using contour adaptive warping prediction and labeling has been proposed (frame interpolation method based on contour adaptive warping prediction, Yokoyama et al., 10th Image Coding Symposium). PCSJ95, 9-19, pp. 205-20
6).

In the warping prediction, motion vectors are detected at several representative points on the screen, and the motion vectors of pixels other than the representative points are interpolated using the motion vectors of the representative points. This realizes motion compensation inter-frame prediction. In this warping prediction, the position of the representative point is adapted to the contour of the subject, and the representative point to be referred to by interpolation of the motion vector is adaptively selected for each patch so that discontinuous motion at the subject boundary can be handled. This is called contour adaptive warping prediction. When generating an interpolated image, the method of interpolation is temporally extrapolated in the forward direction by comparing the contour adaptive warping prediction described above and the order of the degree of overlap of the subject given as a label to each patch. It is realized by selecting one of three methods: extrapolation in the reverse direction in time and interpolation from both directions. According to this, the problem due to the overlapping of the objects can be solved, and a good interpolated image can be obtained.

However, the above-described conventional frame interpolation method using contour adaptive warping prediction and labeling has a problem in that the contour must be extracted and the degree of overlap of the subject must be determined in advance, which complicates the processing. Was.

According to the present invention, the problem caused by the overlapping of the objects in the prior art, that is, in the compensation prediction encoding / decoding method and apparatus, when a small block includes a plurality of objects having different motions, the object is generated. An object of the present invention is to provide a frame interpolation method and apparatus that solves a problem that an image different from an actual image is generated on an interpolated frame by a simple process and configuration.

[0014]

The present invention employs the following means to solve the above-mentioned problems.

One of the means is to divide a frame constituting a moving image into a frame to be coded and a non-coding frame when coding the digital moving image signal, and to convert the image data of the frame to be coded. For each divided small block to be coded, a first motion vector is obtained from the coded / decoded reference frame stored in the frame memory. Calculating a motion-compensated inter-frame difference value between the determined reference block and the current frame to be coded, and encoding and transmitting / recording the same with the first motion vector;
At the time of decoding, the transmission /
A decoded frame is constructed by adding the reference block in the frame memory determined by the recorded first motion vector and the motion-compensated inter-frame difference value. First
In the image encoding and decoding method of forming an interpolation frame by an interpolation block calculated using the motion vector of the motion vector, when determining the pixel value of the interpolation block in the interpolation frame, the motion compensation frame difference absolute value of the small block to be decoded is determined. Is compared with a predetermined threshold value for each pixel, and a first area composed of pixels whose absolute value of the inter-frame difference is smaller than the threshold value, and a first area composed of pixels whose absolute value of the inter-frame difference is larger than the threshold value Dividing the pixel value of the first region into a pixel value corresponding to the first region in the reference block defined by the first motion vector. Configuring an interpolated frame with a weighted average, and for the second region, the distribution state of the pixels constituting the region and the direction of the first motion vector Calculating a relationship between the reference region and one of the corresponding regions on the decoding target block or the reference block defined by the first motion vector based on the calculated relationship. Making a frame different from the re-search reference frame, performing motion detection between the reference region and the re-search reference frame to obtain a second motion vector, A pixel value on a re-search reference frame determined by multiplying the second motion vector by a real number to form an interpolated frame.

Alternatively, at the time of encoding a digital moving image signal, a frame constituting the moving image is divided into an encoding target frame and a non-coding frame, and the image data of the encoding target frame is divided into small blocks. For the divided small block to be encoded, a first motion vector is obtained from the encoded and decoded reference frame stored in the frame memory, and a reference block determined by the first motion vector is obtained. And a motion-compensated inter-frame difference value between the current frame and the current block to be coded, and coded and transmitted / recorded together with the first motion vector. Adding the reference block in the frame memory determined by the recorded first motion vector and the motion-compensated inter-frame difference value. In an image encoding and decoding apparatus for forming a decoded frame and further forming an uncoded frame as an interpolated frame by an interpolation block calculated using the value of the decoded frame and the first motion vector, When determining the pixel value of the interpolation block, the motion compensation inter-frame difference absolute value of the small block to be decoded is compared for each pixel with a predetermined threshold value, and the inter-frame difference absolute value is composed of pixels smaller than the threshold value. Region dividing means for dividing the first region to be divided into a second region composed of pixels whose absolute value of the inter-frame difference is larger than a threshold, and for the first region, the pixel value of the region Interpolated values forming an interpolated frame by weighted average values with pixel values corresponding to a first area in a reference block defined by the first motion vector Calculation means, for the second area, occlusion / appearance determination means for calculating the relationship between the distribution state of the pixels constituting the area and the direction of the first motion vector, and decoding based on the calculated relationship. A reference region extracting means for setting one of the corresponding region on the target block or the reference block defined by the first motion vector as a reference region and a frame different from the frame to which the reference region belongs as a re-search reference frame A second motion vector extracting means for performing motion detection between the reference region and the re-search reference frame to obtain a second motion vector; and a second region in the interpolation block, Region extracting means for forming an interpolation frame by copying a pixel value on the re-search reference frame determined by multiplying the motion vector by a real number. This is a frame interpolation method.

Alternatively, the present invention is a recording medium recording a frame interpolation method, wherein a program for causing a computer to execute the steps in the above-described frame interpolation method is recorded on a computer-readable recording medium.

According to the present invention, the interpolation mode can be set more finely than the small block unit, so that even when occlusion occurs and objects overlap, frame interpolation can be performed with high accuracy. I do.

[0019]

Embodiments of the present invention will be described below in detail with reference to the drawings. In the following description, it is assumed that the interpolation frame is temporally located at the center of the two decoding target frames.

First, an embodiment of the frame interpolation method according to the present invention will be described.

In this method, when determining the pixel value of the interpolation block in the interpolation frame, the motion compensation frame difference absolute value of the small block to be decoded is compared with a predetermined threshold value for each pixel, and the frame difference absolute value is determined. It is divided into a first area composed of pixels whose values are smaller than the threshold value and a second area composed of pixels whose inter-frame difference absolute values are larger than the threshold value.

With respect to the first area, an interpolation frame is formed by an average value of the pixel value of the area and the pixel value corresponding to the first area in the reference block defined by the first motion vector.

On the other hand, for the second region, the relationship between the distribution state of the pixels constituting the region and the direction of the first motion vector is calculated, and the relationship between the distribution and the direction of the first motion vector is calculated.
Determine appearance.

As a result, one of the corresponding area on the decoding target block or the reference block defined by the first motion vector is set as a reference area, and a frame different from the frame to which the reference area belongs is referred to as a re-search reference frame. And

Next, motion detection is performed between the reference area and the re-search reference frame to obtain a second motion vector, and the second motion vector is reduced to a half of the second area on the interpolation frame. The pixel value on the re-search reference frame determined by the scaled vector is copied.

FIG. 1 shows an embodiment of a block configuration for carrying out the method of the present invention, and an embodiment of the method of the present invention will be described in more detail. Note that FIG.
(A) and (b) show the internal block configuration of the first interpolation value calculation unit and the second interpolation value calculation unit in FIG. 1, respectively.

First, the quantized motion-compensated inter-frame difference value 101 transmitted from the encoding side is inversely quantized by an inverse quantization unit 103 to obtain a motion-compensated inter-frame difference value 104.

Further, the motion compensating unit 108 obtains a reference image block 106 on the reference image 109 stored in the frame memory 111 using the first motion vector 102 transmitted from the encoding side. , The sum with the motion compensation inter-frame difference value 104 is calculated to obtain a decoded image block 107.

On the other hand, the following processing is performed on the interpolated image block.

First, the area dividing unit 110 compares the absolute value of the motion compensation inter-frame difference value 104 with a predetermined threshold value for each pixel, and determines the first value whose absolute value is smaller than the threshold value.
Region R ₁ (114) and the large second region R ₂ (115)
And split into

Next, in the first interpolation value calculation section 116, the average calculator 122 extracts the pixel values corresponding to the region R ₁ of the reference image block 106 and the decoded image block 107, the average value thereof Calculate 118. Average value 1
Numeral 18 is stored at an address on the interpolated image frame memory 120 indicated by the first motion vector 113 scaled to half the size by the vector scaling unit 112.

In the second interpolation value calculation section 117, the occlusion / appearance determination section 123 outputs the first motion vector 1
02 The relationship between the distribution region R ₂ of the decoded sub-block (FIG. 1, arrow A in FIG. 2 (b)), determines the shielding-occurrence relationships 124 in the decoding target frame region R ₂ . This relationship 124 switches the switch 125, the reference region extraction section 126 either the motion compensation of the region corresponding to the region R ₂ on the decoding target frame 107 on or reference frame 109 (arrow B in FIG. 1) , And a frame different from the frame to which the reference region belongs is set as the re-search reference frame 132. Table 1 shows combinations of the determination result, the reference area, and the re-search frame.

[0033]

[Table 1]

In accordance with the above combination, in the second interpolation value calculation section 117, the second motion vector extraction section 128 causes the region R ₂ (12
Regarding (7), a motion is detected on the re-search frame 132 to calculate a second motion vector 129.

Subsequently, in the second interpolation value calculation section 117,
In the region extraction unit 133, the vector scaling unit 1
A second motion vector 131 scaled to a size of 、 by 30 and a second motion vector 131 by a vector scaling unit 112.
/ 2, corresponding to the region R ₂ of the interpolated block on the re-search reference frame 132 indicated by the first motion vector 113 (arrow C in FIGS. 1 and 2 (b)). Region to be extracted.

Subsequently, the pixel value 119 of the extracted area
Is stored at the address on the interpolated image frame memory 120 indicated by the first motion vector 113 scaled to half the size by the vector scaling unit 112.

FIG. 3 shows a specific processing example in the above embodiment. This example shows a case where an object moving to the left overlaps a background moving to the right. A (i, j) is a block to be decoded, B (i + x, j + y) is a reference block, and C is a block to be interpolated. (X, y) ≡
v ₁ indicates the first motion vector, and in this example, x =
4, y = 0. At this time, the motion compensation prediction error value is E (i, j) = A (i, j) -B (i
+ X, j + y). Since E (i, j) is transmitted from the encoding side, there is no need to recalculate. E (i,
When the absolute value of j) is subjected to threshold processing at a threshold T = 10, a region R ₁ (a black region on the lower right side of the block) composed of pixel values smaller than the threshold and a region R ₂ (a block composed of large pixel values) (Upper left white area).

First, the pixel corresponding to R ₁ on the interpolation block C is obtained by taking the average value of the area R ₁ of the corresponding blocks A and B. That is, C (i + x /
2, j + y / 2) = (1/2) ｛A (i, j) + B (i
+ X, j + y)}.

On the other hand, for the region R ₂ , the first motion vector is set to v ₁ and the following processing is performed. If a vector u _k (k = 1, 2,..., K) is a vector representation of the K pixel positions forming the region R ₂ with the center of the block as the origin, counter = 0, (for each small block (Initialized with 0) counter ← counter + 1 if (v ₁ ,
u _k )> 0 counter ← counter−1 if (v ₁ ,
u _k ) <0 counter> appears if counter> 0, blocks if counter <0 where (v, u) is the inner product of vectors v, u. In the case of this example, the vector expression of the pixels constituting R ₂ is expressed as u ₁ =
(−2, 1), u ₂ = (− 1, 1), u ₃ = (− 2,
0), u ₄ = (− 1,0) and u ₅ = (− 2, −1), so that (v ₁ , u ₁ ) = (4,0) · (−2,1) = − 8 <0, (v ₁ , u ₂ ) = (4,0) · (−1,1) = − 4 <0, (v ₁ , u ₃ ) = (4,0) · (−2,0) = −8 <0, (v ₁ , u ₄ ) = (4,0) · (−1,0) = − 4 <0, (v ₁ , u ₅ ) = (4,0) · (−2, − 1) = - 8 <0, since counter = -5 <0, the region R ₂ is judged to shielding. From Table 1, the frame serving as the reference area is the decoding target frame, and the re-search reference frame is the reference frame. Assuming that the second motion vector v ₂ is obtained as v ₂ = (p, q) as shown in FIG. 3 as a result of performing the re-motion detection on the reference frame for the region R ₂ in the decoding target frame, From the area corresponding to R ₂ on block C, v ₂
Pixel values of the corresponding region R ₂ of the upper half of the reference frame of performing motion compensation by the size of are copied to the region R _2. That is, C (i + x / 2, j + y / 2) = B (i + x / 2)
+ P / 2, j + y / 2 + q / 2).

In this example, it is assumed that the interpolated frame is temporally centered with respect to the two decoded frames. However, the interpolated frame is located at an arbitrary time position in the two decoded frames. Can be easily realized by changing the scaling value of the vector.

Further, some or all of the blocks shown in FIGS. 1 and 2 can be made to function using a computer, or the processing steps described with reference to FIGS. 1, 2 and 3 can be performed by a computer. Needless to say, a program for causing a computer to function as that means, or a program for causing a computer to execute the processing steps,
The computer-readable recording medium, for example,
It can be recorded on an FD (floppy disk), MO, ROM, memory card, CD, DVD, removable disk, etc., provided, and distributed.

[0042]

According to the method and apparatus of the present invention as described above, since the interpolation mode can be set more finely than in the small block unit, even if occlusion occurs and there is an overlap between objects, Frame interpolation can be performed with high accuracy.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of a block configuration for carrying out the present invention and explaining an embodiment of a method according to the present invention.

FIG. 2A is a block diagram illustrating a first interpolation value calculator in the block diagram, and FIG. 2B is a block diagram illustrating a second interpolation value calculator in the same block diagram.

FIG. 3 is a diagram showing a specific processing example in the embodiment.

FIG. 4 is a conceptual diagram of a frame interpolation method using motion compensated prediction coding.

FIG. 5 is a block diagram of an image encoding / decoding device implementing a conventional frame interpolation method.

FIG. 6 is a diagram showing a problem of a conventional frame interpolation method.

[Explanation of symbols]

101: quantized motion-compensated inter-frame difference value 102: first motion vector 103: inverse quantizer 104: motion-compensated inter-frame difference value 105: adder 106: reference image block 107: decoded image block 108: motion Compensation unit 109 Reference image 110 Region division unit 111 Frame memory 112 Vector scaling unit 113 Scaled first motion vector 114 Region R ₁ 115 Region R ₂ 116 First interpolation value calculation unit 117 ... second interpolation value calculation unit 118 ... average value of pixel values 119 ... pixel value of extracted area 120 ... interpolation image frame memory 121 ... interpolation image block 122 ... average value calculation unit 123 ... shield / appearance determination unit 124 ... Shielding / appearance relationship 125 ... Switch 126 ... Reference area extraction unit 127 ... Reference area 128 ... No. 2 motion vector extraction unit 129 second motion vector 130 vector scaling unit 131 scaled second motion vector 132 re-search frame 133 area extraction unit

 ────────────────────────────────────────────────── ─── Continued on the front page (72) Inventor Yoshiyuki Yajima 3-19-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo F-Term within Nippon Telegraph and Telephone Corporation (reference) 5C053 FA27 GB19 GB29 GB32 HA33 KA04 KA05 LA11 LA15 5C059 KK01 LB07 LB13 MA05 MD02 NN02 NN28 PP04 RB02 RC16 SS20 TA09 TB08 TC03 TD12 UA02 UA05 UA33 UA38 UA39

Claims (3)

[Claims] When encoding a digital video signal, a frame constituting the video is divided into an encoding target frame and a non-coding frame, and the image data of the encoding target frame is divided into small blocks. For the divided small block to be encoded, a first motion vector is obtained from the encoded and decoded reference frame stored in the frame memory, and a reference block determined by the first motion vector is obtained. And a motion compensation inter-frame difference value between the current block and the current block to be coded, and coded together with the first motion vector for transmission / recording.・
A decoded frame is constructed by adding the reference block in the frame memory determined by the recorded first motion vector and the motion-compensated inter-frame difference value. First
In the image encoding and decoding method of forming an interpolation frame by an interpolation block calculated using the motion vector of the motion vector, when determining the pixel value of the interpolation block in the interpolation frame, the motion compensation frame difference absolute value of the small block to be decoded is determined. Is compared with a predetermined threshold value for each pixel, and a first area composed of pixels whose absolute value of the inter-frame difference is smaller than the threshold value, and a first area composed of pixels whose absolute value of the inter-frame difference is larger than the threshold value Dividing the first region into pixel values of the first region and the pixel values of the first region.
Constructing an interpolated frame with a weighted average of pixel values corresponding to a first region in a reference block defined by the motion vector of the reference block; and, for the second region, a distribution of pixels constituting the region. Calculating a relationship between a state and the direction of the first motion vector; and either the corresponding area on the decoding target block or on the reference block defined by the first motion vector based on the calculated relationship. As a reference region, and a frame different from the frame to which the reference region belongs as a re-search reference frame; and performing a motion detection between the reference region and the re-search reference frame to obtain a second motion vector. And, for a second area in the interpolation block, copy and interpolate pixel values on a re-search reference frame determined by multiplying the second motion vector by a real number. Constructing a frame. 2. Encoding a digital moving image signal, when encoding, divides a frame constituting the moving image into an encoding target frame and a non-encoding frame, and converts the image data of the encoding target frame into small blocks. For the divided small block to be encoded, a first motion vector is obtained from the encoded and decoded reference frame stored in the frame memory, and a reference block determined by the first motion vector is obtained. And a motion compensation inter-frame difference value between the current block and the current block to be coded, and coded together with the first motion vector for transmission / recording.・
A decoded frame is constructed by adding the reference block in the frame memory determined by the recorded first motion vector and the motion-compensated inter-frame difference value. First
In an image encoding and decoding apparatus configured as an interpolated frame by an interpolated block calculated by using the motion vector of Is compared with a predetermined threshold value for each pixel, and a first area composed of pixels whose absolute value of the inter-frame difference is smaller than the threshold value, and a first area composed of pixels whose absolute value of the inter-frame difference is larger than the threshold value Region dividing means for dividing into a second region to be processed, and for the first region, a pixel value of the region and the first region
Interpolation value calculating means for forming an interpolated frame by a weighted average value with a pixel value corresponding to a first area in a reference block defined by a motion vector, and for the second area, the area is formed. Occlusion / appearance determination means for calculating the relationship between the distribution state of pixels and the direction of the first motion vector; and on the block to be decoded or on the reference block determined by the first motion vector based on the calculated relationship. A reference region extracting means for setting one of the corresponding regions as a reference region and a frame different from the frame to which the reference region belongs as a re-search reference frame; and detecting a motion between the reference region and the re-search reference frame. A second motion vector extracting means for performing a second motion vector to obtain a second motion vector; and multiplying the second motion vector by a real number for a second region in the interpolation block. And a region extracting means for forming an interpolated frame by copying pixel values on the re-search reference frame determined by the above method. 3. A recording medium recording a frame interpolation method, wherein a program for causing a computer to execute the steps in the frame interpolation method according to claim 1 is recorded on a recording medium readable by the computer.