JP2919211B2 - Video frame interpolation method and coding / decoding method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image frame interpolation method and apparatus, and more particularly to a moving image frame interpolation method and apparatus capable of synthesizing an interpolation image with little distortion even in the vicinity of a plurality of subject areas.

[0002] The present invention also relates to a method and an apparatus for encoding and decoding moving images using frame interpolation.

[0003]

2. Description of the Related Art In a conventional video frame interpolation method,
The interpolation frame is divided into blocks of a predetermined size, additional information such as motion vector and reference image selection information is detected and encoded in block units, and the decoding side uses the additional information to perform decoding. It has been general to synthesize an interpolated image from the reference images of the two frames. As an example, a moving picture encoding / decoding system (ISO-IEC / JTC1 SC29) whose international standardization is being advanced.
In the IEC DIS 11172), the interpolation frame is divided into blocks of 16 pixels and 16 lines on the encoding side, motion vectors are detected for each of the two preceding and succeeding frames of the reference image for each block, and the reference image of the two frames is further detected. It is determined which of them is used for the interpolation processing, and the result is encoded. On the decoding side, first, a reference image 2
The image of the frame is decoded, and then the interpolation data of the motion-compensated frame is read from the reference image for each block of the interpolation frame to synthesize the interpolated image. If the difference between the synthesized interpolated image and the original image is large, the difference data can be encoded and decoded.

[0004]

In the conventional moving picture frame interpolation method, the interpolation frame is divided into blocks of a predetermined size regardless of the content of the image, and the motion compensation frame is inserted from the reference image in block units. Since the interpolated image is synthesized by reading the interpolated data, there is a problem that a change in motion in a frame can be dealt with only in block units, and the interpolated image is easily distorted. In particular, in a block that straddles two or more subject areas having greatly different motions, only at most one subject area can be accurately compensated for, and a large distortion is generated near the boundary between a plurality of areas. Also, when encoding and decoding the difference data between the synthesized interpolated image and the original image, the interpolated image has many distortions for the above-described reason, and the difference information to be encoded cannot be sufficiently reduced. There were problems such as pressure on efficiency.

An object of the present invention is to provide a moving picture frame interpolation method and apparatus capable of synthesizing an interpolation image with little distortion even in the vicinity of a plurality of subject areas by switching the motion compensation frame interpolation method for each pixel. Is to do.

It is another object of the present invention to provide a method and apparatus for encoding and decoding a moving image using frame interpolation.

[0007]

According to the present invention, there is provided a moving picture frame in which two frames of a moving picture are used as a reference picture, and a frame at a position between the reference pictures in time is interpolated. In the insertion method, motion information between frames of the reference image is detected from the reference image, and the reference image is divided into subject regions having different motions using the detected motion information. Edge information describing a boundary is detected, and a motion vector for synthesizing an interpolated image using the motion information and the edge information and which of the two frames of the reference image is used for the interpolation process is described. It is characterized in that both the interpolation control information and the interpolation control information are detected for each pixel, and an interpolation image is synthesized from the reference image by a motion compensation frame interpolation process using the information detected for each pixel.

The present invention also relates to a method for encoding and decoding a moving image using frame interpolation, comprising detecting motion information between temporally consecutive image frames from an input moving image,
Measure the fluctuation of the motion information in the time axis direction between frames, and
Determine a thinned frame according to the amount of fluctuation , output as frame thinning control information, thin out only the image of the frame specified by the frame thinning control information from the input video, encode the other images, The decoded image is decoded, and two frames temporally adjacent to the decoded image are used as reference images, and images of frames located temporally between the reference images are synthesized by motion compensation frame interpolation. It is characterized by doing.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a moving picture frame interpolation method and apparatus according to the present invention will be described with reference to FIG. FIG. 1 is a block diagram of an embodiment of an apparatus for realizing a method for interpolating a moving image frame according to the present invention.

This frame interpolation device includes a motion detection circuit 11 for detecting motion information between frames of the reference image from the reference image, and divides the reference image into object regions having different motions using the detected motion information. A region detection circuit 12 for detecting edge information describing a boundary of the divided region;
Both a motion vector for synthesizing an interpolated image using the motion information and the edge information and interpolation control information describing which of the two frames of the reference image are used for the interpolation process are detected for each pixel. It comprises a motion compensation control circuit 13 and a motion compensation interpolation circuit 14 for synthesizing an interpolated image from a reference image by a motion compensation frame interpolation process using information supplied from the motion compensation control circuit 13.

The operation of the embodiment will be described. First, the motion detection circuit 11 detects and outputs motion information 103 between the reference images from the two frames of reference images 101 and 102. As an example of the motion information detection method, a block matching method can be used. In this case, a motion vector for each block having a predetermined size is detected as the motion information 103.

The area detection circuit 12 divides each of the reference images 101 and 102 into objects having different motions using the detected motion information 103 between the reference images, and outputs area boundaries as edge information 104 and 105. . As an example of the area detection, first, edge detection processing such as a two-dimensional high-pass filter in a frame is performed on the reference images 101 and 102 to detect various edges, and an edge overlapping a portion in the reference image frame where motion is changing is detected. It is extracted and output as edge information 104 and 105 indicating the area boundaries of different motions. Here, when the motion information 103 is a motion vector detected for each representative point, if the difference between adjacent motion vectors is larger than a predetermined threshold,
A portion between the corresponding representative points is determined as a portion having a change in motion. In FIG. 1, the motion information 103 and the edge information 10
Are detected from the reference images 101 and 102. If at least one of the motion information 103 and the edge information 104 and 105 is detected in advance on the encoding side and supplied as encoded information, the May be used.

In the motion compensation control circuit 13, the edge information 10
4, 105 and the motion information 103 between the reference images,
A motion vector 106 required to execute motion compensation frame interpolation is calculated and output for each pixel. Also, interpolation control information 107 for selecting which of the two frames of the reference images 101 and 102 is used for the motion compensation frame interpolation is similarly output for each pixel.

The motion compensation interpolation circuit 14 uses the information output from the motion compensation control circuit 13 to read reference data from at least one of the reference images 101 and 102 for each pixel of the interpolation frame. This is performed on all the pixels of the interpolation frame to synthesize the interpolation image 108. Here, when the reference data is read from only one of the reference images, the read value is directly used as the pixel value of the interpolated image. When reading is performed from both reference images, the average value of the two read values or the weighted average value in consideration of the inter-frame distance is used as the pixel value of the interpolated image.

The concept of frame interpolation according to the present invention will be described with reference to FIG. The moving image in FIG. 2 includes a stationary background portion B and a subject region A moving in the upper left direction. An interpolated image can be synthesized by internally dividing a motion vector between reference images to obtain a motion-compensated frame interpolation position, and substituting pixel values read from the reference image. A requires different processing from A. It is also necessary to consider that a part of the image appears and disappears as the subject moves. In FIG. 2, since the subject area A1 and the background area B2 on the interpolated image are displayed on both reference images, the motion compensation frame interpolation may be performed from either reference image. On the other hand, the region B3 is an uncovered portion newly displayed in accordance with the movement of the subject, and is an image portion not displayed in the reference image Ef. Therefore, only the reference data read out from the temporally later reference image Fb is stored. Used for frame interpolation. Conversely, the region B4 is a covered portion that is hidden with the movement of the subject and is an image portion that is not displayed in the reference image Fb. Therefore, only the reference data read from the temporally previous reference image Ff is used for frame interpolation.

The adaptive control of the frame interpolation process including the covered / uncovered judgment on the interpolated image shown in FIG. 2 is executed by the motion compensation control circuit 13 in FIG. The procedure will be described with reference to FIG. Note that in FIG. 3, for simplification of the description, the description will be made on one dimension assuming that one subject moves in one direction. FIG. 3 shows a case where an interpolated image Fi is synthesized from two frames of reference images Ff and Fb separated by n frames, and the interpolated image Fi is located m frames away from the reference image Ff in the time direction. The motion compensation control circuit 13 in FIG. 1 knows the edge points Eb1 and Eb2 on Fb and the corresponding data points Ef1 and Ef2 on Ff from the edge information 104 and 105. Also, the motion compensation positions Pf, Qf, Rf, and Sf on the representative points Pb, Qb, Rb, and Sb on the Fb based on the motion information 103, respectively.
I know. First, the positions of the corresponding edge points and the representative points between the reference images of two frames are internally divided in consideration of the frame interval, and the edge points Ei1 and Ei2 and the representative points Pi, Qi, Ri, and Si on the interpolated image. And get Reference images Ff, Fb
And the interpolated image Fi are respectively divided into three areas (Df1, Df2, Df3), (Db1, Db2, Db) by these edge points.
3), (Di1, Di2, Di3).
The divided area is (Dfj → Dij → Db) between three frames.
j), the same index areas correspond to each other, and the reference data when Dij is interpolated are Dfj, Db
Read only from j. Also, the representative point P on the interpolated image Fi
The motion vectors of i, Qi, Ri, and Si are represented by the reference image Fb.
The motion vector of the above corresponding representative point is internally divided and obtained.
Specifically, if the motion vector of the representative point Qb is v _q ,
The motion vector of Qi is

[0017]

(Equation 1)

And the reference image Fb

[0019]

(Equation 2)

The following two values are obtained. Next, the motion vectors of all the pixel points on the interpolated image Fi are obtained by interpolation / extrapolation calculation from the motion vectors of the two representative points. This calculation differs depending on whether or not two representative points adjacent to the pixel of interest are in the same area. Since Qi and Ri are in the same area Di2, the motion vector of the pixel point between Qi and Ri is obtained by linear interpolation from the respective motion vectors of Qi and Ri. Since Ri and Si are representative points on different areas, the motion vector of the pixel point between them is a value extrapolated from one of the representative points. As a simple concrete example, the value of the motion vector of the representative point Ri is directly substituted for the motion vector of each pixel point between Ri and Ei2.
At each pixel point between 2 and Si, the value of the motion vector of the representative point Si is substituted as it is. Finally, which of the two frames of the reference image is used for the frame interpolation is determined for each pixel from the motion vector obtained by the above calculation. For example, the area Di
2 at the interpolation pixel point Ti, the reference positions Tf and Tb between the motion compensation frames are on the regions Df2 and Db2 with the same index for both the reference images Ff and Fb. Data read from the reference image may be used for interpolation of Ti. On the other hand, at the interpolation pixel point Ui in the region Di3, the motion-compensated inter-frame reference position Ub on the reference image Fb is the region D having the same index.
Although it is within b3, the reference position Uf on the reference image Ff is on a region Df2 of a different index. Since data reference between regions with different indexes cannot be read, only data read from the reference image Fb is used for frame interpolation for the interpolation pixel point Ui. The above processing is performed on all pixel points of the interpolated image Fi to obtain the motion vector 106 and the interpolation control information 107 in FIG. Note that in FIG. 3, the description has been made on one dimension for simplicity, but the same processing can be performed on an image having a two-dimensional spread. Generally, on a two-dimensional image, four nearby representative points (Pi1,
Pi2, Pi3, Pi4) are given, and interpolation / extrapolation calculation is performed from the motion vector of the representative point in the same area as Zi among these four points to obtain the motion vector of the interpolated pixel point Zi.

Next, an embodiment of a method and an apparatus for frame interpolation coding / decoding of a moving image according to the present invention will be described with reference to FIG. FIG. 4 is a block diagram of an embodiment of an apparatus for realizing the method of frame interpolation encoding / decoding of a moving image according to the present invention.

This frame interpolation coding / decoding apparatus includes a motion detecting circuit 21 for detecting motion information between temporally continuous image frames from an input moving image, and a thinning frame using the detected motion information. A frame thinning control circuit 22 for determining and outputting as frame thinning control information;
An encoding circuit 23 for thinning out only the image of the frame specified by the frame thinning control information from the input moving image and encoding the other images, a decoding circuit 24 for decoding the encoded image, and a decoding circuit A memory 25 for holding the decoded image obtained, and a frame for combining two temporally adjacent frames of the decoded image as reference images and synthesizing an image of a frame located temporally between the reference images by motion compensation frame interpolation processing It comprises an interpolation circuit 26 and a display order conversion circuit 27 for rearranging the reference image and the interpolation image in a correct display order and outputting the reproduced image as a reproduced image.

The operation of this embodiment will be described. First, the motion detection circuit 21 detects a motion between frames for each subject region having a different motion from the input moving image 201 and obtains the region motion information 20.
Output as 2. The frame thinning control circuit 22 analyzes the fluctuation of the motion in units of area using the area motion information 202, determines a frame in which the motion does not change in any area, and displays frame thinning information 203 indicating that this frame is thinned. Is output. The encoding circuit 23 refers to the frame thinning information 203 and encodes the input moving image 201 while performing frame thinning. On the other hand, on the decoding side,
The decoding circuit 24 decodes the encoded data 204 into a decoded image 205 and stores it in the memory 25. When the inter-frame motion information and the intra-frame edge information of the image encoded by the encoding circuit 23 are also encoded and supplied together, the decoding circuit 24 decodes these and decodes the decoded image 20.
5 is stored in the memory 25 as additional information. The frame interpolation circuit 26 reads the decoded images of the two frames held in the memory 25 as reference images 206 and 207, and synthesizes an interpolation image 208. Here, when the additional information is read from the memory 25, the interpolation processing is executed with reference to the additional information. On the other hand, when the additional information is not supplied, the reference images 206 and 2 are stored inside the frame interpolation circuit 26.
07, information necessary for frame interpolation is detected, and then an interpolated image is synthesized. The display order conversion circuit 27 outputs the reference image 2
06 and 207 and the interpolated image 208 are rearranged in a correct display order and output as a reproduced image 209.

An example of the frame thinning process according to the present invention will be described with reference to FIG. In FIG. 5, for simplification of the description, a case where one subject region is moving in one direction is described on one dimension. In FIG. 5, the subject area C is the first
During the period from the frame to the Fm-th frame, it moves at a slow constant speed, from the Fm-th frame to the Fn-th frame has a fast constant speed, and after the Fn-th frame, it moves again at a slow constant speed. That is, the movement of the area C changes only in the Fm and Fn frames, and in other frame periods, the movement is uniform and does not change. As an example of a specific method of detecting a motion change, the difference between the motion vector between the Fm-1 and Fm-th frames and the motion vector between the Fm and Fm + 1-th frames is larger than the predetermined threshold value. It is determined that the movement has changed in the frame. In addition, when the fluctuation of the motion in several frames before and after the Fm-th frame is approximated by a linear function, the Fm-th
If the magnitude of the primary gradient changes by a predetermined threshold value or more from the frame, the Fm-th frame may be determined as the change point of the motion. In the frame thinning control circuit 22 of FIG.
It is determined that frames other than Fn are to be thinned out. However, in the case of FIG. 5, since the movement of the area C is not zero in the entire frame period, the first frame and the last frame of the moving image sequence are not thinned out. Different from FIG. 5, the first to F-th
For example, when there is no motion in the image in a frame period up to m, the first frame may be skipped. The same holds for the last frame. In FIG. 5, there is only one moving area, but when there are a plurality of subject areas having different movements, frame thinning is executed when there is no change in the movement in all the areas.

[0025]

As described above, according to the present invention, in synthesizing an interpolated image from a decoded two-frame reference image, an object region having a different motion is detected on the reference image, and the motion compensated frame interpolation is performed. And the reference data actually used for motion compensation frame interpolation are determined for each pixel, so that it is possible to synthesize an interpolated image with little distortion up to the vicinity of the boundary between a plurality of subject regions. Further, in the present invention, since only the image frames for which motion-compensation frame interpolation can be performed on the decoding side are determined in advance on the encoding side and then the frame thinning-out coding is performed, degradation of the interpolated image on the decoding side is suppressed. This is effective in reducing the amount of codes while reducing the number of codes.

[Brief description of the drawings]

FIG. 1 is a basic block diagram of an embodiment of a moving picture frame interpolation apparatus according to the present invention.

FIG. 2 shows a case where there are a plurality of subject areas having different movements.
FIG. 7 is a diagram illustrating that an image referred to in frame interpolation differs for each region.

FIG. 3 shows a case where there are a plurality of subject areas having different movements.
FIG. 7 is a diagram illustrating a procedure for determining selection of a motion vector and a reference image for each pixel of an interpolation image.

FIG. 4 is a basic block diagram of a moving image encoding / decoding apparatus according to an embodiment of the present invention.

FIG. 5 shows a case where there are a plurality of subject areas having different movements.
FIG. 7 is a diagram illustrating a procedure for determining an image frame to be subjected to frame thinning.

[Explanation of symbols]

 11, 21 motion detection circuit 12 area detection circuit 13 motion compensation control circuit 14 motion compensation interpolation circuit 22 frame thinning control circuit 23 encoding circuit 24 decoding circuit 25 memory 26 frame interpolation circuit 27 display order conversion circuit 101, 102, 206, 207 Reference image 103 Motion information 104, 105 Edge information 106 Motion vector 107 Interpolation control information 108, 208 Interpolated image 201 Input moving image 202 Area motion information 203 Frame thinning control information 204 Encoded data 205 Decoded image 209 Reproduction image

Continuation of the front page (56) References JP-A-62-108687 (JP, A) JP-B-56-2829 (JP, B2) PROCEEDINGS OF THE IEEE, VOL 73 [4] 1985 523-548 (V. MOTION-AD APTIVE FRAME INTERPOLATION (P. 542-545)); (58) Fields investigated (Int. Cl. ⁶ , DB name) H04N 7/137 H04N 7/12 H03M 7/30

Claims (4)

(57) [Claims] 1. A frame interpolation method for a moving image in which two frames of a moving image are used as a reference image and a frame at a position between the reference images in time is synthesized as an interpolation image. Detecting motion information between frames from the reference image, dividing the reference image into subject regions having different motions using the detected motion information, and detecting edge information describing boundaries of the divided regions. Then, both the motion vector for synthesizing the interpolated image using the motion information and the edge information and the interpolation control information that describes which of the two frames of the reference image is used for the interpolation process. A frame interpolation method for a moving image, wherein an interpolation image is synthesized by a motion compensation frame interpolation process from the reference image using information detected for each pixel and information detected for each pixel. 2. A moving image frame interpolation device for combining two frames of a moving image as a reference image and synthesizing a frame at a position between the reference images with respect to time as an interpolation image. Means for detecting motion information between frames from the reference image, and edge information for dividing the reference image into object regions having different motions using the detected motion information, and describing boundaries of the divided regions. And a motion vector for synthesizing an interpolated image using the motion information and the edge information, and interpolation control information describing which of the two frames of the reference image is used for the interpolation process. Using the information supplied from the motion compensation control means for detecting both of the above for each pixel, and synthesizing an interpolated image from the reference image by motion compensation frame interpolation processing. Means for interpolating a frame of a moving image. 3. A moving picture encoding / decoding method using frame interpolation, wherein motion information between temporally consecutive image frames is detected from an input moving picture, and a motion of the motion information is detected.
Frames with uniform fluctuation in the time axis between frames
Determined as a frame, output as frame thinning control information, thinning out only the image of the frame specified by the frame thinning control information from the input video, encoding the other images, decoding the encoded image And using two frames temporally adjacent to each other in the decoded image as a reference image, and synthesizing an image of a frame positioned temporally between the reference images by a motion compensation frame interpolation process. A coding and decoding method for a moving image. 4. An apparatus for encoding and decoding a moving image using frame interpolation, comprising: a motion detecting means for detecting motion information between temporally consecutive image frames from an input moving image;
A frame in which the fluctuation of the motion information in the frame
Means determines that the frame thinning over arm, and means for outputting a frame thinning control information, which thinned out from the input moving image only the image of the frame specified by the frame thinning control information, to encode the other image, Means for decoding the encoded image, and using two frames temporally adjacent to the decoded image as a reference image, and motion-compensating an image of a frame located in time between the reference images. A coding / decoding device for a moving image, comprising: means for synthesizing by a frame interpolation process.