JPH06197331A - Frame interpolation system for animation picture - Google Patents

Abstract

PURPOSE:To improve the picture quality of a decoded reproduction picture and to realize the effective utilization of an entire code quantity by improving the efficiency of frame interpolation in a high efficiency coding/decoding of the animation picture employing frame interpolation. CONSTITUTION:A motion detection circuit 11 detects motion information 103 between frames of two frame reference pictures 101, 102. An area detection circuit 12 utilizes motion information 103 to divide the reference pictures 101, 102 for each of object areas of different motion and detects respective area border line as edge information sets 104, 105. A motion compensation control circuit 13 utilizes detected information sets 103, 104, 105 to detect a motion vector 106 and interpolation control information 107 for each picture element of the interpolation frame. A motion compensation interpolation circuit 14 controls the motion compensation frame interpolation processing for each pixel based on the detected information sets 106, 107 and synthesizes an interpolated picture 108 from the two frame reference pictures 101, 102.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture frame interpolating method and apparatus, and more particularly to a moving picture frame interpolating method and apparatus capable of synthesizing interpolated images with little distortion even in the vicinity of a plurality of subject areas.

The present invention also relates to a moving picture coding / decoding method and apparatus using frame interpolation.

[0003]

2. Description of the Related Art In the conventional frame interpolation method for moving images,
The interpolated frame is divided into blocks of a predetermined size, additional information such as motion vector and reference image selection information is detected and coded in block units, and the decoding side uses the additional information to perform decoding. It was common to synthesize an interpolated image from two frames of reference images. As an example, a moving image encoding / decoding method (ISO-ISO / IEC / JTC1 SC29) whose international standardization is being advanced.
In IEC DIS 11172), an interpolating frame is divided into blocks of 16 pixels and 16 lines on the encoding side, a motion vector is detected for reference images of two frames before and after for each block, and further, a reference image of the reference images of the two frames is detected. It is determined which of them is used for the interpolation processing, and the results are encoded. At the decoding side, it becomes the reference image first 2
The image of the frame is decoded, and then the motion compensated frame interpolation data is read from the reference image for each block of the interpolated frame to synthesize the interpolated image. When the difference between the synthesized interpolated image and the original image is large, the difference data can be coded and decoded.

[0004]

In the conventional moving image frame interpolation method, the interpolation frame is divided into blocks of a predetermined size regardless of the contents of the image, and the reference image is inserted into the motion compensation frame in block units. Since the interpolated image is read out and the interpolated image is synthesized, there is a problem in that the interpolated image is likely to be distorted because the interpolated image is combined only with a change in the movement within the frame. In particular, in a block that spans two or more subject regions that greatly differ in motion, at most, only the motion of one subject region can be accurately compensated, so that a large distortion occurs near the boundary of a plurality of regions. Further, when the difference data between the synthesized interpolated image and the original image is encoded and decoded, the interpolated image has a lot of distortion for the above reason, and the difference information to be encoded cannot be sufficiently reduced. There were problems such as pressure on efficiency.

An object of the present invention is to provide a moving image frame interpolation method and apparatus capable of synthesizing an interpolated image with little distortion even in the vicinity of a plurality of subject areas by switching the motion compensation frame interpolation method for each pixel. To do.

Another object of the present invention is to provide a moving picture coding / decoding method and apparatus using frame interpolation.

[0007]

SUMMARY OF THE INVENTION According to the present invention, two frames of a moving image are used as reference images, and frames at positions temporally between the reference images are combined as interpolated images. In the insertion method, motion information between frames of the reference image is detected from the reference image, the reference image is divided for each subject region having different motion using the detected motion information, and the divided regions are Edge information describing a boundary is detected, and which of the two frames of the reference image and the motion vector for combining the interpolated image using the motion information and the edge information is described for the interpolation processing. Both of the interpolation control information are detected for each pixel, and the information detected for each pixel is used to synthesize an interpolated image from the reference image by motion compensation frame interpolation processing.

Further, according to the present invention, in a moving picture coding / decoding method using frame interpolation, motion information between temporally consecutive image frames is detected from an input moving picture, and the detected motion information is detected. Determine the thinning-out frame using, output as frame thinning-out control information, thin out only the image of the frame specified by the frame thinning-out control information from the input moving image, encode the other images, and encode the Decoding an image, using two frames temporally adjacent to the decoded image as reference images, and synthesizing images of frames positioned temporally between the reference images by motion compensation frame interpolation processing. Is characterized by.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a moving image frame interpolation method and apparatus according to the present invention will be described with reference to FIG. FIG. 1 is a block diagram of an embodiment of an apparatus for realizing a moving picture frame interpolation method of the present invention.

This frame interpolating device divides a reference image into object regions having different motions by using a motion detection circuit 11 for detecting motion information between frames of a reference image from the reference image and the detected motion information. And an area detection circuit 12 that detects edge information that describes the boundaries of the divided areas,
For each pixel, both a motion vector for synthesizing an interpolated image using motion information and edge information and interpolation control information describing which of two frames of a reference image is to be used for interpolation processing are detected. It is composed of a motion compensation control circuit 13 and a motion compensation interpolation circuit 14 which synthesizes an interpolated image from a reference image by a motion compensation frame interpolation process using information supplied from the motion compensation control circuit 13.

The operation of this embodiment will be described. First, the motion detection circuit 11 detects and outputs the motion information 103 between the reference images from the reference images 101 and 102 of the two frames. A block matching method can be used as an example of the motion information detection method. In this case, a motion vector for each block having a predetermined size is detected as the motion information 103.

The area detection circuit 12 uses the detected movement information 103 between the reference images to divide the reference images 101 and 102 into areas for each subject having different movements, and outputs the area boundaries as edge information 104 and 105. . As an example of area detection, first, the reference images 101 and 102 are subjected to edge detection processing such as an intra-frame two-dimensional high-pass filter to detect various edges, and edges that overlap with portions of the reference image in which the movement is changed are detected. It is extracted and output as edge information 104 and 105 indicating area boundary lines with different movements. Here, when the motion information 103 is a motion vector detected for each representative point, if the difference between adjacent motion vectors is larger than a predetermined threshold value,
The part between the corresponding representative points is determined to be a part having a change in motion. In FIG. 1, the motion information 103 and the edge information 10
4 and 105 are detected from the reference images 101 and 102, but if at least one of the motion information 103 and the edge information 104 and 105 is detected in advance by the encoding side and is supplied as encoded information, these are detected. You may use.

In the motion compensation control circuit 13, the edge information 10
4, 105 and the motion information 103 between the reference images,
The motion vector 106 necessary for executing the motion compensation frame interpolation is calculated for each pixel and output. Also, interpolation control information 107 for selecting which of the reference images 101 and 102 of the two frames is to be used for motion compensation frame interpolation is similarly output for each pixel.

The motion compensation interpolation circuit 14 uses the information output from the motion compensation control circuit 13 to read out reference data from at least one of the reference images 101 and 102 for each pixel of the interpolation frame. This is executed for all the pixels of the interpolated frame to synthesize the interpolated image 108. Here, when the reference data is read from only one of the reference images, the read value is directly used as the pixel value of the interpolated image. In the case of reading from both reference images, the average value of the two read values or the weighted average value considering the inter-frame distance is used as the pixel value of the interpolated image.

The concept of frame interpolation according to the present invention will be described with reference to FIG. The moving image in FIG. 2 is composed of a stationary background portion B and a subject area A moving in the upper left direction. The interpolated image can be synthesized by internally dividing the motion vector between the reference images to obtain the motion compensation frame interpolated position, and substituting the pixel values read from the reference image, but the background part B and the region having different motions can be combined. Different processing is required for A. It is also necessary to consider that a part of the image appears and disappears as the subject moves. In FIG. 2, since the subject area A1 and the background area B2 on the interpolated image are displayed on both reference images, the motion compensation frame interpolation may be performed from any of the reference images. On the other hand, the region B3 is an uncovered part that is newly displayed as the subject moves, and is an image part that is not displayed in the reference image Ef. Therefore, only the reference data read from the reference image Fb that is temporally later is displayed. Used for frame interpolation. On the other hand, the area B4 is a covered portion that is hidden by the movement of the subject and is an image portion that is not displayed in the reference image Fb, so only reference data read from the reference image Ff that is temporally previous is used for frame interpolation.

The adaptive control of the frame interpolation process including the covered / uncovered decision on the interpolated image shown in FIG. 2 is executed by the motion compensation control circuit 13 of FIG. The procedure will be described with reference to FIG. Note that, in FIG. 3, for simplification of the description, description will be given in one dimension assuming that one subject moves in one direction. FIG. 3 shows a case where the interpolated image Fi is synthesized from the reference images Ff and Fb of two frames separated by n frames, and the interpolated image Fi is located at a position m frames away from the reference image Ff in the time direction. In the motion compensation control circuit 13 in FIG. 1, the edge information 104 and 105 identify the edge points Eb1 and Eb2 on Fb and the corresponding data points Ef1 and Ef2 on Ff. Further, according to the motion information 103, the motion compensation positions Pf, Qf, Rf, Sf on Ff of the representative points Pb, Qb, Rb, Sb on Fb, respectively.
Is known. First, the positions of corresponding edge points and representative points between the reference images of two frames are internally divided in consideration of the frame interval, and the edge points Ei1, Ei2 and the representative points Pi, Qi, Ri, Si on the interpolated image are internally divided. And get. Reference images Ff and Fb
And the interpolated image Fi are divided into three regions (Df1, Df2, Df3), (Db1, Db2, Db) by these edge points.
3), (Di1, Di2, Di3).
The divided area is (Dfj → Dij → Db) between three frames.
j), the same index areas correspond to each other, and the reference data for interpolating Dij are Dfj and Db.
Read only from j. In addition, the representative point P on the interpolated image Fi
The motion vectors of i, Qi, Ri, and Si are the reference images Fb.
The motion vector of the corresponding representative point above is internally calculated.
Specifically, if the motion vector of the representative point Qb is v _q ,
The motion vector of Qi is relative to the reference image Ff

[0017]

[Equation 1]

With respect to the reference image Fb,

[0019]

[Equation 2]

Two values are obtained. Next, the motion vectors of all pixel points on the interpolated image Fi are calculated from the motion vectors of the two representative points by interpolation / extrapolation calculation. This calculation method differs depending on whether or not two adjacent representative points of the pixel of interest are in the same area. Since Qi and Ri are in the same area Di2, the motion vector of the pixel point between Qi and Ri is obtained by linear interpolation from the motion vectors of Qi and Ri. Further, since Ri and Si are representative points on different regions, the motion vector of the pixel point between them is a value extrapolated from one of the representative points. As a simple concrete example, for the motion vector of each pixel point between Ri and Ei2, the value of the motion vector of the representative point Ri is directly substituted, and Ei
At each pixel point between 2 and Si, the value of the motion vector of the representative point Si is directly substituted. Finally, which of the two reference images is used for frame interpolation is determined for each pixel from the motion vector obtained by the above calculation. For example, the area Di
At the interpolated pixel point Ti within 2, since the motion compensation inter-frame reference positions Tf and Tb are on the regions Df2 and Db2 with the same index for both Ff and Fb reference images, both or either one of them. Data read from the reference image may be used for Ti interpolation. On the other hand, at the interpolation pixel point Ui of the area Di3, the motion compensation inter-frame reference position Ub on the reference image Fb is the area D of the same index.
Although it is within b3, the reference position Uf on the reference image Ff is on the region Df2 of a different index. Since the data reference between the regions of different indexes cannot be read, only the data read from the reference image Fb is used for the frame interpolation for the interpolation pixel point Ui. The above processing is executed for all pixel points of the interpolated image Fi to obtain the motion vector 106 and the interpolated control information 107 in FIG. It should be noted that although FIG. 3 illustrates one dimension for simplification of description, similar processing can be performed on an image having a two-dimensional spread. Generally, on a two-dimensional image, four neighboring representative points (Pi1,
Pi2, Pi3, Pi4) are given, and the interpolation / extrapolation calculation is performed from the motion vector of the representative point in the same area as Zi among these four points to obtain the motion vector of the interpolated pixel point Zi.

Next, an embodiment of the moving picture frame interpolation coding / decoding method and apparatus of the present invention will be described with reference to FIG. FIG. 4 is a block diagram of an embodiment of an apparatus for implementing the moving picture frame interpolation coding / decoding method of the present invention.

This frame interpolation coding / decoding apparatus uses a motion detection circuit 21 for detecting motion information between temporally consecutive image frames from an input moving image and a thinned frame using the detected motion information. A frame thinning control circuit 22 which determines and outputs as frame thinning control information;
An encoding circuit 23 that thins out only the images of the frames specified by the frame thinning control information from the input moving image and encodes the other images, a decoding circuit 24 that decodes the encoded images, and a decoding circuit. A memory 25 for holding the decoded image and a frame for synthesizing images of frames located temporally between the reference images with two frames temporally adjacent to the decoded image by motion compensation frame interpolation processing. It is composed of an interpolation circuit 26 and a display order conversion circuit 27 which rearranges the reference image and the interpolated image in a correct display order and outputs the reproduced image as a reproduced image.

The operation of this embodiment will be described. First, the motion detection circuit 21 detects the motion between frames for each subject region having a different motion from the input moving image 201, and the region motion information 20
Output as 2. The frame thinning control circuit 22 analyzes the movement variation in each area by using the area movement information 202, determines a frame in which there is no movement variation in any area, and thins out the frame information 203 indicating that this frame is thinned. Is output. The encoding circuit 23 refers to the frame thinning information 203 and encodes the input moving image 201 while performing frame thinning. On the other hand, on the decryption side,
The decoding circuit 24 decodes the encoded data 204 into a decoded image 205 and holds the decoded image 205 in the memory 25. Further, when the inter-frame motion information and the intra-frame edge information of the image coded by the coding circuit 23 are also coded and supplied, the decoding circuit 24 decodes these to decode the decoded image 20.
It is held in the memory 25 as additional information of No. 5. The frame interpolation circuit 26 reads the decoded images of the two frames held in the memory 25 as reference images 206 and 207, and synthesizes the interpolated image 208. Here, when the additional information is read from the memory 25, the interpolation processing is executed with reference to the additional information. On the other hand, when the additional information is not supplied, the reference images 206 and 2 are generated inside the frame interpolation circuit 26.
The information required for frame interpolation is detected from 07, and then the interpolated image is synthesized. The display order conversion circuit 27 uses the reference image 2
06 and 207 and the interpolated image 208 are rearranged in the correct display order and output as the reproduced image 209.

An example of the frame thinning process according to the present invention will be described with reference to FIG. In FIG. 5, for simplification of description, it is assumed that one subject region is moving in one direction, and description will be given in one dimension. In FIG. 5, the subject area C is the first
The frame moves from the frame to the Fmth frame at a gentle constant speed, from the Fmth frame to the Fnth frame has a fast constant speed, and after the Fnth frame, moves at a gentle constant speed again. That is, the movement of the region C changes only in the Fm and Fn frames, and the movement is uniform and does not change in the other frame periods. As an example of a specific method of detecting the motion variation, if the difference between the motion vector between the Fm−1 and Fm frames and the motion vector between the Fm and Fm + 1 frames is larger than a predetermined threshold value, the Fmth frame. It is determined that the movement has changed in the frame. In addition, when the movement variation in the several frame periods before and after the Fm-th frame is approximated by a linear function,
If the magnitude of the first-order gradient changes by a predetermined threshold value or more at a frame, the Fm-th frame may be determined as a change point of motion. In the frame thinning control circuit 22 of FIG. 4, the Fm,
Decide to thin out frames other than Fn. However, in the case of FIG. 5, since the movement of the area C is not zero in the entire frame period, the first frame and the last frame of the moving image sequence are not thinned out. Unlike FIG. 5, 1st to Fth
The first frame may be thinned out when the image has no movement in the frame period up to m. The same holds for the final frame. In FIG. 5, only one region has a motion, but when there are a plurality of subject regions having different motions, frame thinning is performed when the motion does not change in all regions.

[0025]

As described above, according to the present invention, when synthesizing an interpolated image from decoded two-frame reference images, a subject region having a different motion is detected on the reference image, and motion compensation frame interpolation is performed. Since the motion vector used for the above and the reference data actually used for the motion compensation frame interpolation are determined for each pixel, it is possible to synthesize an interpolated image with less distortion up to the vicinity of the boundaries of a plurality of subject areas. Further, according to the present invention, since only the image frame that can be motion compensated frame interpolated on the decoding side is determined on the encoding side in advance and the frame thinning coding is performed, deterioration of the interpolated image on the decoding side is suppressed. However, it is effective in reducing the code amount.

[Brief description of drawings]

FIG. 1 is a basic block diagram of an embodiment of a moving picture frame interpolation apparatus of the present invention.

[FIG. 2] When there are a plurality of subject regions with different movements,
It is a figure explaining that the image referred by frame interpolation differs for every field.

[FIG. 3] When there are a plurality of subject regions with different movements,
It is a figure explaining the procedure which determines the selection of a motion vector and a reference image for every pixel of an interpolated image.

FIG. 4 is a basic block diagram of an embodiment of a moving picture coding / decoding apparatus of the present invention.

[FIG. 5] When there are a plurality of subject regions with different movements,
It is a figure explaining the procedure which determines the image frame used as the object of frame thinning.

[Description of Codes] 11, 21 Motion Detection Circuit 12 Region Detection Circuit 13 Motion Compensation Control Circuit 14 Motion Compensation Interpolation Circuit 22 Frame Thinning Control Circuit 23 Encoding Circuit 24 Decoding Circuit 25 Memory 26 Frame Interpolation Circuit 27 Display Order Conversion Circuits 101, 102, 206, 207 Reference image 103 Motion information 104, 105 Edge information 106 Motion vector 107 Interpolation control information 108, 208 Interpolated image 201 Input moving image 202 Area motion information 203 Frame thinning control information 204 Encoded data 205 Decoded image 209 Reproduced image

Claims (4)

[Claims] 1. A frame interpolating method for a moving image, wherein two frame images of the moving image are used as reference images, and frames at temporal positions between the reference images are combined as an interpolating image. Motion information between frames is detected from the reference image, the reference image is divided into object regions having different motions using the detected motion information, and edge information describing a boundary of the divided regions is detected. However, both the motion vector for synthesizing the interpolated image using the motion information and the edge information and the interpolating control information describing which of the two frames of the reference image is used for the interpolating process are stored. A frame interpolating method for a moving image, which comprises detecting an image for each pixel and synthesizing an interpolated image from the reference image by a motion compensation frame interpolation process using the information detected for each pixel. 2. A frame interpolating apparatus for a moving image, wherein two frame images of a moving image are used as reference images, and frames at positions temporally between the reference images are combined as interpolated images. Means for detecting motion information between frames from the reference image; edge information describing a boundary of the divided regions by dividing the reference image into subject regions having different motions using the detected motion information Detecting means, and interpolation control information describing which of the two vectors of the motion vector and the reference image for combining the interpolation image using the motion information and the edge information is used for the interpolation processing. And a motion compensation control means for detecting both of each pixel and each pixel, and information supplied from the motion compensation control means is used to synthesize an interpolated image from the reference image by motion compensation frame interpolation processing. And a frame interpolating device for a moving image. 3. A moving image coding / decoding method using frame interpolation, wherein motion information between temporally consecutive image frames is detected from an input moving image, and the detected motion information is used. Decimates frames, outputs them as frame thinning control information, thins only the images of the frames specified by the frame thinning control information from the input moving image, encodes the other images, and decodes the encoded images. Characterized in that two frames temporally adjacent to the decoded image that have been decoded are used as reference images, and images of frames located temporally between the reference images are combined by motion compensation frame interpolation processing. Method for encoding and decoding moving images. 4. A moving picture coding / decoding apparatus using frame interpolation, a motion detecting means for detecting motion information between temporally consecutive image frames from an input moving picture, and the detected motion. Means for determining a thinning-out frame using information and outputting it as frame thinning-out control information; and means for thinning out only the image of the frame specified by the frame thinning-out control information from the input moving image and encoding the other images. Means for decoding the coded image, two frames adjacent in time of the decoded decoded image as reference images, and an image of a frame located temporally between the reference images is moved. And a means for synthesizing by compensation frame interpolation processing.