JP6059899B2 - Frame interpolation apparatus and program - Google Patents

Description

  The present invention relates to a frame interpolation apparatus that generates an interpolation frame with reference to a plurality of frames and temporally interpolates the frame, and a program therefor.

  Conventionally, there is a method of presenting the same frame multiple times in order to convert the image speed (frame frequency) to a higher frame frequency. For example, the frame frequency can be converted to 60 Hz by presenting each frame with a frame frequency of 30 Hz twice. There is also a technique in which the number of times the same frame is presented is variable. For example, in a technique called 2-3 conversion, an odd-numbered frame having a frame frequency of 24 Hz is presented twice and an even-numbered frame is presented three times, whereby an image having a frame frequency of 60 Hz can be obtained.

  There is also a method of determining the pixel value of the interpolation frame by the average of the pixel values of two frames adjacent to the time of the interpolation frame or the weighted average.

  Furthermore, a method of generating a smoother interpolation frame by performing motion compensation using a motion vector of a video is known (see, for example, Patent Document 1).

Japanese Patent No. 3495485

  However, the conventional method of presenting the same frame multiple times has a problem that the motion of the video becomes awkward. Also, according to the method of generating an interpolation frame by averaging or weighted averaging, although the awkwardness of video motion is improved, there is a problem that the moving object becomes a double image in the interpolation frame and the resolution is lowered, and the interpolation frame There is a problem in that the difference in image quality from the original frame which is not interpolated looks like flicker.

  In addition, in the conventional technique as disclosed in Patent Document 1, although the awkwardness of movement and the double image are improved, the motion compensation error and the accuracy of the motion estimation in the case where subjects with different motions are included in the same block are improved. Due to the decimal precision error of the sample points caused by roughness, the boundaries of the blocks used for motion compensation may be noticeable. If motion compensation is performed with decimal pixel accuracy, the decimal accuracy error of the sample point can be improved, but the blurring and ringing of the contour may occur due to the characteristics of the digital filter that moves the pixel position (phase shift) with decimal pixel accuracy. May occur.

  An object of the present invention made in view of such circumstances is a frame capable of preventing a decrease in resolution and flicker and obtaining a sharp contour image including a high-frequency component when generating an interpolated image from an input video. An interpolation apparatus and a program thereof are provided.

In order to solve the above problems, a frame interpolation device according to the present invention is a frame interpolation device that generates an interpolated image of an input video, and includes a motion estimation source image, a motion estimation destination image, and a motion compensation source image from the input video. A frame selection unit that determines a plurality of sets; a motion estimation unit that obtains a reference motion vector that is a motion vector between the motion estimation source image and the motion estimation destination image for each set; and a scalar multiplication of the reference motion vector A motion compensation unit for obtaining an interpolated motion vector that is a motion vector between the motion compensation source image and the interpolated image corresponding to the reference motion vector ; and the interpolated motion vector for the plurality of motion compensation source images and an image superimposing unit configured to generate the interpolation image by superimposing a plurality of motion compensation image subjected to motion compensation in decimal precision, the image superimposition section, before Depending on the degree of similarity when calculating a motion vector, it characterized that you superimposes the motion compensated image by weighting each partial region.

The frame interpolation device according to the present invention, there is provided a frame interpolation device that generates an interpolated image of the input image, the motion estimation based on the image from the input image, the motion estimation target image, and a plurality of sets of motion compensated original image determination A frame selection unit that performs a motion estimation unit that obtains a reference motion vector that is a motion vector between the motion estimation source image and the motion estimation destination image for each of the groups; A motion compensation unit that obtains an interpolated motion vector that is a motion vector between the motion compensation source image and the interpolated image corresponding to a motion vector; and a motion of the plural sets of motion compensated source images with decimal accuracy using the interpolated motion vector It includes an image superimposing unit configured to generate the interpolation image by superimposing a plurality of motion compensation images subjected to compensation, and the image superimposing section, the point of the motion compensation image Superimposed after convolution operation by Galli function and generates the interpolation image.

The frame interpolation device according to the present invention, there is provided a frame interpolation device that generates an interpolated image of the input image, the motion estimation based on the image from the input image, the motion estimation target image, and a plurality of sets of motion compensated original image determination A frame selection unit that performs a motion estimation unit that obtains a reference motion vector that is a motion vector between the motion estimation source image and the motion estimation destination image for each of the groups; A motion compensation unit that obtains an interpolated motion vector that is a motion vector between the motion compensation source image and the interpolated image corresponding to a motion vector; and a motion of the plural sets of motion compensated source images with decimal accuracy using the interpolated motion vector by superimposing a plurality of motion compensation images subjected to compensation and an image superimposing unit configured to generate the interpolation image, the image superimposing section, enter the motion compensated image Characterized by a high-resolution image than the resolution of the image.

The frame interpolation device according to the present invention, there is provided a frame interpolation device that generates an interpolated image of the input image, the motion estimation based on the image from the input image, the motion estimation target image, and a plurality of sets of motion compensated original image determination A frame selection unit that performs a motion estimation unit that obtains a reference motion vector that is a motion vector between the motion estimation source image and the motion estimation destination image for each of the groups; A motion compensation unit that obtains an interpolated motion vector that is a motion vector between the motion compensation source image and the interpolated image corresponding to a motion vector; and a motion of the plural sets of motion compensated source images with decimal accuracy using the interpolated motion vector by superimposing a plurality of motion compensation images subjected to compensation and an image superimposing unit configured to generate the interpolation image, the image superimposing section, heavy the motion compensated image After, and generates the interpolation image by converting the resolution.

  In order to solve the above problem, a program according to the present invention causes a computer to function as the frame interpolation device.

  According to the present invention, since the pixel values of a plurality of reference frames affect one pixel of an interpolation frame, resolution reduction and flicker can be prevented, and the contour of the subject of the interpolation frame becomes smooth. In addition, when the reference frame includes aliasing distortion in the spatial direction, the phase rotation of the aliasing distortion component due to the alignment is canceled out by superimposing a plurality of frames, and the jaggy of the contour is reduced and the high frequency component is reduced. A sharp contour image including can be obtained.

It is a block diagram which shows the structural example of the frame interpolation apparatus which concerns on one Embodiment of this invention. It is a figure which shows the example of the look-up table which the selection control part in the frame interpolation apparatus which concerns on one Embodiment of this invention refers. It is a figure for demonstrating the outline | summary of operation | movement of the frame interpolation apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows operation | movement of the image superimposition part in the frame interpolation apparatus which concerns on one Embodiment of this invention.

  Hereinafter, an embodiment of a frame interpolation device according to the present invention will be described in detail with reference to the drawings. In this specification, f indicates a frame number, and I (f) indicates an image of the frame number f. The frame time position is represented by a frame number.

  FIG. 1 is a block diagram showing a configuration example of a frame interpolation apparatus according to an embodiment of the present invention. As illustrated in FIG. 1, the frame interpolation device 1 includes a frame selection unit 10, a motion estimation unit 20, a motion compensation unit 30, and an image superimposition unit 40. The frame selection unit 10 includes a frame storage unit 11, an input / output switching unit 12, and a selection control unit 13.

The frame interpolation device 1 generates an interpolation image at the frame time position f _q from a plurality of image frames of the input video. Note that the frame time position f _q may be an integer time position or a non-integer time position. For example, f _q = 3.25 indicates a time position obtained by internally dividing each time of the image of frame number 3 and the image of frame number 4 into 1: 3.

  The frame selection unit 10 determines a plurality of sets of a motion estimation source image, a motion estimation destination image, and a motion compensation source image from the input video. The motion estimation original image is an image frame that is a motion estimation target source in the motion estimation unit 20 described later. The motion estimation destination image is an image frame that is a target of motion estimation in the motion estimation unit 20 described later. The motion compensation original image is an image frame that is subjected to a superimposition process to generate an interpolated image by the image superimposing unit 40 described later.

The frame storage unit 11 stores at least two frames of the input video before the current time. For example, the frame storage unit 11 can be realized by a ring buffer. In the example shown in FIG. 1, the frame storage unit 11 can store and output seven frames from the current time (frame number f _t-0 ) to the sixth previous time point (frame number f _t-6 ).

The input / output switching unit 12 is configured by a matrix switch, for example, and operates so that each output A, B, P selects one of the inputs 0 to 6. That is, the columns 121 of the input and output switching unit 12 (output A) selects an image I of the frame number _{f a} as a motion estimation original image _{(f a)} from the frame memory unit 11, and outputs the motion estimation unit 20. The column 122 (output B) of the input / output switching unit 12 selects the image I (f _b ) of the frame number f _b as the motion estimation destination image from the frame storage unit 11 and outputs it to the motion estimation unit 20. Column 123 of the input and output switching unit 12 (output P) selects the image I _{(f p)} of the frame number _{f p} as a motion compensated original image from the frame memory unit 11, and outputs the image superimposing section 40.

The input / output switching unit 12 may select one input for a plurality of outputs, but it is not allowed for one output to select a plurality of inputs. Although not coincide with the frame number f _b of the frame number f _a and the motion estimation target image motion estimation original image, the frame number f _p of the frame number f _a and the motion compensation based on image motion estimation original image it may coincide, also, and the frame number f _p of the frame number f _b and the motion compensation based on image motion estimation target image may coincide.

  The selection control unit 13 controls the contacts of the input / output switching unit 12. The selection control unit 13 tries a plurality of contact combination patterns to be closed in the input / output switching unit 12. For example, the selection control unit 13 changes the combination pattern of the contacts in a time division manner.

  The selection control unit 13 stores, for example, a plurality of inputs to be selected by each output in a lookup table, and sequentially refers to the lookup table to generate a plurality of contact combination patterns to be closed in the input / output switching unit 12. You can try as follows.

  FIG. 2 is a diagram illustrating an example of a lookup table referred to by the selection control unit 13. In the example illustrated in FIG. 2, 13 combination patterns with k = 1 to 13 are stored. In the pattern of k = 1 to 12, the output P and the output A, or the output P and the output B are selected as the common input, and in the pattern of K = 13, the input different from each output is selected.

Further, the selection control unit 13 may have a counter inside, and may sequentially select a plurality of combinations except when the output A and the output B select the same input. For example, the selection control unit 13 can be implemented as in the following pseudo code (described in a C ++ language manner).
for (int j = 0; j <nFrames; j ++) {
Connect (B, j);
for (int i = 0; i <nFrames; i ++) {
if (i == j) continue;
Connect (A, i);
MotionEstimation ();
for (int m = 0; m <nFrames; m ++) {
Connect (P, m);
MotionCompensation ();
Overlay ();
}
}
}

  Note that the selection control unit 13 is controlled by the function Connect (s, t) so as to open and close the contact group so that only the input t is selected for the output s. NFrames is a natural number representing the number of frame points that can be output from the frame storage unit 11. In the example of FIG. 1, nFrames = 7. MotionEstimation (), MotionCompensation (), and Overlay () are functions that perform operations of a motion estimation unit 20, a motion compensation unit 30, and an image superimposition unit 40 described later, respectively.

FIG. 3 is a diagram for explaining the outline of the operation of the frame interpolation apparatus. Hereinafter, the motion estimation unit 20, the motion compensation unit 30, and the image superimposing unit 40 will be described with reference to FIG. Note that the time relationship of the frame numbers f _a , f _b , f _p , and f _q is arbitrary.

Based on the similarity between the motion estimation original image I (f _a ) and the motion estimation target image I (f _b ) input from the selection control unit 13, the motion estimation unit 20 includes the motion estimation original image I (f _a ) in the motion estimation original image I (f _a ). each region is sought corresponds to which region within the motion estimation target image I (f _b), the motion a motion vector v _n between motion estimation original image I (f _a) and the motion estimation target image I (f _b) Output to the compensation unit 30.

For example, the motion estimation unit 20 corresponds to the _n-th partial area (block area) B _n (n is a natural number where 1 ≦ n ≦ N. N is a total number of partial areas and a natural number) in the motion estimation original image. It obtains an area in motion estimation target image I (f _b), a vector v _n motion results. Shows an example of calculation of the motion vector _{v n} based on the similarity of the motion estimation original image I _{(f a)} and the motion estimation target image I _{(f b)} in the following equation (1) to (3). Here, I (f; x) is a pixel value at the image coordinate x (x is a two-dimensional vector) of the image I (f) of the frame number f.

Equation (1) minimizes the sum of absolute differences (SAD), Equation (2) minimizes the sum of square errors (SSD), and Equation (3) normalizes the cross-correlation. a calculation method of a motion vector v _n based on the maximization of; (Normalized Cross Correlation NCC).

Incidentally, as long as it is a method for evaluating the similarity of a partial region between the images may be obtained motion vector v _n by a measure other than that shown in equation (1) to (3). Although not shown in the figure, not only the motion vector v _n representing the translation between partial areas in the image, also and determining the enlargement ratio or the rotation angle, the geometry of affine transformation and projective transformation (homography) You may perform matching by conversion.

Motion compensation unit 30, the frame number f _a motion estimation original _image, based on the frame number f _p of the frame number f _b, and motion compensation based on the image motion estimation target image, the motion vector v with _n partial region B _n Is converted (typically translated) to calculate a partial region C _n and output to the image superimposing unit 40. For example, the motion compensation unit 30 calculates the partial region C _n by the following equation (4).

The motion compensation unit 30, as shown in the following equation (5), the frame number f _a motion estimation original _image, the frame number f _b of the motion estimation target _image, the frame number f _p of the motion compensation based on the _image, and interpolation by scalar multiplication of the motion vector v _n calculates a vector u _n by a multiple based on the frame time and position f _q of the image, and outputs the image superimposing section 40.

In FIG. 3 and the following description, a superscript (k) is added to identify the processing when the selection control unit 13 performs the k-th selection, and the frame number of the motion estimation source image is set to f. _a ^(k) , the frame number of the motion estimation destination image is f _b ^(k) , the frame number of the motion compensation source image is f _p ^(k) , the motion vector is v _n ^(k) , and a vector obtained by multiplying the motion vector by a scalar is u _n ^(k) , and the converted partial region is _denoted as C _n ^(k) . Here, k is an integer from 0 to K, and K is a natural number.

The image superimposing unit 40 performs motion compensation with decimal accuracy (accuracy finer than the pixel interval of the input video) based on the motion vector for the motion compensation original image I (f _p ) of a plurality of frames, and performs motion compensation with decimal accuracy. An interpolated image is generated by superimposing the performed images with phase alignment with decimal accuracy. More specifically, the image superimposing unit 40, accumulated image ^{J (k),} and for generating an interpolated image H using the weight image ^{W (k),} the integrated image ^{J (k)} and the weighted image ^{W (k) is} It has a memory to store.

Both of the resolutions of the integrated image J ^(k) and the weighted image W ^(k) are preferably the same as or higher than the resolution of the image frames constituting the input video. For the integrated image J ^(k) and the weighted image W ^(k) , the image coordinate x can take a decimal value, and the pixel values at the image coordinate x are represented by J ^(k) (x) and W ^(k) (x ). The image coordinate x in the image frame constituting the input video is assumed to correspond to the image coordinate x in the integrated image J ^(k) and the weighted image W ^(k) . For example, when the image frames constituting the input video are horizontal X pixels and vertical Y pixels, the resolution of the integrated image J ^(k) and the weighted image W ^(k) is horizontal S _X X pixels and vertical S _Y Y pixels. In some cases, the accumulated image J ^(k) and the weighted image W ^(k) are sampled in increments of 1 / S _X pixels in the horizontal direction and 1 / S _Y pixels in the vertical direction.

FIG. 4 is a flowchart showing the operation of the image superimposing unit 40. First, the image superimposing unit 40 initializes all sample point positions (image coordinates) x by setting J ⁽⁰⁾ (x) = W ⁽⁰⁾ (x) = 0 (step S101). Then, the selection control unit 13 sets a new motion estimation original image I (f _a ^(k) ), motion estimation target image I (f _b ^(k) ), and motion compensation original image I (f _p ^(k) ). Each time is selected, k is incremented by 1 (step S102).

In repeated k-th (k ≧ 1), the frame interpolation device 1, the frame number f a motion estimation original image I (f a (k)) of the (k) and frame number f b (k) of the motion estimation target image A pixel value is superimposed on the integrated image J (k−1) based on I (f b (k) ) and the motion compensation original image I (f p (k) ) of the frame number f p (k). Thus, a new integrated image J (k) is generated (step S103).

は畳み込み演算を表す。また、ｗ_ｎ ^（ｋ）は非負の重み値、ｈ（ｘ）は点拡がり関数（ＰＳＦ；Point Spread Function）を表す。 Specifically, when the selection control unit 13 performs the k-th selection (k ≧ 1), the image superimposing unit 40 performs a calculation represented by the following equation (6). In formula (6)

Represents a convolution operation. W _n ^(k) represents a non-negative weight value, and h (x) represents a point spread function (PSF).

The image superimposing section 40, the frame number f a (k) of the motion estimation original image I (f a (k)) and the motion estimation target image I frame number f b (k) (f b (k)) Based on the above, the image is superimposed on the weighted image W (k−1) to generate a new weighted image W (k) (step S104).

  Specifically, when the selection control unit 13 performs the k-th selection (k ≧ 1), the image superimposing unit 40 performs a calculation represented by the following equation (7).

The weight value w _n ^(k ) in the expressions (6) and (7) may be a constant (for example, 1), may be variable for each partial region C _n (that is, according to n), It may be variable according to the selection pattern of the selection control unit 13 (that is, according to k), or may be variable with respect to both n and k.

As a method of changing the weight value w _n ^(k) for each partial region C _n , the similarity is determined according to the optimum value (minimum value or maximum value) of the similarity of the images evaluated by the expressions (1) to (3). There is a method of increasing the weight as the degree is high (the error is small or the correlation is high).

Further, as a method of changing the weight value w n (k) in accordance with the selection pattern of the selection control unit 13, the frame number f a motion estimation original image (k), the frame number of the motion estimation target image f b (k ) , A method of setting according to the time interval of the frame number f p (k ) of the motion compensation original image and the frame time position f q of the interpolated image. For example, as shown in the following equation (8), the shorter the motion estimation frame interval | f p (k) −f q | or the motion compensation frame interval | f a (k) −f b (k) | Can be set to be large.

  The point spread function h (x) used in Expression (6) may be a two-dimensional unit impulse, or may be a function other than that. For example, a censored Sinc function or a Lanczos function can be used as the point spread function h (x). For example, based on the Lanczos-3 function L, the point spread function h (x) can be defined as the following equation (9). C is a coefficient for normalization so that the DC gain of h (x) is 1. α and β are parameters for adjusting the spread (that is, the cut-off frequency) of the point spread function. α and β are, for example, positive constants, preferably 1 or less. For example, α = β = 0.8.

The image superimposing unit 40 repeatedly performs the processing from step S102 to step S104 until k = K (step S105). The image superimposing unit 40 then repeats the interpolation image H (f ⁾ by dividing the integrated image J ^(K) by the weighted image W ^(K) for each pixel as shown in the following equation (10) at the Kth iteration. _q ) is generated and output (step S106).

The image superimposing unit 40 may convert the resolution of the image obtained by Expression (10) and output it as necessary. For example, the image superimposing unit 40 may convert the resolution of the interpolated image H (f _q ) so as to match the resolution of the image frame of the input video. The image superimposing unit 40 may further increase the resolution by interpolating the interpolated image H (f _q ) by interpolation or the like.

As described above, in the frame interpolation device 1, the frame selection unit 10 determines a plurality of sets of the motion estimation source image, the motion estimation destination image, and the motion compensation source image from the input video, and the motion estimation unit 20 performs the motion estimation. obtains the reference motion vector v _n is a motion vector between the original image and the motion estimation target image, by scalar multiplication of the reference motion vector by the motion compensation unit 30, the interpolation motion is a motion vector between the motion compensated original image and the interpolation image obtains a vector u _n, the image superimposing unit 40 for a plurality of sets of motion compensated original image interpolation motion by the vector u _n by superimposing a plurality of motion compensation image subjected to motion compensation in decimal precision to generate an interpolated image.

  For this reason, according to the frame interpolation device 1, since the pixel values of the pixels of the plurality of reference frames affect one pixel of the interpolation frame, the contour of the subject of the interpolation frame becomes smooth. In addition, when the reference frame includes aliasing distortion in the spatial direction, the phase rotation of the aliasing distortion component due to the alignment is canceled out by superimposing a plurality of frames, and the jaggy of the contour is reduced and the high frequency component is reduced. A sharp contour image including can be obtained. This sharpening of the contour image can achieve the same effect with the multi-frame super-resolution technology that restores the high-frequency component by aligning a plurality of frames. While a sharpened image is generated at a certain frame time in a frame, the method according to the present invention not only includes the time points included in a plurality of overlapped frames but also the time points not included (may be a fractional frame time point). It is upward compatible in that a sharpened image can be generated.

  Here, the image superimposing unit 40 weights the motion compensation image according to the similarity when calculating the motion vector, and the time position of the motion estimation source image, the motion estimation destination image, the motion compensation source image, and the interpolation image. It is preferable to superimpose. This makes it possible to reflect the degree of reliability according to the similarity of motion estimation and the time interval between the image used to calculate the interpolated image and the interpolated image by weighting. Is possible.

  In addition, it is preferable that the image superimposing unit 40 generates an interpolated image by superimposing the motion compensated image after performing a convolution operation using a point spread function. As a result, even if the sample position does not correspond completely as a result of motion compensation, it is possible to compensate the value by interpolation. In addition, if an appropriate design is made according to the pixel interval of the output image of the spread of the point spread function (for example, standard deviation) (for example, the standard deviation is about 0.5 pixels), (when the input image includes aliasing distortion) Can be sharpened by the super-resolution effect.

  In addition, it is preferable that the image superimposing unit 40 performs the superimposing process by scaling the image onto an image having a resolution higher than the resolution of the input video. As a result, an interval of one pixel on the high-resolution image corresponds to an interval of less than one pixel on the input video, so that a superimposition process with a decimal accuracy when the resolution of the input video is used as a reference is realized. Can do. Further, when the superimposition processing with decimal precision is performed on an image that is the same as or less than the resolution of the input video, a digital filter process for performing phase shift with decimal precision is necessary, whereas the present invention relates to When the superimposition process is performed on an image having a resolution higher than the resolution of the input video as in the method, the digital filter process may be omitted. Furthermore, according to the method of the present invention, since the image obtained as a result of the superimposition process is higher than the resolution of the input video, it is possible to concurrently perform image enlargement processing with higher image quality than simple interpolation.

  In addition, the image superimposing unit 40 may superimpose a motion compensated image and then convert the resolution converted image as an interpolated image as necessary. When the resolution is lowered, an effect of oversampling is produced, so that aliasing distortion hardly occurs in the interpolated image. Further, when the resolution is increased, the required memory capacity can be reduced as compared with the case where the resolution is increased without performing the superimposition process.

  It should be noted that a computer can be suitably used to cause the above-described frame interpolation device 1 to function, and such a computer stores a program describing processing contents for realizing each function of the frame interpolation device 1 in the storage of the computer. The program can be realized by reading out and executing the program by the CPU of the computer.

  Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiments can be combined into one or divided.

  Thus, according to the present invention, a high-quality interpolated image can be generated from a plurality of reference images, which is useful for any application for generating an interpolated image.

DESCRIPTION OF SYMBOLS 1 Frame interpolation apparatus 10 Frame selection part 11 Frame memory | storage part 12 Input / output switching part 121,122,123 Row of matrix switches 13 Selection control part 20 Motion estimation part 30 Motion compensation part 40 Image superimposition part 100,200 Motion estimation original image 101 , 201 Motion estimation destination image 102, 202 Motion compensation original image 103, 203 Time position of interpolated image 300 Interpolated image

Claims (5)

A frame interpolator for generating an interpolated image of an input video,
A frame selection unit that determines a plurality of sets of a motion estimation source image, a motion estimation destination image, and a motion compensation source image from an input video;
A motion estimation unit for obtaining a reference motion vector that is a motion vector between the motion estimation source image and the motion estimation destination image for each of the sets ;
A motion compensation unit that multiplies the reference motion vector by a scalar to obtain an interpolation motion vector that is a motion vector between the motion compensation source image and the interpolation image corresponding to the reference motion vector ;
An image superimposing unit that generates the interpolated image by superimposing a plurality of motion compensated images that have been subjected to motion compensation with decimal accuracy using the interpolated motion vector for the plurality of sets of motion compensated original images;
Equipped with a,
The image superimposing unit, depending on the degree of similarity when calculating the motion vector, frame interpolation device characterized that you superimposed by weighting the motion compensation image in each partial area. A frame interpolator for generating an interpolated image of an input video,
A frame selection unit that determines a plurality of sets of a motion estimation source image, a motion estimation destination image, and a motion compensation source image from an input video;
A motion estimation unit for obtaining a reference motion vector that is a motion vector between the motion estimation source image and the motion estimation destination image for each of the sets;
A motion compensation unit that multiplies the reference motion vector by a scalar to obtain an interpolation motion vector that is a motion vector between the motion compensation source image and the interpolation image corresponding to the reference motion vector;
An image superimposing unit that generates the interpolated image by superimposing a plurality of motion compensated images that have been subjected to motion compensation with decimal accuracy using the interpolated motion vector for the plurality of sets of motion compensated original images;
With
The image superimposing unit, the motion compensated image frame interpolation device you and generates the interpolation image superimposed after convolution with the point spread function. A frame interpolator for generating an interpolated image of an input video,
A frame selection unit that determines a plurality of sets of a motion estimation source image, a motion estimation destination image, and a motion compensation source image from an input video;
A motion estimation unit for obtaining a reference motion vector that is a motion vector between the motion estimation source image and the motion estimation destination image for each of the sets;
A motion compensation unit that multiplies the reference motion vector by a scalar to obtain an interpolation motion vector that is a motion vector between the motion compensation source image and the interpolation image corresponding to the reference motion vector;
An image superimposing unit that generates the interpolated image by superimposing a plurality of motion compensated images that have been subjected to motion compensation with decimal accuracy using the interpolated motion vector for the plurality of sets of motion compensated original images;
With
The image superimposing section, frame interpolation device you characterized in that a high-resolution image than the resolution of the input image to the motion-compensated image. A frame interpolator for generating an interpolated image of an input video,
A frame selection unit that determines a plurality of sets of a motion estimation source image, a motion estimation destination image, and a motion compensation source image from an input video;
A motion estimation unit for obtaining a reference motion vector that is a motion vector between the motion estimation source image and the motion estimation destination image for each of the sets;
A motion compensation unit that multiplies the reference motion vector by a scalar to obtain an interpolation motion vector that is a motion vector between the motion compensation source image and the interpolation image corresponding to the reference motion vector;
An image superimposing unit that generates the interpolated image by superimposing a plurality of motion compensated images that have been subjected to motion compensation with decimal accuracy using the interpolated motion vector for the plurality of sets of motion compensated original images;
With
The image superimposing unit, after superimposing the motion compensation image, frame interpolation device you and generates the interpolation image by converting the resolution. The program for functioning a computer as a frame interpolation apparatus as described in any one of Claim 1 to 4 .

Images