JP2014086813A - Frame interpolation device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve picture quality of an interpolation frame by performing high-precision motion compensation even when a subject in an input video image performs movement other than constant velocity linear motion.SOLUTION: A frame interpolation device 1 comprises: a frame selection section 11 which selects a motion estimation source frame and two or more motion estimation destination frames from an input frame group; a collation section 13 which determines a first motion vector with respect to each of the motion estimation destination frames from the motion estimation source frame; a parameter estimation section 14 which substitutes a nonlinear function to a plurality of first motion vectors obtained by the collation section 13 to determine a second motion vector with respect to an interpolation frame from a motion compensation source frame selected from the input frame group; and a motion compensation section 15 which performs motion compensation on the motion compensation source frame on the basis of the second motion vector to generate an image of the interpolation frame.

Description

  The present invention relates to a frame interpolating apparatus that generates an interpolated frame image with reference to a plurality of frames in an input frame group and interpolates the frames in a time direction, and a program therefor.

  Conventionally, as a method of generating an interpolation frame by interpolating frames in the time direction, a method of presenting any frame before or after the interpolation target time as an interpolation frame a plurality of times is known. However, in the method of presenting the same frame multiple times, the movement of the video becomes awkward.

  Therefore, there are known methods that use an average or weighted average of frames before and after the interpolation target time as an interpolation frame, and methods that generate motion vectors from the frames before and after the interpolation target time and generate the interpolation frame by performing motion compensation. (For example, refer to Patent Document 1).

Japanese Patent No. 3495485

  However, in the conventional method using the average of the previous and subsequent frames or the weighted average or the conventional method using motion compensation as described in Patent Document 1, it is assumed that the subject in the video is moving at a constant linear velocity. Since the motion compensation is performed under the object, there is a problem that the motion becomes awkward in a subject that performs nonlinear motion such as acceleration motion.

  The object of the present invention made in view of such circumstances is that even if the subject in the input video moves other than the constant-velocity linear motion, high-precision motion compensation can be performed, and the image quality of the interpolation frame is improved. It is an object of the present invention to provide a frame interpolating apparatus and a program thereof that can be performed.

  In order to solve the above-described problem, a frame interpolation device according to the present invention is a frame interpolation device that generates an interpolation frame image with reference to a plurality of frames in an input frame group, and includes a motion estimation source frame from the input frame group. A frame selection unit that selects two or more motion estimation destination frames, a collation unit that obtains a first motion vector from each of the motion estimation source frames to each of the motion estimation destination frames, and a plurality of frames obtained by the collation unit A parameter estimation unit that applies a nonlinear function to the first motion vector and obtains a second motion vector from the motion compensation source frame selected from the input frame group to the interpolation frame; and A motion compensation unit that generates an interpolated frame image by performing motion compensation based on the second motion vector.

  Furthermore, in the frame interpolation apparatus according to the present invention, the parameter estimation unit performs the fitting of the nonlinear function by error minimization.

  Furthermore, in the frame interpolating apparatus according to the present invention, the parameter estimation unit uses the nonlinear function as a second-order or higher-order polynomial, and linearly solves the fitting problem due to the square error minimization, thereby solving analytically. .

  Furthermore, in the frame interpolation apparatus according to the present invention, the parameter estimation unit defines the nonlinear function by a spline curve or a Bezier curve.

  Furthermore, in the frame interpolation device according to the present invention, the image superimposing unit, the motion compensation unit, and the second motion vector associate a plurality of pixels of the motion compensation source frame with the same position of the interpolation frame. In this case, the average value of the values of the plurality of pixels is used as the pixel value of the interpolation frame.

  Furthermore, in the frame interpolating device according to the present invention, a value of a pixel that is not associated with a pixel of the motion compensation source frame in the interpolation frame is set as a weighted sum of one or more input frames at the pixel position. Features.

  In order to solve the above problem, a program according to the present invention causes a computer to function as the frame interpolation device.

  According to the present invention, even when the subject in the input video moves other than the constant-velocity linear motion, highly accurate motion compensation can be performed, and the image quality of the interpolation frame can be improved.

It is a block diagram which shows the structural example of the frame interpolation apparatus which concerns on one Embodiment of this invention. It is a figure for demonstrating the process example of the collation part in the frame interpolation apparatus which concerns on one Embodiment of this invention. It is a figure for demonstrating the 1st process example of the parameter estimation part in the frame interpolation apparatus which concerns on one Embodiment of this invention. It is a figure for demonstrating the 2nd processing example of the parameter estimation part in the frame interpolation apparatus which concerns on one Embodiment of this invention.

  Hereinafter, an embodiment of a frame interpolation device according to the present invention will be described in detail with reference to the drawings. In this specification, the input frame group (input video) is I, the frame of the frame number f is denoted as I (f), and the pixel value at the image coordinates (x, y) of the frame I (f) is denoted by I (f , X, y). Also, the generated interpolation frame is J, the frame of frame number f is written as J (f), and the pixel value at the image coordinates (x, y) of the interpolation frame J (f) is J (f, x, y). . The frame time position is represented by a frame number.

  FIG. 1 is a block diagram showing a configuration example of a frame interpolation apparatus according to an embodiment of the present invention. As illustrated in FIG. 1, the frame interpolation device 1 includes a frame selection unit 11, a scanning unit 12, a plurality of matching units 13, a parameter estimation unit 14, and a motion compensation unit 15. In the embodiment shown in FIG. 1, the following description will be given assuming that three verification units 13 (13-1 to 13-3) are provided.

  The frame interpolation device 1 refers to a plurality of frames in the input frame group I and generates an interpolation frame image J (f) at the frame time position f. The frame time position f may be an integer time position or a non-integer time position. For example, f = 3.25 indicates a time position obtained by internally dividing the time position of the frame of frame number 3 and the time position of the frame of frame number 4 into 1: 3.

The frame selection unit 11 selects and stores a motion estimation source frame and two or more motion estimation destination frames from the input frame group I. The frame selection unit 11 shown in FIG. 1 selects and stores the motion estimation source frame I (a) and the three motion estimation destination frames I (b ₁ ), I (b ₂ ), and I (b ₃ ). The frame numbers a, b ₁ , b ₂ , and b ₃ may be in any order as long as they are different from each other.

Since the scanning unit 12 sequentially sets partial regions on the motion estimation source frame I (a), the k-th scanning step (k is an integer _satisfying 0 ≦ k <S _x · S _y , S _x and S _y are scanning) Are natural numbers representing the number of sample points in the horizontal and vertical directions), and sequentially generate coordinates (p _k , q _k ). Then, the coordinates (p _k , q _k ) are output to the collation unit 13 and the motion compensation unit 15. For example, the scanning unit 12 sequentially outputs coordinates (p _k , q _k ) represented by the following formula (1).

は、ｋをＳ_ｘで除したときの値を超えない最大の整数を表す。また、Ｗ_ｘ及びＷ_ｙはフレームＩ（ａ）上における水平方向及び垂直方向の標本化間隔を表す自然数である。さらに、（Ｐ_０，Ｑ_０）はフレームＩ（ａ）上における水平方向及び垂直方向の標本化の始点の画像座標である。 In equation (1), k% S _x represents the remainder when k is divided by S _x . Also,

Represents the largest integer that does not exceed the value obtained by dividing k by S _x . W _x and W _y are natural numbers representing sampling intervals in the horizontal direction and the vertical direction on the frame I (a). Further, (P ₀ , Q ₀ ) is the image coordinates of the sampling start point in the horizontal direction and the vertical direction on the frame I (a).

For example, when the scanning unit 12 raster scans a frame I (a) of 1920 pixels × 1080 pixels at an interval of 16 horizontal pixels and 8 vertical pixels to sample 120 horizontal points and 135 vertical points starting from the upper left origin. S _x = 120, S _y = 135, W _x = 16, W _y = 8, P ₀ = 0, and Q ₀ = 0.

The matching unit 13 refers to the motion estimation source frame and the motion estimation destination frame to obtain a motion vector V ₁ (first motion vector) from the motion estimation source frame to each motion estimation destination frame, and sends the motion estimation source frame to the parameter estimation unit 14. Output. In the embodiment shown in FIG. 1, the motion estimation source frame referred to by each matching unit 13-s (s is a natural number equal to or smaller than S; S is a natural number equal to or larger than 2; S = 3 in the example of FIG. 1) is the frame I (a ), The motion estimation destination frames are all different frames I (b _s ). In the following, focusing on one collating unit 13, the operation will be described assuming that the motion estimation source frame is I (a) and the motion estimation destination frame is I (b).

FIG. 2 is a diagram for explaining the processing contents of the collation unit 13, and shows the motion estimation source frame I (a) and the motion estimation destination frame as I (b ₁ ), I (b ₂ ), I (b ₃ ). Is shown. A partial region B _k shown in FIG. 2 is a region set based on the coordinates (p _k , q _k ) designated by the scanning unit 12. In this embodiment, (p _k , q _k ) is set as the upper left coordinate. A rectangular area having a width A _x and a height A _y is set. At this time, the partial region B _k is expressed by the following equation (2).

The collation unit 13 performs association between the motion estimation source frame and the motion estimation destination frame for each partial region _Bk . That is, the collation unit 13 searches the motion estimation source frame I (a) for a region similar to the pixel value pattern of the partial region _Bk of the motion estimation source frame I (a) in the motion estimation destination frame I (b). And a motion vector V ₁ (a, b) = (u _k (a, b), v _k (a, b)) for performing association between the motion estimation destination frames I (b). For example, when the collation unit 13 performs collation based on the minimization of the sum of absolute differences (SAD), the motion vector V ₁ (a, b) = (u _k (a, b), v _k (a, b)) is obtained by the following equation (3). Here, a region D represents a search range of motion vectors.

  Note that the collation unit 13 may, for example, minimize a sum of squared differences (SSD), maximize a cross-correlation (including a normalized cross-correlation), region, instead of minimizing the sum of absolute differences. The motion vector can be obtained by any matching method such as minimizing the Bhattacharrya distance between the pixel value histograms. Further, the matching unit 13 may perform filter processing such as edge detection, smoothing, noise removal, and feature extraction before matching.

The parameter estimation unit 14 includes horizontal components of a plurality of motion vectors V ₁ (a, b _s ) = (u _k (a, b _s ), v _k (a, b _s )) obtained by the collation unit 13-s. And non-linear functions g (θ; t) and h (θ; t) are applied to the vertical component, and for frame interpolation from the motion compensation source frame selected from the input frame group I to the interpolation frame J (c). A motion vector V ₂ (second motion vector) is calculated and output to the motion compensation unit 15. θ is a parameter for changing the shape of the nonlinear function, and t represents time (frame).

FIG. 3 is a diagram for explaining the processing contents of the parameter estimation unit 14, and shows the motion estimation source frame I (a) and the motion estimation destination frame as I (b ₁ ), I (b ₂ ), I (b ₃ ) And an interpolation frame J (c). In FIG. 3, the motion compensation source frame is a motion estimation source frame I (a), and a motion vector V ₂ (a, c) = (for frame interpolation from the motion compensation source I (a) to the interpolation frame J (c) = ( In this example, (u ^ _k (a, c), v ^ _k (a, c)) is obtained.

First, the parameter estimation unit 14 _adds a nonlinear function g (θ; t), h () to the motion vector V ₁ (a, b _s ) = (u _k (a, b _s ), v _k (a, b _s )). θ; t) is applied by error minimization such as square error minimization or absolute value error minimization to obtain a prediction curve P. For example, in the case of performing the fitting by minimizing the square error, the parameter θ ^ _k of the fitting result is derived by the following equation (4). In addition, when fitting by absolute value error minimization, the parameter θ ^ _k of the fitting result is derived by the following equation (5).

  For the fitting of equation (4), for example, a non-linear least square method by iterative calculation can be used. For example, the Levenberg-Marquardt method can be used as the non-linear least square method by known iterative calculation.

Note that the parameter estimation unit 14 linearly solves the fitting problem by minimizing the square error by analytically analyzing the nonlinear functions g (θ; t) and h (θ; t) by using second-order or higher-order polynomial functions. Can be solved. In this case, an N-order function (N is a natural number of 2 or more) and an M-order function (M is a natural number of 2 or more) shown in the following equation (6) can be used as the nonlinear function. Here, α _k ⁽ⁿ⁾ and β _k ⁽ⁿ⁾ are elements constituting the parameter θ _k .

  At this time, the fitting of the equation (6) can be solved analytically by the following equation (7) without performing repetitive calculation if S ≧ N and S ≧ M.

は、α_ｋ ^（１），α_ｋ ^（２），…，α_ｋ ^（Ｎ），β_ｋ ^（１），β_ｋ ^（２），…，β_ｋ ^（Ｍ）の最適化結果である。 here,

^{_{Is, α k (1), α}} k (2), ..., α k (N), β k (1), β k (2), ..., the optimization results of β _k ^(M).

Alternatively, when the nonlinear functions g (θ; t) and h (θ; t) are defined as in the following equation (8) and the parameter θ is defined as in the following equation (9), the following equation (10 ), The parameter θ ^ _k of the fitting result can be derived analytically.

In addition, the parameter estimation unit 14 converts the nonlinear function g and the nonlinear function h into a sequence of points (u _k (a, b _s ), v _k (a, b _s )) (s = 1, 2,..., S). May be defined as a Bezier curve or a spline curve that passes through.

Subsequently, the parameter estimation unit 14 substitutes the time c to be interpolated into the nonlinear function in which the optimized parameters are set, so that the motion vector for frame interpolation V ₂ (a, c) = (u _k (a , C), v ^ _k (a, c)). That is, the parameter estimation unit 14 performs the calculation shown in the following equation (11).

Motion compensation unit 15, motion compensation by the motion compensation based on the frame motion vector V ₂ partial region B _k for frame interpolation (in this embodiment I (a)), to generate an image of the interpolation frame J (c) . For example, as represented by the following equation (12), the motion compensation unit 15 _converts the pixel value pattern of each partial region B _k of the motion compensation source frame I (a) into a motion vector V ₂ (a, c) = (u ^ _k (a, c), v ^ _k (a, c)) is moved and pasted on the interpolation frame J (c).

Here, if the paste operation is performed for all k by the calculation of Expression (12), the paste operation may be performed repeatedly at the same pixel position depending on the motion vector V ₂ for frame interpolation. . In this case, according to Expression (12), the pixel pasted last is reflected in the image of the interpolation frame J (c). On the other hand, with respect to pixels on which the pasting operation has been performed in duplicate, the average value of the pasting results can be used as the pixel value of the interpolation frame J (c). In this case, the motion compensation unit 15 _converts the pixel value pattern of each partial region _Bk of the motion compensation source frame I (a) to the motion vector V ₂ for frame interpolation as represented by the following equation (13). The calculation is performed based on the above and superimposed on the interpolation frame J (c). Here, K is the total number S _x · S _y of the partial regions B _k .

Q ^(k ) in Expression (13) is a buffer (image) for calculating the sum of the pixel values for each pixel position for pixels whose regions overlap each other at the time of pasting. R ^(k) is a buffer for counting how many times the pasting operation has been performed at each pixel position. After pasting operation for all K partial areas, the result Q ^(K-1) is divided by R ^{(K-1) for} each pixel position, thereby obtaining an average value of a plurality of pasting results. Get.

The image P in Expression (13) is an image for filling a region in which the pixel value pattern of the motion compensation source frame I (a) has never been superimposed on the interpolation frame J (c). For example, as shown in the following equation (14), it may be the same as a certain input frame (here, motion compensation source frame I (a)), or a plurality of input frames may be weighted as shown in the following equation (15). It can be the sum (here, weight w _s ).

In the above-described embodiment, as illustrated in FIG. 3, the motion compensation unit 15 has been described as generating the image of the interpolation frame J (c) using the input frame I (a) as the motion compensation source frame. However, the motion compensation source frame subject to motion compensation is not limited to the input frame I (a). For example, as shown in FIG. 4, the motion compensation unit 15 converts the corresponding partial region B _k on the input frame I (b _s ) (example of s = 2 in FIG. 4) by the motion vector V ₂ for frame interpolation. An image of the interpolation frame J (c) may be generated by performing motion compensation.

As described above, in the frame interpolation device 1, the frame selection unit 11 first selects a motion estimation source frame and two or more motion estimation destination frames from the input frame group. Next, the matching unit 13 determines a motion vector V ₁ from the motion estimation source frame to each motion estimation destination frame. Then, the parameter estimation unit 14 applies a non-linear function to the plurality of motion vectors V ₁ obtained by the collation unit 13 and performs motion for frame interpolation from the motion compensation source frame selected from the input frame group I to the interpolation frame. determine the vector _{V 2.} Finally, the motion compensation unit 15, motion compensation to generate an interpolated frame image based on the motion compensation based on a frame the motion vector V ₂ for frame interpolation. Therefore, according to the frame interpolation device 1 of the present invention, it is possible to perform highly accurate motion compensation even when the subject in the input video moves other than the uniform linear motion, and the image quality of the interpolation frame It becomes possible to improve.

  It should be noted that a computer can be suitably used to cause the above-described frame interpolation device 1 to function, and such a computer stores a program describing processing contents for realizing each function of the frame interpolation device 1 in the storage of the computer. The program can be realized by reading out and executing the program by the CPU of the computer.

  Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiments can be combined into one or divided.

  As described above, according to the present invention, since the image quality of the interpolation frame can be improved, it is useful for any application for generating the interpolation frame of the input frame group.

DESCRIPTION OF SYMBOLS 1 Frame interpolation apparatus 11 Frame selection part 12 Scan part 13 Collation part 14 Parameter estimation part 15 Motion compensation part

Claims (7)

A frame interpolation device that generates an interpolation frame image with reference to a plurality of frames in an input frame group,
A frame selection unit that selects a motion estimation source frame and two or more motion estimation destination frames from an input frame group;
A matching unit for obtaining a first motion vector from each of the motion estimation source frame to each of the motion estimation destination frames;
A parameter estimation unit that applies a non-linear function to the plurality of first motion vectors obtained by the collation unit to obtain a second motion vector from the motion compensation source frame selected from the input frame group to the interpolation frame;
A motion compensation unit configured to generate an interpolated frame image by performing motion compensation on the motion compensation source frame based on the second motion vector;
A frame interpolation apparatus comprising:   The frame interpolation apparatus according to claim 1, wherein the parameter estimation unit performs the fitting of the nonlinear function by error minimization.   3. The frame interpolating apparatus according to claim 2, wherein the parameter estimation unit uses the nonlinear function as a second-order or higher-order polynomial, and linearly solves the fitting problem due to square error minimization to solve it analytically.   The frame interpolation device according to claim 1, wherein the parameter estimation unit defines the nonlinear function by a spline curve or a Bezier curve.   When the plurality of pixels of the motion compensation source frame are associated with the same position of the interpolation frame by the second motion vector, the motion compensation unit calculates an average value of the plurality of pixels as an interpolation frame. The frame interpolating apparatus according to claim 1, wherein the frame interpolating apparatus has a pixel value of the following.   The motion compensation unit is characterized in that, in the interpolation frame, a value of a pixel to which a pixel of the motion compensation source frame is not associated is set as a weighted sum of one or more input frames at the pixel position. Item 6. The frame interpolation device according to any one of Items 1 to 5.   A program for causing a computer to function as the frame interpolating device according to any one of claims 1 to 6.

Images