JP2012100099A - Frame rate conversion apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize high picture quality in the time direction by making it possible to obtain a dynamic picture image changing smoothly with a few motion blur.SOLUTION: Time alignment means 5 calculates time deviation amount δt maximizing similarity, by comparing pixel time series of a designated pixel (G(t;x,y))and pixel time series of a peripheral pixel (G(t;ξ,η)), and lets the δt be time movement amount Δt. Interpolation means 6 generates a continuous function g(t;x,y) and a continuous function g(t;ξ,η) by interpolating the pixel time series of the designated pixel (G(t;x,y))and the pixel time series of the peripheral pixel (G(t;ξ,η))in the time direction respectively, and generates an interpolation value F(t;x,y) by synthesizing the continuous function of the designated pixel g(t;x,y) and the continuous function shifted by the time movement amount Δt in the time direction g(t+Δt;ξ,η) by means of a simple average or the like. Thereby, a dynamic picture image changing smoothly with a few motion blur can be obtained.

Description

  The present invention relates to an apparatus and program for converting a frame rate of a moving image, and more particularly to a technique for interpolating a frame in a time direction.

  Conventionally, various techniques have been proposed as techniques for converting the frame rate of a moving image. For example, there is a technique of converting an original moving image into a high frame rate moving image by repeatedly using the same frame a plurality of times. There is also a method of converting to a low frame rate moving image by thinning out frames at a predetermined interval.

  In addition, a technique has been proposed in which an interpolation frame is generated by performing temporal or spatio-temporal linear filtering on an input image and converted to a high frame rate moving image (see Patent Document 1). Specifically, this technique is based on the two input images according to the time distance between the generated interpolation frame and the frames of the two input images existing before and after the interpolation frame. An interpolation frame is generated by apportioning the pixel values in this frame.

  Also, a method has been proposed in which attention is paid to the movement of a subject in a moving image and improvement in image quality is realized by using this movement information (see Patent Document 2). Specifically, this method tracks the movement of a subject in a moving image between frames, and generates an interpolation frame by block matching using movement information.

  Furthermore, a method for reducing calculation resources when converting the frame rate of a moving image has been proposed (see Patent Document 3). Specifically, this method decodes encoded data of a moving image frame into differential encoded data, and interpolates interpolation data inserted between frames of differential encoded data based on a motion vector of the differential encoded data. It generates, converts the frame rate of differentially encoded data based on this interpolation data, and restores the moving image by decoding the differentially encoded data after conversion.

JP 2009-100033 A JP-A-61-26382 JP 2009-21963 A

  However, in the method of repeatedly using the same frame a plurality of times and the method of thinning out the frames at a predetermined interval, the subject moving at a constant speed in the moving image becomes non-uniform or the effect of time aliasing distortion occurs. For this reason, these methods have a problem that the movement of the subject becomes awkward.

  Further, in the method of performing linear filter processing on a frame in time or spatio-temporal direction according to Patent Document 1, the spatial resolution of the moving region in the generated interpolation frame is reduced. For this reason, this method has a problem that blurring occurs. In addition, when the input image frame and the interpolation frame are mixedly output, or when the temporal phase of the interpolation frame changes with respect to the input image frame, temporal image quality fluctuations occur. For this reason, this method has a problem that a moving image by an input image frame and an interpolation frame is perceived as flicker.

  Also, the block matching method using motion information according to Patent Documents 2 and 3 can output a moving image with smoother and less flicker. However, when tracking of the movement of the subject fails, image degradation occurs at the boundary between motion and static. Such image degradation is conspicuous when an object outline, intense movement or illumination fluctuation occurs, and distortion is unevenly distributed temporally or spatially, resulting in an unnatural image.

  Accordingly, the present invention has been made to solve the above-described problems, and an object of the present invention is to provide a frame rate capable of obtaining a smoothly changing moving image with little motion blur and realizing high image quality in the time direction. It is to provide a conversion device and a program.

  In order to achieve the above object, the invention of claim 1 is a frame rate conversion device for converting a frame rate of a moving image, wherein a moving image storage means storing a moving image composed of frames at a plurality of points in time, and A pixel time-series reading unit that reads out a pixel value at a predetermined pixel position as designated pixel time-series data from a frame included in a predetermined time range in the moving image stored in the moving image storage unit; and the moving image storage unit Peripheral pixels for reading out, as peripheral pixel time-series data, pixel values existing at peripheral positions within a predetermined range with reference to the designated pixel position from a frame included in a predetermined time range in the moving image stored in the means A time-series reading unit; a designated pixel time-series data read by the pixel time-series reading unit; and a reading by the peripheral pixel time-series reading unit. A time alignment unit that obtains a time position that is most similar to the peripheral pixel time-series data and outputs the time position as a time movement amount; and designated pixel time-series data read by the pixel time-series reading unit; Alternatively, any one of the peripheral pixel time-series data read by the peripheral pixel time-series reading unit is moved by the time movement amount output by the time alignment unit, and the time series after the movement is performed Interpolation means for obtaining a pixel value at a desired time as an interpolation value based on the data and the non-moving time-series data.

  According to the first aspect of the present invention, since the time change of the pixel values of the peripheral pixels is used in addition to the time change of the pixel value of the designated pixel, it is possible to obtain a smoothly changing moving image with little motion blur. Possible frame rate conversion can be realized. In addition, the conventional block matching is based on the two-dimensional search, but the time-series data is only aligned by the one-dimensional search, so the processing is simplified and the calculation speed can be increased. Become.

  The invention of claim 2 is the frame rate conversion apparatus according to claim 1, wherein the time alignment means interpolates the designated pixel time-series data read by the pixel time-series reading means in the time direction. Interpolate to generate a continuous function of designated pixels having continuous pixel values in the time direction, and interpolate peripheral pixel time-series data read by the peripheral pixel time-series reading means in the time direction, Generating a continuous function of peripheral pixels having continuous pixel values, and when either one of the continuous function of the specified pixel or the continuous function of the peripheral pixel is shifted in the time direction, the continuous function of the specified pixel and A time shift amount having the most similar continuous function of the surrounding pixels is obtained, and the time shift amount is output as a time shift amount.

  According to the second aspect of the present invention, time alignment is performed with continuous functions having temporally continuous pixel values, so that the time with higher accuracy than the time alignment between the original discrete pixel time-series data. The amount of movement can be obtained.

  Further, the invention of claim 3 is the frame rate conversion apparatus according to claim 1, wherein the time alignment means is the designated pixel time-series data read by the pixel time-series reading means or the peripheral pixel time. Any one of the peripheral pixel time-series data read by the series reading means is interpolated in the time direction to generate a continuous function having continuous pixel values in the time direction, the generated continuous function, Alternatively, when any one of the time-series data in which the continuous function is not generated is shifted in the time direction, a time shift amount in which the continuous function and the time-series data are most similar is obtained, and the time shift amount is calculated. It is output as a time movement amount.

  According to the invention of claim 3, since the time alignment is performed between the continuous function having temporally continuous pixel values and the discrete pixel time-series data, the original discrete pixel time-series data can be compared with each other. Thus, it is possible to obtain a time movement amount with higher accuracy than the time alignment in FIG.

  According to a fourth aspect of the present invention, there is provided the frame rate conversion apparatus according to the first aspect, wherein the time alignment means is the designated pixel time-series data read by the pixel time-series reading means or the peripheral pixel time. When any one of the surrounding pixel time-series data read by the series reading means is shifted every natural number times the time interval of the frame, the designated pixel time-series data and the surrounding pixel time-series data Between which the evaluation value indicating a discrete error for each natural number multiple of the time interval of the frame is minimized, and a predetermined value having a temporally continuous value in a predetermined time range including the time Error between the plurality of continuous functions and the discrete evaluation value, respectively, specifying a continuous function that minimizes the error, obtaining an extreme value of the identified continuous function, and calculating the time of the extreme value And outputs as the time movement amount.

  According to the invention of claim 4, since the evaluation value is obtained from discrete time-series data, high-speed processing is possible. In addition, since the discrete evaluation values are interpolated and continuous in the time range including the time when the evaluation value is the minimum, it is possible to obtain a time movement amount with high accuracy.

  According to a fifth aspect of the present invention, in the frame rate conversion apparatus according to the fourth aspect, the time alignment means specifies a continuous function that minimizes the error by parabolic fitting or least square fitting. And

  According to the fifth aspect of the present invention, discrete evaluation values are interpolated and made continuous by parabolic fitting or least square fitting in a time range including the time at which the evaluation value is minimized. This makes it possible to obtain a highly accurate amount of time movement.

  According to a sixth aspect of the present invention, there is provided a frame rate conversion program for causing a computer to function as the frame rate conversion device according to any one of the first to fifth aspects.

  As described above, according to the present invention, it is possible to obtain a smoothly changing moving image with less motion blur. Accordingly, it is possible to realize frame rate conversion capable of obtaining a high-quality moving image in the time direction.

It is a block diagram which shows the whole structure of the frame rate conversion apparatus by embodiment of this invention. It is a figure explaining the concept of a spatiotemporal volume. It is a figure explaining operation which extracts the pixel time series of a specific pixel position from a spatiotemporal volume. It is a figure explaining the process of a time alignment means. It is a flowchart which shows the 1st process by a time alignment means. It is a flowchart which shows the 2nd process by a time alignment means. It is a flowchart which shows the 1st process by an interpolation means. It is a flowchart which shows the 2nd process by an interpolation means.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the overall configuration of a frame rate conversion apparatus according to an embodiment of the present invention. This frame rate conversion apparatus 100 includes an input moving image storage means 1, a counting means 2, a pixel time series reading means 3, a peripheral pixel time series reading means 4, a time alignment means 5, an interpolation means 6, a pixel time series writing. Means 7 and output moving image storage means 8 are provided.

A moving image is stored in the input moving image storage unit 1. This moving image is G, its pixel position is (x, y), and its pixel value at time t is G (t; x, y). Here, both the horizontal pixel position x and the vertical pixel position y are represented by integers of 0 or more. The size of the image is assumed to be X pixels in the horizontal direction and Y pixels in the vertical direction. Therefore, the image coordinates (x, y) are x∈ {0, 1,..., X−1} and y∈ {0, 1,. The time t that is the processing target (interpolation target or collation target to be described later) in the moving image is the sum of tε {t ₀ , t ₁ ,..., T _K−1 } It is assumed that the time point is K and the order is t ₀ <t ₁ <... <T _K−1 . The time sequence (t ₀ , t ₁ ,..., T _K-1 ) is referred to as an input moving image time sample point sequence (t _k ) _{kε {0,1,.} The moving image stored in the input moving image storage means 1 can be randomly accessed as a three-dimensional space-time volume.

  FIG. 2 is a diagram for explaining the concept of the spatiotemporal volume, and shows the moving images stored in the input moving image storage means 1. The spatiotemporal volume 10 of the moving image is composed of pixel values arranged in an array at each pixel position and each time. The spatio-temporal volume 10 shown in FIG. 2 includes a plurality of objects 12, 13, 14, etc., the object 12 is stationary with respect to the time axis t, the object 13 moves to the left, It can be seen that 14 is moving to the right.

2 is a cross-sectional view of the spatio-temporal volume 10 cut along a plane parallel to the xy plane at the time t = t _A , and the frame 15 shown in the upper right of FIG. at the time of t _B, it is a sectional view taken along a plane parallel space-time volume 10 in the xy plane. The frame 11, the pixel position of y = _{y 0,} there is an object 12-1,13-1,14-1, the frame 15, the pixel position of y = _{y 0,} the object 12-2,13 -2, 14-2 exists. In addition, when a plane parallel to the xt plane or a plane parallel to the yt plane is cut, the moving object appears as a texture having an inclination. The texture 16 shown in the lower right of FIG. 2 is a view obtained by cutting the texture 16 at a pixel position y = y _{0 by} a plane parallel to the xt plane, and the objects 12-3, 13-3, and 14− on the x axis. A movement tendency of 3 appears. It can be seen that the objects 12-3, 13-3, and 14-3 exist at positions on the x-axis shown in FIG. 2 at the time point t = t _A. Then, at the time t = t _B when the time has elapsed from the time t = t _A , the position of the object 12-3 does not change on the x axis, and the object 13-3 exists at the position moved to the left, It can be seen that the object 14-3 exists at the position moved to the right.

  The input moving image storage means 1 is hardware such as a memory and a hard disk, and can be randomly accessed in both time t and space (x, y).

  Returning to FIG. 1, the counting means 2 is a counter that generates coordinate values for scanning the pixel position (t; x, y) in the space (x, y) of the moving image G at time t. The order and pattern of scanning are arbitrary. For example, scanning takes a scanning line in the horizontal direction at time t, scans pixels from left to right, and sequentially generates pixel positions (t; x, y) to scan the scanning lines from top to bottom. You may make it do. The counting means 2 outputs the generated pixel position (t; x, y) to the pixel time series reading means 3, the peripheral pixel time series reading means 4, and the pixel time series writing means 7.

FIG. 3 is a diagram for explaining an operation of extracting a pixel time series at a specific pixel position from the spatiotemporal volume 10 shown in FIG. In the frame 23 at the time t = t _A and the frame 24 at the time t = t _B , the images of the designated pixel positions 21-1, 21-2 and the peripheral pixel positions 22-1, 22-2 exist, respectively. A pixel time-series waveform 26 shown at the lower right of FIG. 3 is a pixel value waveform at a designated pixel position 21 in the matching target time range [t _A , t _B ] in the texture 20 shown at the lower left of FIG. This is the time series G (t; x, y) of the designated pixel read by the pixel time series reading means 3. Also, the pixel time-series waveform 25 shown in the lower right of FIG. 3 is a pixel at the peripheral pixel position 22 in the search target time range [t _A + t _C , t _B + t _D ] in the texture 20 shown in the lower left of FIG. It is a value waveform, and is a time series G (t; ξ, η) of peripheral pixels read by the peripheral pixel time series reading means 4 described later.

The time alignment means 5 to be described later compares the pixel time-series waveform 25 (black circle on the lower right in FIG. 3) of the peripheral pixel position 22 in the search target time range [t _A + t _C , t _B + t _D ]. A time range similar to the pixel time-series waveform 26 (white circle at the lower right in FIG. 3) at the designated pixel position 21 in the time range [t _A , t _B ] is searched. A set of time sample points within the verification target time range is set as U, and a set of time sample points within the search target time range is set as S. Note that t _C and t _D are time search ranges in the past and future directions, and preferably t _C <0 and t _D > 0 are set.

Returning to FIG. 1, the pixel time-series reading unit 3 inputs the pixel position (t; x, y) from the counting unit 2, and the moving image at the pixel position (t; x, y) from the input moving image storage unit 1. _{Read out} the pixel time series (G (t; x, y)) _tεU of the image G. The pixel time series reading unit 3 outputs the time series G (t; x, y) of the designated pixel to the time alignment unit 5 and the interpolation unit 6.

The peripheral pixel time-series reading unit 4 receives the pixel position (t; x, y) from the counting unit 2 and receives the pixel position (t; x, y) (designated pixel position (x, y) from the input moving image storage unit 1. y)) as a reference, the pixel time series (G (t; ξ, η)) _tεS of the moving image G at the peripheral pixel position (ξ, η) is read out.

  The peripheral pixel at the designated pixel position (x, y) is a pixel in the image at a position other than the designated pixel position (x, y). Typically, the pixel exists in the vicinity where the distance from the designated pixel position (x, y) is equal to or smaller than a predetermined threshold, for example, a pixel adjacent to the designated pixel position (x, y). The peripheral pixel may be a single pixel or a plurality of pixels. For example, a plurality of pixels adjacent to the designated pixel position (x, y) and pixels sequentially selected from the plurality of pixel positions may be used as the peripheral pixels.

For example, the peripheral pixel time-series reading unit 4 has the position (x-1, y), (x + 1, y), (x, y-1) and (x, y + 1) near the space 4 of the designated pixel position (x, y). ), One pixel position is sequentially set to the peripheral pixel position (ξ, η), and the pixel time series of the moving image G (G (t; ξ, η)) _tε from the input moving image storage means 1. _S may be read out. The peripheral pixel time series reading unit 4 outputs the time series G (t; ξ, η) of the peripheral pixels to the time alignment unit 5 and the interpolation unit 6.

[Time alignment means]
The time alignment means 5 receives the time series G (t; x, y) of the designated pixel from the pixel time series reading means 3 and also the time series G (t; ξ) of the peripheral pixels from the peripheral pixel time series reading means 4. , Η) and the pixel time series of the specified pixel (G (t; x, y)) _t∈U and the pixel time series of the surrounding pixels (G (t; ξ, η)) _t∈S. Perform time alignment in the time direction. The time alignment is an operation for obtaining a time movement amount Δt by translating one of the pixel time series in the time direction so that the waveforms of the two pixel time series overlap most.

FIG. 4 is a diagram for explaining the processing of the time alignment means 5. As shown in FIG. 4, the time alignment means 5 uses the pixel time series waveform 26 of the designated pixel as the pixel time series (G (t; x, y)) _t∈U as a reference. Sequence (G (t; ξ, η)) The pixel time-series waveform 25 with _tεS is shifted in the time direction to obtain the parallel movement amount Δt when the two waveforms overlap most (pixel after time alignment in FIG. 4) (See time-series waveform 27). In the following description, the time alignment means 5 translates the pixel time series (G (t; ξ, η)) _tεS of the peripheral pixels in the time direction.

Both pixel time series (G (t; x, y)) _tεU , (G (t; ξ, η)) _tεS are discrete with respect to time. Hereinafter, an alignment method that can be applied to both the unequal interval and the equal interval, and an alignment method that can be applied when the pixel time series time interval is an equal interval will be described. The time interval of the pixel time series being unequal means that the time intervals of the frames constituting the spatiotemporal volume 10 of the moving image shown in FIG. 2 are not the same. In addition, the time intervals of the pixel time series being equal intervals means that the time intervals of the frames are the same.

(Alignment method applicable to the case where the frame of the spatio-temporal volume 10 is an unequal time interval or an equal time interval)
FIG. 5 is a flowchart showing the first processing (alignment method applicable to both unequal time intervals and equal time intervals) by the time alignment means 5. The time alignment means 5 receives the time series G (t; x, y) of the designated pixel from the pixel time series reading means 3 and also the time series G (t; ξ) of the peripheral pixels from the peripheral pixel time series reading means 4. , Η) is input (step S501). Then, the time alignment means 5 performs the pixel time series (G (t; x, y)) _t∈U of the designated pixel and the pixel time series (G (t; ξ, η)) _{t∈S of the} surrounding pixels. Interpolate in the time direction. A pixel time series (G (t; x, y)) of a designated pixel _Let the interpolation result of _{tεU be} a continuous function g (t; x, y) for time t. Similarly, the interpolated result of the pixel time series (G (t; ξ, η)) _tεS of the peripheral pixels is set as a continuous function g (t; ξ, η) for time t (step S502). For the interpolation, for example, primary interpolation or cubic interpolation can be used.

尚、連続関数ｇ（ｔ；ξ，η）は、前記式（１）において、ｘ及びｙをξ及びηにそれぞれ置き換えたものである。また、ｋは整数である。 In the case of linear interpolation, the time alignment means 5 performs interpolation according to the following formula to generate a continuous function g (t; x, y).

The continuous function g (t; ξ, η) is obtained by replacing x and y with ξ and η in the above equation (1). K is an integer.

ここで、

は床関数を示す。 In particular, when the time series G (t; x, y) is time-sampled at unit time intervals at an integer time point t, the expression (1) becomes the following expression.

here,

Indicates the floor function.

  The time alignment means 5 performs time alignment in the time direction between the continuous function g (t; x, y) of the designated pixel and the continuous function g (t; ξ, η) of the surrounding pixels. For example, the time alignment means 5 has a continuous function g (t + δt; in which the continuous function g (t; x, y) of the designated pixel and the continuous function g (t; ξ, η) of the surrounding pixels are shifted by δt in the time direction. (ξ, η) are compared (step S503), the degree of similarity between them is calculated, and a time shift amount δt that maximizes the degree of similarity is obtained (step S504). Then, the time alignment means 5 outputs the time shift amount δt as the time movement amount Δt (step S505).

For example, when an absolute value error is used as the similarity evaluation scale, the time alignment unit 5 obtains a time shift amount δt that minimizes the following evaluation value m, and sets this as a time movement amount Δt.

Further, for example, when using normalized correlation for the similarity evaluation scale, the time alignment means 5 uses the following equations instead of the equations (3) and (4) to calculate the time movement amount Δt. Ask.

  In addition, you may discretize the calculation of Formula (3)-(6) regarding time. In this case, the integral operation of the equations (3) and (5) is replaced with the sum operation.

  In addition, the time alignment unit 5 selects the continuous function g (t; of the designated pixel from the pair of the continuous function g (t; x, y) of the designated pixel and the continuous function g (t + δt; ξ, η) of the surrounding pixels. Instead of x, y), a discrete time series G (t; x, y) with respect to time may be used.

  Further, the time alignment means 5 includes a peripheral pixel continuous function g (t + δt; out of a pair of a continuous function g (t; x, y) of the designated pixel and a continuous function g (t + δt; ξ, η) of the peripheral pixel. Instead of ξ, η), a discrete time series G (t + δt; ξ, η) may be used.

As described above, according to the first processing in the time alignment unit 5, the continuous function g is obtained by interpolating the pixel time series (G (t; x, y)) _tεU of the designated pixel in the time direction. (T; x, y) is generated, and similarly, the continuous time function g (t; ξ) is _obtained by interpolating the pixel time series (G (t; ξ, η)) _tεS of the surrounding pixels in the time direction. , Η), and a continuous function g (t + δt; ξ, η) in which the continuous function g (t; x, y) of the designated pixel and the continuous function g (t; ξ, η) of the surrounding pixels are shifted by δt in the time direction. ), A time shift amount δt that maximizes the similarity between them is obtained, and this time shift amount δt is output as a time shift amount Δt. As a result, the time interval of the pixel time series G (t; x, y)) _tεU and (G (t; ξ, η)) _tεS is either an unequal time interval or an equal time interval. However, the time movement amount Δt can be obtained at a time interval shorter than the time interval (frame time interval) (at a continuous time). That is, the amount of time movement Δt with a finer accuracy than the time interval of the original discrete pixel time series G (t; x, y)) _t∈U , (G (t; ξ, η)) _t∈S is obtained. Can be sought. In the conventional block matching method using motion information, spatial alignment is performed by a two-dimensional search, whereas in the first processing in the time alignment means 5, time is determined by a one-dimensional search. Since the alignment is performed, the processing is simplified and the calculation speed can be increased.

(Alignment method applicable when the frames of the spatio-temporal volume 10 are equally spaced)
FIG. 6 is a flowchart showing the second processing (alignment method applicable in the case of equal time intervals) by the time alignment means 5. The time alignment means 5 obtains the time movement amount Δt without performing the interpolation of the pixel time series waveform as in the equations (1) and (2). In the following description, the sample point on the time axis is assumed to be a unit time interval (one step interval).

  The time alignment means 5 receives the time series G (t; x, y) of the designated pixel from the pixel time series reading means 3 and also the time series G (t; ξ) of the peripheral pixels from the peripheral pixel time series reading means 4. , Η) is input (step S601). The time alignment means 5 then functions G (t + δt; ξ) by shifting the time series G (t; x, y) of the designated pixel and the time series G (t; ξ, η) of the neighboring pixels by δt in the time direction. , Η) is quantified for each δt. Here, δt is a natural number times the time interval of the frames constituting the spatiotemporal volume 10 of the moving image.

For example, when an absolute value error is used for the similarity evaluation scale, the time alignment means 5 uses the following evaluation value m (δt) as δtε {t _C , t _{C + 1} , t _{C + 2} ,. _{It is obtained in} the range of _D−1 , t _D } (step S602). Then, δt that minimizes the evaluation value m (δt) is obtained, and this is set as time s (step S603).

  Then, the time alignment means 5 performs continuous curve fitting on the discrete graph of the evaluation value m in the time set R including the time s (near the time s). For example, a parabola is used for the curve (this is called parabolic fitting). When parabolic fitting is used, the time set R has at least three elements.

  The time set R may be composed of the time s and a predetermined number of time groups before and after the time s (for example, R = {s−1, s, s + 1}, R = {s−2, s−1, s, s + 1, s + 2}, etc.).

とおける。 Here, let the curve be a function f _p (t) related to t. The function f _p (t) is a continuous function having continuous values with respect to the time axis. p is a parameter that changes the shape of the curve. The parameter p may be a scalar or a vector. For example, when parabolic fitting is used, the function f _p (t) is expressed by the following equation.

You can.

Then, in the time range R, the time alignment means 5 compares the continuous function f _p (t) of the equation (9) with the discrete evaluation value m (t) (step S604), and the evaluation value m ( The parameter p is obtained so that the error between t) and the continuous function f _p (t) in the equation (9) is minimized (step S605).

For example, when the least square fitting is used, the parameter p is obtained by the following equation.

In particular, when parabolic fitting is used, the parameter p is obtained by the following equation.

Then, the time alignment means 5 minimizes the curve y = f _p (t) based on the obtained parameter p (when a measure that is better evaluated as a smaller value, such as an absolute value error or a square error, is used as an evaluation measure of m) ) Or maximizing (when a scale such as a normalized correlation value that has a better evaluation is used as an evaluation scale for m) is obtained (step S606), and this is output as a time shift amount Δt (step S607). ). That is, the time alignment means 5 obtains the extreme value of the curve y = f _p (t) by the obtained parameter p, and outputs t at the extreme value as the time movement amount Δt.

尚、時間位置合わせ手段５は、曲線ｙ＝ｆ_ｐ（ｔ）を最大化するｔを求める場合、前記式（１２）の「ａｒｇ ｍｉｎ」を「ａｒｇ ｍａｘ」に置き換えた式により、時間移動量Δｔを求める。 For example, when the time alignment means 5 obtains t that minimizes the curve y = f _p (t), it obtains the time movement amount Δt by the following equation.

When the time alignment means 5 obtains t that maximizes the curve y = f _p (t), the time shift amount is calculated by an expression in which “arg min” in the expression (12) is replaced with “arg max”. Δt is obtained.

となる。 In particular, when parabolic fitting is used, the time alignment means 5 obtains the time movement amount Δt by the following equation.

It becomes.

Further, when parabolic fitting for R = {s−1, s, s + 1} is used, the time alignment means 5 obtains the time movement amount Δt by the following equation.

Thus, according to the second process in the time alignment means 5, the pixel time series (G (t; x, y)) _{tεU of} the designated pixel and the pixel time series (G (t; ξ, η)) _{Indicates an error} with respect to a pixel time series (G (t + δt; ξ, η)) _tεS obtained by shifting _tεS by δt (natural number times the time interval of the frame) in the time direction. An evaluation value m (δt) is obtained, and a discrete evaluation value m (t) and a continuous function f _p (t) are obtained in a time range including a time shift amount δt at which the evaluation value m (δt) is minimized. A parameter p that minimizes the error between them is obtained, a time t for minimizing or maximizing the function f _p (t) is obtained, and this time t is output as a time movement amount Δt. As a result, when the time intervals of the pixel time series G (t; x, y)) _tεU , (G (t; ξ, η)) _tεS are equal time intervals, the time is shorter than the time interval. In the interval (at successive times), the time movement amount Δt can be obtained. That is, the amount of time movement Δt with a finer accuracy than the time interval of the original discrete pixel time series G (t; x, y)) _t∈U , (G (t; ξ, η)) _t∈S is obtained. Can be sought. In addition, time alignment is performed by a one-dimensional search, and a discrete pixel time series G (t; x, y)) _tεU and (G (t; ξ, η)) _tεS is used. Since the evaluation value m (δt) is obtained, the processing is simplified and the calculation speed can be increased.

[Interpolation means]
Returning to FIG. 1, the interpolation means 6 performs the pixel time series (G (t; x, y)) _t∈U of the designated pixel and the pixel time series (G (t; ξ, η)) _{t of the} surrounding pixels. _{Based on εS} and the time movement amount Δt, an interpolation value F (t; x, y) of the pixel value at the time t at the pixel position (x, y) is estimated. The interpolated value F (t; x, y) is the pixel time series (G (t; x, y)) _t∈U of the designated pixel and the pixel time series (G (t; ξ, η)) of the surrounding pixels. It can be defined as a sample point at time t on a smooth waveform that approximates both waveforms of a pixel time series (G (t + Δt; ξ, η)) in which _tεS is shifted by a time movement amount Δt.

(A method to generate a continuous function by interpolation and obtain an interpolation value)
FIG. 7 is a flowchart showing a first process (a technique for generating a continuous function by interpolation and obtaining an interpolation value) by the interpolation means 6. The interpolation means 6 receives the time series G (t; x, y) of the designated pixel from the pixel time series reading means 3 and the time series G (t; ξ, η) of the peripheral pixels from the peripheral pixel time series reading means 4. ) And a time movement amount Δt are input from the time alignment means 5 (step S701). Then, the interpolation means 6 interpolates the pixel time series (G (t; x, y)) _t∈U of the designated pixel in the time direction, and _{converts it} into a continuous function g (t; x, y) for the time t. Conversion is performed (step S702). For example, in the case of linear interpolation, a continuous function g (t; x, y) is generated by the following equation.

Similarly, the interpolating means 6 interpolates the pixel time series (G (t; ξ, η)) _t∈S of the peripheral pixels in the time direction, and continuously functions g (t; ξ, η) with respect to time t. (Step S702). For example, in the case of linear interpolation, a continuous function g (t; ξ, η) is generated by the following equation.

  Then, the interpolation means 6 calculates the waveform of the continuous function g (t; x, y) of the designated pixel generated by the equation (15) and the continuous function g () of the surrounding pixels generated by the equation (16). The waveform of the continuous function g (t + Δt; ξ, η) obtained by shifting the waveform of t; ξ, η) by Δt in the time direction is synthesized (step S703), and the interpolated value F (t; x, y) is obtained. To the pixel time-series writing means 7 (step S704).

In the above description, the pixel positions (ξ, η) of the peripheral pixels have been formulated at one place, but there may be a plurality of pixel positions of the peripheral pixels. Hereinafter, it is assumed that there are M pixel positions of peripheral pixels (M is a natural number), and the m-th pixel position (m is an integer of 0 or more and less than M) is (ξ _m , η _m ). In addition, the time movement amount Δt at the pixel position (ξ _m , η _m ) of the m-th peripheral pixel is Δt _m and the set S of time sample points within the search target time range is S _m .

As a synthesis method for obtaining the interpolated value F (t; x, y), for example, a simple average is used. The interpolation means 6 calculates | requires the interpolation value F (t; x, y) by the following formula | equation which shows a simple average.

ここで、ｗ_ｍ（ｔ）は、時刻ｔにおける周辺画素位置（ξ_ｍ，η_ｍ）に対する重み係数である。尚、ｗ_−１（ｔ）は、時刻ｔにおける指定画素位置（ｘ，ｙ）に対する重み係数である。 For example, a weighted average is used as a synthesis method for calculating the interpolated value F (t; x, y). The interpolation means 6 calculates | requires the interpolation value F (t; x, y) by the following formula | equation which shows a weighted average.

Here, w _m (t) is a weighting factor for the peripheral pixel position (ξ _m , η _m ) at time t. Note that w ₋₁ (t) is a weighting coefficient for the designated pixel position (x, y) at time t.

The weighting factor w _m (t) may be determined according to the distance between the designated pixel position (x, y) and the peripheral pixel position (ξ _m , η _m ). For example, the weighting factor w _m (t) is expressed by the following equation using the function α (d) relating to the distance d (the example of the equation (19) is invariant with respect to the time t).

ここで、λは、距離ｄに対する関数値α（ｄ）の減衰の速さを決めるための減衰係数であり、λが大きいほど減衰が緩やかになる。例えば、λ＝１とする。 The function α (d) is preferably a decreasing function with respect to the distance d. That is, the function alpha (d), when the distance d is short, the weighting factor w _{m (t)} increases, the distance d is longer is a function, such as the weighting factor w _{m (t)} becomes smaller. For example, it is represented by the following formula.

Here, λ is an attenuation coefficient for determining the speed of attenuation of the function value α (d) with respect to the distance d, and the attenuation becomes gentler as λ increases. For example, λ = 1.

このｅ_ｍ（ｔ）は、位置合わせをした時系列と注目時刻ｔとの間の最小時間距離である。 The weighting factor w _m (t) may be determined according to the time distance between the time sequence (t _k + Δt _m ) _kεSm or (t _k ) _kεU and the time t. First, the time distance e _m (t) is defined as follows, for example.

This e _m (t) is the minimum time distance between the aligned time series and the attention time t.

The weighting factor w _m (t) is expressed by the following equation using the function β (e) regarding the time distance e (the example of the equation (22) is invariant with respect to the spatial position).

ここで、μは、時間距離ｅに対する関数値β（ｅ）の減衰の速さを決めるための減衰係数であり、μが大きいほど減衰が緩やかになる。例えば、μ＝１とする。 The function β (e) is preferably a decreasing function with respect to the time distance e. That is, the function β (e) is a function such that the weighting factor w _m (t) increases when the time distance e is short, and the weighting factor w _m (t) decreases when the time distance e is long. For example, it is represented by the following formula.

Here, μ is an attenuation coefficient for determining the speed of attenuation of the function value β (e) with respect to the time distance e, and the attenuation becomes gentler as μ increases. For example, μ = 1.

ここで、γ（α，β）は、α及びβの値を合成するための関数であり、好ましくはαに関して増加関数、βに関して増加関数とする。例えば、以下の式で表される。

Further, the weighting factor w _{m (t)} may also be determined according to both the distance of the spatial distance d _m and the time distance e _m. For example, the weight coefficient w _m (t) is expressed by the following equation.

Here, γ (α, β) is a function for synthesizing the values of α and β, and is preferably an increase function for α and an increase function for β. For example, it is represented by the following formula.

Further, the weight coefficient w _m (t) may be determined in consideration of the minimum (or maximum) evaluation value when the time movement amount Δt is obtained in the equation (12). For example, as shown in the following expression, the minimum (or maximum) evaluation value when calculated time shift amount Delta] t _m of pixel positions around the pixel (xi] _m, eta _m) and r _m. At this time, (as weight as the maximum evaluation value _{r m} is greater coefficient _w m (t) increases) as the minimum evaluation value _{r m} is smaller as the weighting factor _w m (t) increases, the weighting factor _{w m} (T) may be determined.

(Method to obtain the interpolated value by restoration processing based on the degradation model)
FIG. 8 is a flowchart showing a second process (a technique for obtaining an interpolated value by a restoration process based on a deterioration model) by the interpolating means 6. The interpolation means 6 receives the time series G (t; x, y) of the designated pixel from the pixel time series reading means 3 and the time series G (t; ξ, η) of the peripheral pixels from the peripheral pixel time series reading means 4. ) And a time movement amount Δt from the time alignment means 5 (step S801).

Here, an observation model (time sampling model) D (D is a functional) from the interpolation value F to be obtained to the pixel time series G is defined as follows.

As the observation model D, for example, a model obtained by modeling motion blur is used. If the camera shutter time (or image sensor accumulation time) when an image (frame) at time t _k is taken is τ _k , the motion blur observation model D is expressed by the following equation.

ここで、Ｖは、最適化を行う時刻の範囲である。例えば、時刻範囲Ｖは、注目時刻ｔ近傍の時間区間（例えば、時刻ｔとその前後１秒の時間区間）とする。 The provisional result of the interpolation value F (t; x, y) that is the interpolation result is Φ (t; x, y). The interpolation means 6 evaluates whether or not the temporary result Φ matches the time series G (t; x, y) of the designated pixel by using the error evaluation functional J. The interpolation means 6 performs trial and error (for example, optimization by the steepest descent method) of the temporary result Φ so that the evaluation value J [Φ] is minimized by the following formula, and the temporary result Φ The optimization result is obtained as an interpolation value F.

Here, V is a time range for performing optimization. For example, the time range V is a time interval in the vicinity of the attention time t (for example, a time interval of time t and 1 second before and after the time t).

  Specifically, the interpolation means 6 calculates an error (designated pixel error) between the time series G (t; x, y) of the designated pixel and the value of the observation model D, and the time series of the surrounding pixels. The error between the time series G (t + Δt; ξ, η) obtained by shifting G (t; ξ, η) by the time movement amount Δt and the value of the observation model D is calculated, and the total error (peripheral pixels) of all peripheral pixels is calculated. The error of the designated pixel and the error of the surrounding pixels are added, and the addition result is summed in the time range V for optimization to obtain the evaluation value J [Φ] (step S802). . Then, the interpolation means 6 calculates a provisional result Φ that minimizes the evaluation value J [Φ] (step S803), and uses the provisional result Φ as the interpolation value F (t; x, y) for pixel time. The data is output to the series writing means 7 (step S804).

[Pixel time series writing means]
Returning to FIG. 1, the pixel time-series writing unit 7 inputs the pixel position (t; x, y) from the counting unit 2, and receives the interpolation value F (t; x, y) from the interpolation unit 6. The time sequence (t ₀ , t ₁ ,..., T _L−1 ) (output moving image time sample point sequence () is input by changing the write address according to the pixel position (t; x, y). T _k ) _{k∈ {0,1,..., L-1}} .) _Interpolated value sequence (F (T _k ; x, y)) _{k∈ {0,1,.} Is stored in the output moving image storage means 8. As a result, the output moving image storage unit 8 stores a spatio-temporal volume 10 of a moving image having a frame rate converted as compared to the moving image stored in the input moving image storage unit 1. Here, time T _k is the k-th time sample point, and L is the total number of sample points in the time sequence.

Note that the time T _k may be an integer value or a non-integer value. For example, the output video time sample point sequence (T _k ) _{kε {0,1,..., L-1} is changed} to the input video time sample point sequence (t _k ) _{kε {0,1,. If} it is finer than _K-1} , frame rate upsampling is realized. Conversely, the output video time sample point sequence (T _k ) _{kε {0,1,..., L-1}} is _replaced with the input video time sample point sequence (t _k ) _{kε {0,1,. , K-1}, the} frame rate downsampling can be realized. Further, the time interval of the output video time sample point sequence (T _k ) _{kε {0,1,..., L-1}} and the input video time sample point sequence (t _k ) _{kε {0,1, ·} The time intervals of _K−1} may be equal intervals, or one or both may be unequal intervals. Also, the time interval of the output video time sample point sequence (T _k ) _{kε {0,1,..., L-1}} and the input video time sample point sequence (t _k ) _{kε {0,1, · When} the time intervals of _K−1} are both equal, the ratio of the frame rates of both may be a rational number ratio or an irrational ratio. Furthermore, the output video time sample point sequence (T _k ) _{kε {0,1,..., L-1}} and the input video time sample point sequence (t _k ) _{kε {0,1,. −1}} phase may be aligned, shifted, or may change with time.

The output moving image storage means 8 stores a spatio-temporal volume 10 of the processed moving image. In the spatiotemporal volume 10 of the stored moving image, the pixel time series at the pixel position (t; x, y) is (F (T _k ; x, y)) _{kε {0,1,. }} .

  The output moving image storage means 8 is hardware such as a memory and a hard disk, and can be randomly accessed for both time t and space (x, y).

As described above, according to the frame rate conversion apparatus 100 according to the embodiment of the present invention, the time alignment unit 5 performs the pixel time series (G (t; x, y)) _tεU and the pixel time series of surrounding pixels (G (t; ξ, η)) _tεS are compared to obtain a time shift amount δt that maximizes the similarity, and this is moved over time. It was made to output as quantity (DELTA) t. Further, in the first process, the interpolating means 6 interpolates the pixel time series (G (t; x, y)) _tεU of the designated pixel in the time direction so that the continuous function g (t; x, y) and similarly, a continuous function g (t; ξ, η) is obtained by interpolating the pixel time series (G (t; ξ, η)) _t∈S in the time direction of the surrounding pixels. , And a continuous function g (t + Δt; ξ,) in which the continuous function g (t; x, y) of the designated pixel and the continuous function g (t; ξ, η) of the surrounding pixels are shifted in the time direction by the time movement amount Δt. η) is synthesized by a simple average or the like to obtain an interpolated value F (t; x, y). In addition, the interpolation means 6 performs an error and time shift between the pixel time series (G (t; x, y)) _{tεU of} the designated pixel and the value of the observation model D in the second process. Pixel time series of neighboring pixels shifted by the amount Δt (G (t + Δt; ξ, η)) _An evaluation value J [Φ] is obtained based on an error between _t∈S and the value of the observation model D, and the evaluation value J [ The provisional Φ that minimizes Φ] is calculated, and the interpolated value F (t; x, y) is obtained.

Accordingly, by taking a temporal correlation with respect to a plurality of pixel positions, the pixel time series (G (t; x, y)) of the _designated pixel is the pixel time series (G (t ; Ξ, η)) A time shift Δt at _tεS is obtained, and a pixel time series of the designated pixel (G (t; x, y)) A pixel time series of peripheral pixels shifted by _tεU from the time shift Δt ( Since G (t + Δt; ξ, η)) _t∈S is used to generate the interpolated value F (t; x, y), a moving image with little motion blur and smoothly changing in time Can be obtained. Accordingly, it is possible to realize frame rate conversion capable of obtaining a high-quality moving image in the time direction.

  Further, according to the frame rate conversion apparatus 100 according to the embodiment of the present invention, the time alignment unit 5 performs local time series matching between pixel positions instead of performing local texture matching (block matching or the like) between frames. By performing the above, the registration using the spatio-temporal correlation of moving images was performed. For this reason, unlike the block matching that performs a two-dimensional search, a one-dimensional search may be performed, which simplifies the processing and reduces the calculation cost. In addition, block noise is not generated because processing is performed in units of pixel positions without performing block division within a frame. In addition, the time correlation calculation between pixels can be regarded as the time axis and the space axis of the multiple-resolution super-resolution technique for performing registration are interchanged. That is, the frame rate conversion apparatus 100 can realize high image quality in the one-dimensional time direction as compared with the conventional multiple-resolution super-resolution technique that realizes high image quality in a two-dimensional space. Therefore, image degradation (mainly blur) when converting the frame rate hardly occurs, and high-quality frame rate conversion can be realized.

  The frame rate conversion apparatus 100 according to the embodiment of the present invention can be used for video conversion between television systems having different frame rates, telecine conversion, and the like. Moreover, since an arbitrary time sample point can be set, it can also be used for dynamic frame rate control according to the video content.

  As a hardware configuration of the frame rate conversion apparatus 100 according to the embodiment of the present invention, a normal computer can be used. The frame rate conversion apparatus 100 is configured by a computer including a volatile storage medium such as a CPU and a RAM, a non-volatile storage medium such as a ROM, an interface, and the like. Input video storage means 1, counting means 2, pixel time series reading means 3, peripheral pixel time series reading means 4, time alignment means 5, interpolation means 6, pixel time series writing provided in the frame rate conversion apparatus 100 Each function of the means 7 and the output moving image storage means 8 is realized by causing the CPU to execute a program describing these functions. These programs can also be stored and distributed in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

DESCRIPTION OF SYMBOLS 1 Input moving image memory | storage means 2 Counting means 3 Pixel time series reading means 4 Peripheral pixel time series reading means 5 Time alignment means 6 Interpolation means 7 Pixel time series writing means 8 Output moving image storage means 10 Space-time volume 11, 15, 23, 24 Frame 12, 12-1, 12-2, 12-3, 13, 13-1, 13-2, 13-3, 14, 14-1, 14-2, 14-3 Object 16, 20 Texture 21, 21-1, 21-2 Designated pixel position 22, 22-1, 22-2 Peripheral pixel position 23, 24 Frame 25 Pixel time-series waveform of peripheral pixel 26 Pixel time-series waveform of designated pixel 27 Time alignment Later pixel time-series waveform 100 Frame rate conversion device

Claims (6)

In a frame rate conversion device for converting the frame rate of a moving image,
Moving image storage means storing moving images composed of frames at a plurality of time points;
Pixel time-series reading means for reading out the pixel value at a pixel position designated in advance as designated pixel time-series data from a frame included in a predetermined time range in the moving image stored in the moving image storage means;
Pixel values present at peripheral positions within a predetermined range with reference to the specified pixel position from a frame included in a predetermined time range in the moving image stored in the moving image storage means Peripheral pixel time-series readout means for reading out as
A time position where the specified pixel time-series data read by the pixel time-series reading means and the peripheral pixel time-series data read by the peripheral pixel time-series reading means are most similar is obtained, and the time position is determined as a time. A time alignment means for outputting as a movement amount;
Either the designated pixel time-series data read by the pixel time-series reading means or the peripheral pixel time-series data read by the peripheral pixel time-series reading means is output by the time alignment means. Interpolating means for obtaining a pixel value at a desired time as an interpolated value based on the time-series data after the movement and the time-series data that has not been moved.
A frame rate conversion apparatus comprising: The frame rate conversion apparatus according to claim 1,
The time alignment means interpolates the specified pixel time-series data read by the pixel time-series reading means in the time direction, and generates a continuous function of the specified pixels having pixel values continuous in the time direction; The peripheral pixel time-series data read by the peripheral pixel time-series reading means is interpolated in the time direction to generate a continuous function of peripheral pixels having pixel values continuous in the time direction, and the continuous function of the designated pixel Alternatively, when any one of the continuous functions of the peripheral pixels is shifted in the time direction, a time shift amount in which the continuous function of the designated pixel and the continuous function of the peripheral pixels are most similar is obtained, and the time shift amount is calculated. A frame rate conversion device characterized in that the frame rate conversion device outputs the amount of time movement. The frame rate conversion apparatus according to claim 1,
The time alignment means sets one of the designated pixel time-series data read by the pixel time-series reading means and the peripheral pixel time-series data read by the peripheral pixel time-series reading means as time. Interpolating in the direction, generating a continuous function having continuous pixel values in the time direction, and generating either the generated continuous function or time series data in which the continuous function is not generated in the time direction A frame rate conversion device characterized in that when the shift is performed, a time shift amount that most closely resembles the continuous function and the time-series data is obtained, and the time shift amount is output as a time shift amount. The frame rate conversion apparatus according to claim 1,
The time alignment means, either the designated pixel time series data read by the pixel time series reading means or the peripheral pixel time series data read by the peripheral pixel time series reading means, When the natural time multiple of the time interval of the frame is shifted, a discrete error between the specified pixel time-series data and the peripheral pixel time-series data for each natural number of the time interval of the frame. Obtaining a time at which the evaluation value shown is minimum, and obtaining an error between a plurality of predetermined continuous functions having a temporally continuous value and the discrete evaluation value in a predetermined time range including the time, A frame rate conversion apparatus characterized by identifying a continuous function that minimizes the error, obtaining an extreme value of the identified continuous function, and outputting the time of the extreme value as a time movement amount. In the frame rate conversion device according to claim 4,
The frame rate conversion apparatus characterized in that the time alignment means specifies a continuous function that minimizes the error by parabolic fitting or least square fitting.   A frame rate conversion program for causing a computer to function as the frame rate conversion device according to any one of claims 1 to 5.

Images