JP2005204075A - Device and method for detecting motion vector, frame interpolation apparatus using the same, frame interpolation method using the method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate the image signal of a virtual frame by precisely detecting the motion vector of a pixel in the virtual frame which is positioned between the m-th frame and the n-th frame. <P>SOLUTION: A motion vector setting part 112 sets the motion vector MV0. A motion vector calculator 113 assumes that the motion vector MV0 is the motion vector of the pixel in the virtual frame, determines positions in the m-th and n-th frames corresponding to the pixel, performs a gradient method arithmetic processing through the use of a pixel block with the determined positions as reference, and obtains a motion vector va1. A vector adder 114 generates a motion vector MVa1 by adding the motion vector MV0 to the motion vector va1. In the same way, a motion vector calculator 115 generates a motion vector va2. A vector adder 116 adds the motion vector MV0 to the motion vector va1, and generates the motion vector MVc of the pixel in the virtual frame. The image signal of the virtual frame is generated by performing interpolation through the use of the motion vector MVc and the image signals of the m-th and n-th frames. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a motion vector detection device, a frame interpolation device using the same, a motion vector detection method, a frame interpolation method using the same, a program, and a recording medium. Specifically, the first motion vector is set to assume the motion vector of the pixel on the virtual frame, the positions on the mth frame and the nth frame corresponding to the pixel on the virtual frame are determined, and the determined first Gradient method arithmetic processing is performed using pixel blocks based on positions on the m-th frame and n-th frame to obtain a second motion vector that is a displacement of motion from the first motion vector. Through the addition of the second motion vector, the motion vector of the pixel on the virtual frame is obtained. In addition, based on the obtained motion vector of the pixel on the virtual frame, the image signal of the mth frame and the nth frame is used to perform an interpolation process to generate the image signal of the virtual frame.

  Conventionally, when performing frame rate conversion in which a frame frequency, which is the number of frames per unit time, is set to a desired frame frequency, a frame interpolation device is used. FIG. 10 shows a configuration of a conventional frame interpolation device 50.

  The motion vector detection unit 51 of the frame interpolation device 50 detects the motion vector MVa using the input digital image signal DSa, and supplies the detected motion vector MVa to the motion vector allocation unit 52. The motion vector allocation unit 52 allocates motion vectors to pixels on the virtual frame to be interpolated using the motion vector MVa, and supplies the allocated motion vector MVb to the image interpolation unit 53. The image interpolation unit 53 generates an image signal of the virtual frame using the allocated motion vector MVb.

  Here, a gradient method is known as one of representative methods used for motion vector detection. Hereinafter, the gradient method will be described.

  In a moving image, the coordinate value in the horizontal direction is “x”, the coordinate value in the vertical direction is “y”, and the coordinate value in the time direction is “t”, which is expressed by coordinates (x, y, t). Let the luminance value of the pixel be g (x, y, t).

  When the image corresponding to a certain pixel (x0, y0, t0) moves by (dx, dy, dt) during a minute time, the gradients in the horizontal, vertical, and time directions are gx (x0, y0, t0), gy, respectively. When expressed as (x0, y0, t0), gt (x0, y0, t0), the luminance value of the pixel after the transition can be expressed as shown in Equation (1) using Taylor expansion approximation.

  Here, when a certain pixel of interest in the moving image moves by horizontal vx and vertical vy after one frame (hereinafter referred to as (vx, vy)), the following expression (2) is obtained.

This expression (2) is changed from expression (1) to expression (3).

  Since the equation (3) is a two-variable equation of vx and vy, the solution cannot be obtained alone. Therefore, considering the surrounding area of the pixel of interest as one block and assuming the same movement (vx, vy), each pixel in the area is an element, and the square of the motion compensation interframe difference of each pixel in the area Find (vx, vy) that minimizes the sum.

  When the pixel (x, y, t) is moved by (vx, vy) between one frame, the motion compensation inter-frame difference d is expressed as equation (4).

  Here, Δx = gx (x, y, t), which represents a horizontal gradient. Similarly, Δy represents a gradient in the vertical direction, and Δt represents a gradient in the time direction. When the square sum E of the motion compensation inter-frame difference is obtained using these, the square sum E is expressed by Equation (5).

  Here, (vx, vy) at which the sum of squares E is minimum is when the partial differential value in each variable is “0”, that is, when the condition of (∂E / ∂vx) = 0 is satisfied. Equations (6) and (7) can be obtained from (5).

  From the equations (6) and (7), (vx, vy) which is the desired motion can be obtained by the equation (8).

  In this equation (8), since the denominator is a square equation, the number of bits in the denominator is large. For this reason, the calculation of Expression (8) cannot be easily processed by hardware. Therefore, formula (8) is obtained by simplifying and approximating formula (8) by assuming that there is no gradient in the oblique direction of the image and that the gradient in the horizontal and vertical directions is sufficiently large. be able to. Equation (9) represents a condition assumed to simplify and approximate Equation (8).

  In addition, the motion vector detection method using the gradient method has a relatively high accuracy with respect to a minute motion, but has a problem that it cannot always follow the motion in an actual moving image. For this reason, an iterative gradient method is known in which the gradient method is repeatedly performed as disclosed in Patent Document 1, and a correct motion vector is gradually obtained by using the convergence of the obtained motion amount.

  FIG. 11 shows the configuration of a conventional motion vector detection unit 51, and FIG. 12 shows the operation. Note that the motion vector detection unit 51 shown in FIG. 11 obtains the motion vector MVa by setting the gradient method iteration count in the iterative gradient method to two.

  The input digital image signal DSa is supplied to the delay unit 511, the motion vector setting unit 512, and the gradient method arithmetic processing units 513 and 515.

  The delay unit 511 delays the image signal DSa by one or a plurality of frame periods, generates an image signal DSb indicating the mth frame when the image signal DSa is the nth frame, and performs a gradient method calculation with the motion vector setting unit 512 This is supplied to the processing units 513 and 515. The motion vector setting unit 512 sets a motion vector MV0 to be a calculation reference when detecting a motion vector of a pixel on a virtual frame using the iterative gradient method, and supplies the motion vector MV0 to the gradient method calculation processing unit 513.

  The gradient method computation processing unit 513 sets a pixel block corresponding to the pixel block on the m-th frame on the n-th frame based on the motion vector MV0 with reference to the pixel block on the m-th frame, and on the m-th frame. Gradient method arithmetic processing is performed using the pixel block and the pixel block on the nth frame. The motion vector v1 obtained by performing the gradient method calculation processing in the gradient method calculation processing unit 513 is supplied to the vector addition unit 514.

  The vector addition unit 514 adds the motion vector MV0 and the motion vector v1, and supplies the result to the gradient method arithmetic processing unit 515 as the motion vector MV1.

  The gradient method arithmetic processing unit 515 sets a pixel block corresponding to the pixel block on the mth frame based on the motion vector MV1 on the nth frame based on the pixel block on the mth frame, Gradient method arithmetic processing is performed using the pixel block and the pixel block on the nth frame. The motion vector v 2 obtained by performing the gradient method calculation processing in the gradient method calculation processing unit 515 is supplied to the vector addition unit 516.

The vector adder 516 adds the motion vector MV1 and the motion vector v2 and outputs the result as a motion vector MVa. That is, the motion vector MVa is obtained as equation (11).
MVa = MV0 + v1 + v2 (11)

  This iterative gradient method has a sufficiently high accuracy as a motion vector detection method required for compression coding and is widely used.

JP 60-158786 A

  By the way, if the motion vector MVa is obtained with reference to the mth frame, it is not possible to directly obtain the motion vector of the pixel on the virtual frame located between the mth frame and the nth frame. For this reason, as shown in FIG. 10, a motion vector allocating unit 52 is provided at the subsequent stage of the motion vector detecting unit 51, and the motion vector of the motion vector MVa obtained with reference to the m-th frame is detected from the motion vector MVa A process of assigning and obtaining a motion vector MVb of a pixel on the virtual frame is required.

  Further, depending on the motion vector MVa obtained with the m-th frame as a reference, it may be difficult to appropriately allocate the motion vector to the pixels on the virtual frame. FIG. 13 is a diagram for explaining the operation of the motion vector allocation unit 52. Here, for example, as shown in FIG. 13A, when motion vectors MVk, MVk + 1, and MVk + 2 are detected for the pixels PMk, PMk + 1, and PMk + 2 on the Mth frame, The pixel PIk + 1 may be allocated using the motion vector MVk + 2 as indicated by a broken line. However, as shown in FIG. 13B, for example, motion vectors MVk ′, MVk + 1 ′, and MVk + 2 ′ are detected for the pixels PMk, PMk + 1, and PMk + 2, and the pixel PIk + on the virtual frame is detected. If the motion vectors MVk + 2 ′ and MVk + 3 ′ are close to each other with respect to 1, it is appropriate if the motion vectors MVk + 2 ′ and MVk + 3 ′ are used for allocation. It is not easy to determine whether the motion vector can be obtained. Further, as shown in FIG. 13C, if the motion vectors MVk + 2 ′ and MVk + 3 ′ are close to other pixels on the virtual frame and there is no motion vector close to the pixel PIk + 1, It is not easy to determine what kind of motion vector can be assigned.

  Therefore, in the present invention, a motion vector detection device that accurately detects a motion vector of a pixel on a virtual frame located between the m-th frame and the n-th frame and generates an image signal of the virtual frame and the same are used. A frame interpolation apparatus, a motion vector detection method, a frame interpolation method using the same, a program, and a recording medium are provided.

  The motion vector detection apparatus according to the present invention uses a digital image signal including an m-th frame and an n-th frame (m ≠ n) to detect pixels on a virtual frame located between the m-th frame and the n-th frame. In a motion vector detecting device for detecting a motion vector, a motion vector setting means for setting a first motion vector, and assuming that the first motion vector is a motion vector of a pixel on a virtual frame, the motion vector is used. The position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined, and the gradient method is calculated using pixel blocks based on the determined positions on the mth frame and the nth frame. A motion vector calculating means for obtaining a second motion vector that is a displacement of motion from the first motion vector, and adding the first to second motion vectors. Those having a vector adding means for the motion vector of the pixel on the virtual frame.

  The frame interpolating apparatus according to the present invention uses a digital image signal including an mth frame and an nth frame (m ≠ n) to detect pixels on a virtual frame positioned between the mth frame and the nth frame. In a frame interpolation device that detects a motion vector and generates an image signal of a virtual frame using the detected motion vector, vector setting means for setting a first motion vector, and the first motion vector on a virtual frame Assuming the motion vector of the pixel, the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using the motion vector, and the determined on the mth frame and the nth frame A motion for calculating a second motion vector, which is a displacement of motion from the first motion vector, by performing an arithmetic processing of the gradient method using a pixel block based on the position of Vector calculation means, vector addition means for adding the first and second motion vectors to obtain a motion vector of a pixel on the virtual frame, and the m-th and n-th frames based on the motion vector obtained by the vector addition means And interpolating means for generating an image signal of a virtual frame by performing an interpolation process using the image signal.

  The motion vector detection method according to the present invention uses a digital image signal including an mth frame and an nth frame (m ≠ n) to detect pixels on a virtual frame located between the mth frame and the nth frame. In the motion vector detection method for detecting a motion vector, a motion vector setting step for setting a first motion vector, and assuming that the first motion vector is a motion vector of a pixel on a virtual frame, the motion vector is used. The position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined, and the gradient method is calculated using pixel blocks based on the determined positions on the mth frame and the nth frame. A motion vector calculating step for obtaining a second motion vector that is a displacement of motion from the first motion vector, and first to second motion vectors; Adding to those having a vector addition step for the motion vector of the pixel on the virtual frame.

  The frame interpolation method according to the present invention uses a digital image signal including an m-th frame and an n-th frame (m ≠ n) to move a pixel on a virtual frame located between the m-th frame and the n-th frame. In a frame interpolation method for detecting a vector and generating an image signal of a virtual frame using the detected motion vector, a vector setting step for setting a first motion vector, and a first motion vector as a pixel on the virtual frame Are used to determine the positions on the m-th frame and the n-th frame corresponding to the pixels on the virtual frame using the motion vector, and on the determined m-th frame and n-th frame. A motion for calculating a second motion vector, which is a displacement of motion from the first motion vector, by performing a gradient method calculation process using a pixel block based on the position A vector calculation step of adding a first motion vector and a second motion vector to obtain a motion vector of a pixel on the virtual frame, and an mth frame and an nth frame based on the motion vector obtained in the vector addition step. And an interpolation step of generating an image signal of a virtual frame by performing an interpolation process using the image signal.

  A program according to the present invention assumes that a motion vector setting step for setting a first motion vector in a computer and a first motion vector as a motion vector of a pixel on a virtual frame are used to perform virtual processing using the motion vector. Gradient method arithmetic processing is performed by determining positions on the m-th frame and n-th frame corresponding to pixels on the frame, and using pixel blocks based on the determined positions on the m-th frame and n-th frame. And a motion vector calculation step for obtaining a second motion vector, which is a displacement of motion from the first motion vector, and the first to second motion vectors are added to obtain a motion vector of a pixel on the virtual frame. The vector addition step is executed. Further, an interpolation step is further executed to generate an image signal of a virtual frame by performing an interpolation process using the image signals of the mth frame and the nth frame based on the motion vector obtained in the vector addition step.

  The recording medium according to the present invention assumes a motion vector setting step for setting a first motion vector, and a first motion vector as a motion vector of a pixel on the virtual frame. The position on the m-th frame and the n-th frame corresponding to the pixel is determined, and the gradient method is calculated using the pixel block based on the determined position on the m-th frame and the n-th frame. , A motion vector calculation step for obtaining a second motion vector that is a displacement of motion from the first motion vector, and vector addition that adds the first and second motion vectors to obtain a motion vector of a pixel on the virtual frame A program for causing a computer to execute the steps is recorded. Also, a program for further executing an interpolation step of generating an image signal of a virtual frame by performing an interpolation process using the image signals of the m-th frame and the n-th frame based on the motion vector obtained in the vector addition step is recorded. It is a thing.

  In the present invention, as the first motion vector, the motion vector of the pixel adjacent to the pixel for which the motion vector on the virtual frame is obtained, or the pixel of the previous frame having the same position as the pixel for which the motion vector on the virtual frame is obtained. The detected motion vector is set. The first motion vector is assumed to be a motion vector of a pixel on the virtual frame, and the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using this motion vector. Further, in determining the position on the mth frame and the nth frame, the time phase between the mth frame and the nth frame of the virtual frame is used. Gradient method arithmetic processing is performed using pixel blocks based on the determined positions on the mth frame and the nth frame, and the second motion, which is the displacement of the motion from the first motion vector. A vector is required. The first and second motion vectors are added to obtain the motion vector of the pixel on the virtual frame. In addition, the motion vector obtained by adding the first and second motion vectors is used as the first motion vector, and the calculation method of the gradient method is repeated. Here, for example, when the number of iterations reaches a preset number, or when the magnitude of the second motion vector becomes smaller than a predetermined level, the iteration is terminated. At this time, the first to second motions are completed. A motion vector obtained by adding the vectors is used as a motion vector of a pixel on the virtual frame. Based on the obtained motion vector of the pixel on the virtual frame, the positions on the m-th frame and the n-th frame corresponding to the pixel are determined, and the image signals at the determined positions on the m-th frame and the n-th frame are determined. Is used to calculate the pixel signal of the pixel for which the motion vector has been obtained, thereby generating an image signal of the virtual frame.

  According to the present invention, the first motion vector is assumed to be a motion vector of a pixel on the virtual frame, and the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame using this motion vector. Is determined, and gradient method calculation processing is performed using pixel blocks based on the determined positions on the m-th frame and the n-th frame, and is a displacement of the motion from the first motion vector. A second motion vector is determined. The obtained second motion vector is added to the first motion vector to obtain a motion vector of a pixel on the virtual frame. In addition, based on the obtained motion vector of the pixel on the virtual frame, the image signal of the virtual frame is generated by performing interpolation processing using the image signals of the m-th frame and the n-th frame. Thus, since the motion vector of the pixel on the virtual frame can be directly obtained, there is no need for post-processing such as motion vector assignment, and the motion vector can be easily detected. Furthermore, since there is no need for post-processing such as motion vector assignment, the configuration of the frame interpolation device can be simplified.

  Further, the second motion vector is calculated using the motion vector obtained by adding the first to second motion vectors, so that the calculation of the second motion vector and the first to second motions are performed. Since the vector addition is repeated, the motion vector of the pixel on the virtual frame can be obtained with high accuracy.

  Furthermore, since the iteration is terminated when the second motion vector becomes smaller than a predetermined level, the motion vector of the pixel on the virtual frame can be efficiently obtained with a desired accuracy.

  Further, since the positions on the mth frame and the nth frame corresponding to the pixels on the virtual frame are determined according to the time phase between the mth frame and the nth frame of the virtual frame, the pixels on the virtual frame The positions on the mth frame and the nth frame corresponding to can be easily and correctly determined.

  Also, the first motion vector is detected with respect to the motion vector of the pixel adjacent to the pixel for which the motion vector on the virtual frame is obtained, or the pixel in the previous frame whose position is the same as the pixel for which the motion vector on the virtual frame is obtained. Therefore, the motion vector can be quickly converged when the calculation method of the gradient method is repeated.

  Further, using the obtained motion vector on the virtual frame, the positions on the mth frame and the nth frame corresponding to the pixels on the virtual frame are determined, and the determined positions on the mth frame and the nth frame. The image signal of the virtual frame is generated by performing the interpolation process using the image signal of the virtual frame to obtain the pixel signal of the pixel on the virtual frame. Therefore, a good frame interpolation image can be obtained based on the obtained motion vector on the virtual frame.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows the configuration of the frame interpolation apparatus. The motion vector detection unit 11 of the frame interpolation apparatus 10 delays the input digital image signal DSa by one or a plurality of frame periods, and when the image signal DSa is the n (m) th frame, the m (n) th. An image signal DSb indicating a frame is generated (m ≠ n). Using the image signal DSa and the image signal DSb, a motion vector MVc of a pixel on a virtual frame located between the mth frame and the nth frame is obtained and supplied to the image interpolation unit 13 together with the image signals DSa and DSb. . The image interpolation unit 13 generates and outputs a virtual frame image signal DSc using the image signals DSa and DSb and the motion vector MVc.

  FIG. 2 shows the configuration of the motion vector detection unit 11. Note that the motion vector detection unit 11 shown in FIG. 2 obtains the motion vector MVc by setting the number of iterations of the gradient method in the iterative gradient method to two.

  The image signal DSa is supplied to the delay unit 111, the motion vector setting unit 112, and the motion vector calculation units 113 and 115.

  The delay unit 111 delays the image signal DSa by one or a plurality of frame periods, generates an image signal DSb indicating the mth frame when the image signal DSa is the nth frame, for example, and the motion vector setting unit 112 and the motion vector It supplies to the calculation parts 113 and 115.

  The motion vector setting unit 112 sets a motion vector MV0 as a reference for calculation when detecting a motion vector of a pixel on the virtual frame, and supplies the motion vector MV0 to the motion vector calculation unit 113 and the vector addition unit 114. For example, a motion vector detected for a pixel adjacent to a pixel on a virtual frame for detecting a motion vector is set as the motion vector MV0. Alternatively, the motion vector detected for the pixel in the previous frame having the same pixel position as the pixel on the virtual frame for detecting the motion vector is set as the motion vector MV0. As described above, the motion vector MV0 can be easily set by using the motion vector detected for the adjacent pixel and the motion vector detected for the pixel of the previous frame. Further, the motion vector MV0 often has a small error from the detected motion vector MVc, and the detection accuracy of the motion vector MVc when the gradient method is performed a predetermined number of times can be improved. Alternatively, a rough motion vector may be detected using the image signals DSa and DSb, and the detected motion vector may be set as the motion vector MV0.

  The motion vector calculation unit 113 includes a start point setting coefficient supply unit 113a, an end point setting coefficient supply unit 113b, multipliers 113c and 113d, and a gradient method calculation processing unit 113e. When the reference of the motion vector MV0 is a virtual frame, the start point setting coefficient supply unit 113a generates a coefficient KS for determining the start point of the motion vector MV0 on the mth frame and supplies the coefficient KS to the multiplier 113c. The multiplier 113c multiplies the motion vector MV0 by the coefficient KS and supplies the result to the gradient method arithmetic processing unit 113e as the motion vector hs1. When the reference of the motion vector MV0 is a virtual frame, the end point setting coefficient supply unit 113b generates a coefficient KS for determining the end point of the motion vector MV0 on the nth frame and supplies the coefficient KS to the multiplier 113d. The multiplier 113d multiplies the motion vector MV0 and the coefficient KE, and supplies the result to the gradient method arithmetic processing unit 113e as the motion vector he1.

  Here, when a virtual frame located between the m-th frame and the n-th frame has a time interval ratio (time phase) of “α: (1-α)” as shown in FIG. If the reference of the motion vector MV0 is, for example, the m-th frame, the start point and end point of the motion vector MV0 are set as shown in FIG. 3A, and the motion vector MV0 may not pass through the pixel PI of the virtual frame. Therefore, the coefficient KS supplied from the start point setting coefficient supply unit 113a is a value “−α” corresponding to the time phase, and the coefficient KE supplied from the end point setting coefficient supply unit 113b is a value “1-α” corresponding to the time phase. The motion vectors hs1 and he1 are supplied to the gradient method arithmetic processing unit 113e, and as shown in FIG. 3B, when the reference of the motion vector MV0 is set to the virtual frame, the start point on the mth frame and the nth frame The end point can be discriminated by the motion vectors hs1 and he1.

  The gradient method computing unit 113e sets pixel blocks in the m-th frame and the n-th frame with reference to the start point on the m-th frame and the end point on the n-th frame determined by the motion vectors hs1, he1, and the m-th frame. Gradient method arithmetic processing is performed using the pixel block of the nth frame to calculate a motion vector va1. The calculated motion vector va1 is supplied to the vector adder 114.

  The vector addition unit 114 adds the motion vector va1 to the motion vector MV0 to generate a motion vector MVa1, and supplies the motion vector MVa1 to the motion vector calculation unit 115 and the vector addition unit 116.

  The motion vector calculation unit 115 is configured similarly to the motion vector calculation unit 113, calculates a motion vector va2 based on the motion vector MVa1, and supplies the motion vector va2 to the vector addition unit 116. The vector addition unit 116 adds the motion vector va2 to the motion vector MVa1 and supplies the motion vector MVc to the image interpolation unit 13.

  If the gradient method is repeated twice in this way, as shown in FIG. 4, the motion vectors va1 and va2 calculated on the basis of the virtual frame are added to the motion vector MV0, and the motion of the pixel PI on the virtual frame The vector MVc can be detected with high accuracy.

As shown in FIG. 5, the image interpolation unit 13 determines the start point PS on the mth frame and the end point PE on the nth frame of the motion vector MVc that passes through the pixel PI on the virtual frame, and the pixel of the start point PS. The signal Dps is obtained using pixel signals of peripheral pixels on the mth frame. For example, interpolation is performed using pixel signals of adjacent pixels to calculate a pixel signal Dps. Similarly, the pixel signal Dpe of the end point PE is obtained using the pixel signals of the peripheral pixels on the nth frame. For example, interpolation is performed using pixel signals of adjacent pixels to calculate a pixel signal Dpe. Further, using the calculated pixel signal Dps and the pixel signal Dpe, an interpolation process according to the temporal phase of the virtual frame is performed to calculate the pixel signal Dpi of the pixel PI. For example, the arithmetic processing shown in Expression (12) is performed to calculate the pixel signal Dpi corresponding to the temporal phase of the virtual frame.
Dpi = (1−α) Dps + αDpe (12)

  The image signal DSc of the virtual frame is generated and output using the pixel signal Dpi of each pixel generated in this way.

  As described above, the motion vector calculation units 113 and 115 perform the gradient method calculation processing based on the virtual frame by moving the start and end points of the motion vectors MV0 and MVa1 according to the time phase α of the virtual frame. Therefore, the motion vector of the pixel on the virtual frame can be detected with high accuracy, and the process of assigning the motion vector to the virtual frame becomes unnecessary. In addition, since the process of assigning motion vectors is not required, the configuration can be simplified, which is effective in reducing the size of the device. Furthermore, since the motion vector is calculated using the iterative gradient method, the motion vector of the pixel on the virtual frame can be detected with higher accuracy.

  FIG. 2 shows a configuration in which the gradient method is repeated two times, but the iterative gradient method is obtained by connecting the motion vector calculation unit and the vector addition unit in multiple stages for the number of gradient method iterations. It is possible to change the number of iterations of gradient method arithmetic processing in.

Incidentally, the motion vector detection unit 11 shown in FIG. 2 obtains the motion vector MVc by providing the motion vector calculation unit and the vector addition unit for the number of iterations of the gradient method. As one example, the motion vector MVc can be obtained by repeating the gradient method once or a plurality of times. The configuration of the motion vector detection unit in this case is shown in FIG. In FIG. 6, portions corresponding to those in FIG. 2 are denoted by the same reference numerals. The delay unit 111 generates the image signal DSb by delaying the image signal DSa by one or a plurality of frame periods, and the motion vector setting unit 112. And supplied to the motion vector calculation unit 113.

  The motion vector setting unit 112 sets the motion vector MV0 and supplies it to the vector selection unit 120. The vector selection unit 120 is supplied with the motion vector MVa1 output from the vector addition unit 114.

  The vector selection unit 120 supplies the motion vector MV 0 supplied from the motion vector setting unit 112 to the motion vector calculation unit 113 and the vector addition unit 114. Alternatively, the motion vector MVa1 supplied from the vector addition unit 114 is supplied to the motion vector calculation unit 113 and the vector addition unit 114 as the motion vector MV0.

  The motion vector calculation unit 113 performs the gradient method calculation as described above using the motion vector MV0 supplied from the vector selection unit 120, generates the motion vector va1, and supplies the motion vector va1 to the vector addition unit 114. The vector addition unit 114 adds the motion vector MV0 supplied from the vector selection unit 120 and the motion vector va1 supplied from the motion vector calculation unit 113 to generate a motion vector MVa1.

  As described above, the motion vector detection unit having one motion vector calculation unit and vector addition unit performs the first gradient method calculation on the pixels on the virtual frame, and the motion vector set by the motion vector setting unit 112 MV 0 is supplied from the vector selection unit 120 to the motion vector calculation unit 113 and the vector addition unit 114. At this time, the motion vector calculation unit 113 calculates the above-described motion vector va1 and supplies it to the vector addition unit 114. The vector addition unit 114 adds the motion vector va1 to the motion vector MV0 and outputs the motion vector MVa1.

  Next, when the second gradient method calculation is performed, the motion vector MVa1 output from the vector addition unit 114 is supplied from the vector selection unit 120 to the motion vector calculation unit 113 and the vector addition unit 114 as a motion vector MV0. At this time, the motion vector calculation unit 113 performs the same processing as the motion vector calculation unit 115 described above, and supplies the calculated motion vector va2 to the vector addition unit 114 as the motion vector va1. The vector addition unit 114 adds the motion vector va1 to the motion vector MV0 and outputs the motion vector MVa1. That is, a motion vector obtained by adding the motion vector MVa1 and the motion vector va2 is output as the motion vector MVa1.

  Similarly, if the motion vector MVa1 output from the vector adder 114 is supplied as the motion vector MV0 from the vector selector 120 to the motion vector calculator 113 and the vector adder 114, the motion obtained by repeating the gradient method Different vectors are output from the vector addition unit 114 as a motion vector MVa1.

  For this reason, when the gradient method calculation is repeated a predetermined number of times, the vector selection unit 120 replaces the motion vector MVa1 supplied from the vector addition unit 114 with respect to the next pixel supplied from the motion vector setting unit 112. The set motion vector MV0 is supplied to the motion vector calculation unit 113 and the vector addition unit 114. Further, the motion vector MVa1 obtained at this time is supplied to the image interpolation unit 13 as a motion vector MVc. In this case, the gradient method calculation is ended and the motion vector detection of the next pixel is started.

  The vector selection unit 120 replaces the motion vector MVa1 supplied from the vector addition unit 114 when the magnitude of the motion vector va1 calculated by the motion vector calculation unit 113 is smaller than a predetermined level. The motion vector MV0 set for the next pixel supplied from the setting unit 112 may be supplied to the motion vector calculation unit 113 and the vector addition unit 114. In this case, when the magnitude of the motion vector va1 becomes smaller than a predetermined level and the accuracy of the motion vector MVc is ensured, the gradient method calculation is automatically repeated and the motion vector detection of the next pixel is started. Therefore, the motion vector MVc can be efficiently detected with a desired accuracy.

  In this way, by supplying the motion vector MVa1 output from the vector addition unit 114 to the motion vector calculation unit 113 and the vector addition unit 114 as the motion vector MV0, the gradient method calculation can be performed repeatedly. It is not necessary to provide the vector calculation unit and vector addition unit for the number of iterations, and the configuration can be simplified.

  Incidentally, the frame interpolation apparatus can be performed not only using a motion vector detection unit and an image interpolation unit configured by hardware, but also using software. FIG. 7 shows a configuration when motion vector detection and frame interpolation are performed by software.

  The digital image signal DSa is supplied to the input unit 21. The input unit 21 stores the supplied image signal DSa in a RAM (Random Access Memory) 22.

  A CPU (Central Processing Unit) 23 reads out and executes a program stored in a ROM (Read Only Memory) 24 or a recording medium 26 attached to the recording medium reproducing unit 25, and displays an image signal indicating the nth frame. The DSa and the image signal DSb indicating the mth frame are read from the RAM 22 and the motion vector of the pixel on the virtual frame is detected. Further, interpolation processing is performed using the detected motion vector and the image signals DSa and DSb to generate pixel signals of pixels on the virtual frame. In this way, the image signal DSc of the virtual frame is generated using the pixel signal of each pixel of the generated virtual frame and is output from the output unit 27. The RAM 22 not only stores image signals, but is also used as a work area when performing iterative gradient method arithmetic processing and image interpolation processing. The input unit 21, RAM 22, CPU 23, ROM 24, recording medium playback unit 25, and output unit 27 are connected via a bus 28.

  FIG. 8 is a flowchart showing the frame interpolation operation when the gradient method is repeated twice. In step ST1, the CPU 23 sets the motion vector MV0 in the same manner as the motion vector setting unit 112 described above, and proceeds to step ST2.

  In step ST2, the start point on the mth frame and the end point on the nth frame of the motion vector MV0 are discriminated on the basis of the virtual frame. That is, the motion vector MV0 is multiplied by a coefficient corresponding to the time phase of the virtual frame to generate the motion vectors hs1 and he1, and the start point on the mth frame based on the pixel position on the virtual frame and the motion vectors hs1 and he1. And the end point on the nth frame.

  In step ST3, gradient method calculation processing is performed to calculate a motion vector va1. In this gradient method calculation processing, gradient method calculation processing is performed using a pixel block of the mth frame with reference to the start point determined in step ST2 and a pixel block of the nth frame with reference to the determined end point.

  In step ST4, vector addition is performed and the motion vector MVa1 is generated by adding the motion vector va1 calculated in step ST3 to the motion vector MV0.

  In step ST5, the start point on the mth frame and the end point on the nth frame of the motion vector MVa1 are discriminated in the same manner as in step ST2, using the virtual frame as a reference.

  In step ST6, gradient method calculation processing is performed to calculate a motion vector va2. In this gradient method calculation processing, gradient method calculation processing is performed using a pixel block of the mth frame with reference to the start point determined in step ST5 and a pixel block of the nth frame with reference to the determined end point.

  In step ST7, vector addition is performed, and the motion vector MVc is added to the motion vector va2 calculated in step ST6 to generate a motion vector MVc.

  In step ST8, image interpolation processing is performed, and an image signal DSc of a virtual frame is generated using the motion vector MVc and the image signals of the mth frame and the nth frame. That is, the position on the mth frame and the position on the nth frame corresponding to the pixel are determined using the motion vector MVc for the pixel on the virtual frame. Next, the signal level at the determined position on the mth frame and the position on the nth frame is obtained, and the pixel signal of the pixel is obtained using the signal level at the position on the mth frame and the position on the nth frame. Is obtained by, for example, interpolation processing. Further, the image signal DSc is generated using the pixel signal of each pixel of the virtual frame.

  FIG. 9 is a flowchart showing a frame interpolation operation that can vary the number of iterations of the gradient method. In step ST11, the CPU 23 sets a motion vector MV0 in the same manner as in step ST1, and proceeds to step ST12.

  In step ST12, as in step ST2, the start point on the mth frame and the end point on the nth frame of the motion vector MV0 are determined using the virtual frame as a reference.

  In step ST13, the gradient method calculation process is performed in the same manner as in step ST3, the motion vector va1 is calculated, and the process proceeds to step ST14.

  In step ST14, vector addition is performed as in step ST4, and the motion vector va1 calculated in step ST13 is added to the motion vector MV0 to generate a new motion vector MVa1.

  In step ST15, it is determined whether or not the motion vector MVa1 calculated in step ST14 is selected as the motion vector MV0. Here, for example, when the gradient method arithmetic processing has not been performed for a preset number of iterations, the motion vector MVa1 is selected as the motion vector MV0, and the motion vector MVa1 calculated in step ST14 is set as the motion vector MV0. Then, the process returns to step ST12. Also, for example, when gradient method calculation processing is performed for a preset number of iterations, or when the motion vector va1 calculated in step ST13 is smaller than a predetermined level, the motion vector MVa1 is not selected as the motion vector MV0. As an example, the motion vector MVa1 calculated in step ST14 is set as the motion vector MVc, and the process proceeds to step ST16.

  In step ST16, image interpolation processing is performed in the same manner as in step ST8, and the image signal DSc of the virtual frame is generated using the motion vector MVc and the image signals of the mth frame and the nth frame.

  In this way, the image signal of the virtual frame located between the mth frame and the nth frame can also be generated by software.

  Further, if the program for performing the operations of FIG. 8 and FIG. 9 is recorded on the recording medium 26, for example, a recording medium using light or magnetism, a recording medium using a semiconductor element, etc. It is possible to easily perform frame interpolation with this computer apparatus.

  As described above, the present invention is useful when performing frame interpolation, and is applied to frame interpolation using an image signal with image motion.

It is a figure which shows the structure of a frame interpolation apparatus. It is a figure which shows the structure of a motion vector detection part. It is a figure for demonstrating the start point and end point of a motion vector. It is a figure for demonstrating operation | movement of a motion vector detection part. It is a figure for demonstrating operation | movement of an image interpolation part. It is a figure which shows the other structure of a motion vector detection part. It is a figure which shows the structure in the case of using software. It is a flowchart which shows frame interpolation operation | movement. It is a flowchart which shows another frame interpolation operation | movement. It is a figure which shows the structure of the conventional frame interpolation apparatus. It is a figure which shows the structure of the conventional motion vector detection part. It is a figure for demonstrating operation | movement of the conventional motion vector detection part. It is a figure for demonstrating operation | movement of a motion vector allocation part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10,50 ... Frame interpolation apparatus 11,51 ... Motion vector detection part, 13,53 ... Image interpolation part, 21 ... Input part, 22 ... RAM, 23 ... CPU, 24 ... ROM, 25 ... recording medium playback unit, 26 ... recording medium, 27 ... output unit, 52 ... motion vector allocation unit, 111,511 ... delay unit, 112,512 ... motion vector setting unit, 113, 115 ... motion vector calculation unit, 113a ... start point setting coefficient supply unit, 113b ... end point setting coefficient supply unit, 113c, 113d ... multiplier, 113e, 513, 515... Gradient method arithmetic processing unit, 114, 116, 514, 516 ... Vector addition unit, 120 ... Vector selection unit

Claims (30)

Motion vector detection for detecting a motion vector of a pixel on a virtual frame located between the m-th frame and the n-th frame using a digital image signal including the m-th frame and the n-th frame (m ≠ n) In the device
Motion vector setting means for setting a first motion vector;
Assuming that the first motion vector is a motion vector of a pixel on the virtual frame, the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using the motion vector. The gradient is calculated using a pixel block based on the determined positions on the m-th frame and the n-th frame, and is a displacement of the motion from the first motion vector. Motion vector calculating means for obtaining a motion vector of 2;
A motion vector detection apparatus comprising: vector addition means for adding the first and second motion vectors to obtain a motion vector of a pixel on the virtual frame. A pair of the motion vector calculation means and the vector addition means are connected in multiple stages;
The motion vector obtained by the vector addition means is supplied as the first motion vector to the motion vector calculation means in the next stage, and the motion vector obtained by the vector addition means in the last stage is supplied to the pixel on the virtual frame. The motion vector detection apparatus according to claim 1, wherein the motion vector is a motion vector. In place of the first motion vector set by the motion vector setting means, a vector selection means for supplying the motion vector obtained by the vector addition means to the motion vector calculation means as the first motion vector is provided,
2. The motion vector detection apparatus according to claim 1, wherein the motion vector calculation unit and the vector addition unit are iterated to obtain a motion vector of a pixel on the virtual frame. When the motion vector calculation means has not performed the gradient method arithmetic processing for a preset number of iterations, the vector selection means uses the motion vector obtained by the vector addition means as the first motion vector. Supplying the motion vector calculating means;
When the motion vector calculation means performs the gradient method calculation processing for a preset number of iterations, or when the second motion vector becomes smaller than a predetermined level, the vector selection means The motion vector obtained by the vector addition means is not supplied to the motion vector calculation means as the first motion vector, and the motion vector obtained by the vector addition means is used as a motion vector of a pixel on the virtual frame. 4. The motion vector detection device according to claim 3, wherein The motion vector calculation means uses the time phase between the m-th frame and the n-th frame of the virtual frame, and positions on the m-th frame and the n-th frame corresponding to pixels on the virtual frame. The motion vector detection apparatus according to claim 1, wherein: The motion vector setting means, as the first motion vector, a motion vector of a pixel adjacent to a pixel for which a motion vector on the virtual frame is obtained, or a previous frame having the same position as a pixel for which a motion vector on the virtual frame is obtained The motion vector detection apparatus according to claim 1, wherein a motion vector detected for each pixel is used. Using a digital image signal including an mth frame and an nth frame (m ≠ n), a motion vector of a pixel on a virtual frame located between the mth frame and the nth frame is detected, and the detection is performed. In a frame interpolation device that generates an image signal of the virtual frame using the motion vector that has been obtained,
Vector setting means for setting a first motion vector;
Assuming that the first motion vector is a motion vector of a pixel on the virtual frame, the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using the motion vector. The gradient is calculated using a pixel block based on the determined positions on the m-th frame and the n-th frame, and is a displacement of the motion from the first motion vector. Motion vector calculating means for obtaining a motion vector of 2;
Vector addition means for adding the first to second motion vectors to obtain a motion vector of a pixel on the virtual frame;
Interpolating means for generating an image signal of the virtual frame by performing interpolation processing using the image signals of the m-th frame and the n-th frame based on the motion vector obtained by the vector adding means, Frame interpolation device. A pair of the motion vector calculation means and the vector addition means are connected in multiple stages;
The motion vector obtained by the vector addition means is supplied as the first motion vector to the motion vector calculation means in the next stage, and the motion vector obtained by the vector addition means in the last stage is supplied to the pixel on the virtual frame. 8. The frame interpolating apparatus according to claim 7, wherein the frame interpolating apparatus is a motion vector. In place of the first motion vector set by the motion vector setting means, a vector selection means for supplying the motion vector obtained by the vector addition means to the motion vector calculation means as the first motion vector is provided,
8. The frame interpolation apparatus according to claim 7, wherein the motion vector calculation unit and the vector addition unit are iterated to obtain a motion vector of a pixel on the virtual frame. When the motion vector calculation means has not performed the gradient method arithmetic processing for a preset number of iterations, the vector selection means uses the motion vector obtained by the vector addition means as the first motion vector. Supplying the motion vector calculating means;
When the motion vector calculation means performs the gradient method calculation processing for a preset number of iterations, or when the second motion vector becomes smaller than a predetermined level, the vector selection means The motion vector obtained by the vector addition means is not supplied to the motion vector calculation means as the first motion vector, and the motion vector obtained by the vector addition means is used as a motion vector of a pixel on the virtual frame. The frame interpolating apparatus according to claim 9, wherein: The motion vector calculation means uses the time phase between the m-th frame and the n-th frame of the virtual frame, and positions on the m-th frame and the n-th frame corresponding to pixels on the virtual frame. The frame interpolating apparatus according to claim 7, wherein: The motion vector setting means, as the first motion vector, a motion vector of a pixel adjacent to a pixel for which a motion vector on the virtual frame is obtained, or a previous frame having the same position as a pixel for which a motion vector on the virtual frame is obtained 8. The frame interpolating apparatus according to claim 7, wherein a motion vector detected for each of the pixels is used. The interpolation means discriminates the positions on the m-th frame and the n-th frame corresponding to the pixels on the virtual frame using the motion vector obtained by the vector addition means, and the determined m-th frame 8. The image signal of the virtual frame is generated by performing interpolation using the image signal at the position on the nth frame and the pixel signal of the pixel on the virtual frame. Frame interpolator. Motion vector detection for detecting a motion vector of a pixel on a virtual frame located between the m-th frame and the n-th frame using a digital image signal including the m-th frame and the n-th frame (m ≠ n) In the method
A motion vector setting step for setting a first motion vector;
Assuming that the first motion vector is a motion vector of a pixel on the virtual frame, the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using the motion vector. The gradient is calculated using a pixel block based on the determined positions on the m-th frame and the n-th frame, and is a displacement of the motion from the first motion vector. A motion vector calculating step for obtaining a motion vector of 2;
And a vector addition step of adding the first and second motion vectors to obtain a motion vector of a pixel on the virtual frame. Provide a plurality of pairs of the motion vector calculation step and the vector addition step,
The motion vector obtained in the vector addition step is supplied as the first motion vector to the next motion vector calculation step, and the motion vector obtained in the final vector addition step is used as the motion vector of the pixel on the virtual frame. The motion vector detection method according to claim 14, wherein: In place of the first motion vector set in the motion vector setting step, a vector selection step for supplying the motion vector obtained in the vector addition step to the motion vector calculation step as the first motion vector is provided,
15. The motion vector detection method according to claim 14, wherein the motion vector calculation step and the vector addition step are repeated to obtain a motion vector of a pixel on the virtual frame. When the motion vector calculation step is not performed for a preset number of iterations, the vector selection step supplies the motion vector obtained in the vector addition step to the motion vector calculation step as the first motion vector. ,
When the motion vector calculation step is performed a preset number of times, or when the second motion vector becomes smaller than a predetermined level, the vector selection step includes the motion vector obtained in the vector addition step. 17 as the first motion vector is not supplied to the motion vector calculation step, and the motion vector obtained in the vector addition step is used as a motion vector of a pixel on the virtual frame. Motion vector detection method. In the motion vector calculation step, a position on the m-th frame and the n-th frame corresponding to a pixel on the virtual frame using a time phase between the m-th frame and the n-th frame of the virtual frame. The motion vector detection method according to claim 14, wherein: In the motion vector setting step, as the first motion vector, a motion vector of a pixel adjacent to a pixel for which a motion vector on the virtual frame is obtained, or a previous frame having the same position as the pixel for which the motion vector on the virtual frame is obtained 15. The motion vector detection method according to claim 14, wherein a motion vector detected for each of the pixels is used. Using a digital image signal including an mth frame and an nth frame (m ≠ n), a motion vector of a pixel on a virtual frame located between the mth frame and the nth frame is detected, and the detection is performed. In a frame interpolation method for generating an image signal of the virtual frame using a motion vector obtained,
A vector setting step for setting a first motion vector;
Assuming that the first motion vector is a motion vector of a pixel on the virtual frame, the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using the motion vector. The gradient is calculated using a pixel block based on the determined positions on the m-th frame and the n-th frame, and is a displacement of the motion from the first motion vector. A motion vector calculating step for obtaining a motion vector of 2;
A vector addition step of adding the first to second motion vectors to obtain a motion vector of a pixel on the virtual frame;
An interpolation step of generating an image signal of the virtual frame by performing an interpolation process using the image signals of the m-th frame and the n-th frame based on the motion vector obtained in the vector addition step. Frame interpolation method. Provide a plurality of pairs of the motion vector calculation step and the vector addition step,
The motion vector obtained in the vector addition step is supplied as the first motion vector to the next motion vector calculation step, and the motion vector obtained in the final vector addition step is used as the motion vector of the pixel on the virtual frame. 21. The frame interpolation method according to claim 20, wherein: In place of the first motion vector set in the motion vector setting step, a vector selection step for supplying the motion vector obtained in the vector addition step to the motion vector calculation step as the first motion vector is provided,
The frame interpolation method according to claim 20, wherein the motion vector calculation step and the vector addition step are repeated to obtain a motion vector of a pixel on the virtual frame. When the motion vector calculation step is not performed for a preset number of iterations, the vector selection step supplies the motion vector obtained in the vector addition step to the motion vector calculation step as the first motion vector. ,
When the motion vector calculation step is performed a preset number of times, or when the second motion vector becomes smaller than a predetermined level, the vector selection step includes the motion vector obtained in the vector addition step. 23. The method according to claim 22, wherein the first motion vector is not supplied to the motion vector calculation step, and the motion vector obtained in the vector addition step is used as a motion vector of a pixel on the virtual frame. Frame interpolation method. In the motion vector calculating step, positions on the mth frame and the nth frame corresponding to pixels on the virtual frame according to a time phase between the mth frame and the nth frame of the virtual frame. 21. The frame interpolation method according to claim 20, wherein: In the motion vector setting step, as the first motion vector, a motion vector of a pixel adjacent to a pixel for which a motion vector on the virtual frame is obtained, or a previous frame having the same position as the pixel for which the motion vector on the virtual frame is obtained 21. The frame interpolation method according to claim 20, wherein a motion vector detected for each of the pixels is used. In the interpolation step, the position on the m-th frame and the n-th frame corresponding to the pixel on the virtual frame is determined using the motion vector obtained in the vector addition step, and the determined m-th frame The image signal of the virtual frame is generated by performing interpolation using the image signal at the position on the nth frame and the pixel signal of the pixel on the virtual frame. Frame interpolation method. In order to detect a motion vector of a pixel on a virtual frame located between the m-th frame and the n-th frame using a digital image signal including the m-th frame and the n-th frame (m ≠ n),
A motion vector setting step for setting a first motion vector;
Assuming that the first motion vector is a motion vector of a pixel on the virtual frame, the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using the motion vector. The gradient is calculated using a pixel block based on the determined positions on the m-th frame and the n-th frame, and is a displacement of the motion from the first motion vector. A motion vector calculating step for obtaining a motion vector of 2;
A program for executing a vector addition step of adding the first and second motion vectors to obtain a motion vector of a pixel on the virtual frame. Using a digital image signal including an mth frame and an nth frame (m ≠ n), a motion vector of a pixel on a virtual frame located between the mth frame and the nth frame is detected, and the detection is performed. In order to generate the image signal of the virtual frame using the motion vector
On the computer,
A motion vector setting step for setting a first motion vector;
Assuming that the first motion vector is a motion vector of a pixel on the virtual frame, the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using the motion vector. The gradient is calculated using a pixel block based on the determined positions on the m-th frame and the n-th frame, and is a displacement of the motion from the first motion vector. A motion vector calculating step for obtaining a motion vector of 2;
A vector addition step of adding the first to second motion vectors to obtain a motion vector of a pixel on the virtual frame;
A program for executing an interpolation step of generating an image signal of the virtual frame by performing an interpolation process using the image signals of the m-th frame and the n-th frame based on the motion vector obtained in the vector addition step. In order to detect a motion vector of a pixel on a virtual frame located between the mth frame and the nth frame using a digital image signal including the mth frame and the nth frame (m ≠ n),
A motion vector setting step for setting a first motion vector;
Assuming that the first motion vector is a motion vector of a pixel on the virtual frame, the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using the motion vector. The gradient is calculated using a pixel block based on the determined positions on the m-th frame and the n-th frame, and is a displacement of the motion from the first motion vector. A motion vector calculating step for obtaining a motion vector of 2;
A computer-readable recording medium storing a program for causing a computer to execute a vector addition step of adding the first and second motion vectors to obtain a motion vector of a pixel on the virtual frame. Using a digital image signal including an mth frame and an nth frame (m ≠ n), a motion vector of a pixel on a virtual frame located between the mth frame and the nth frame is detected, and the detection is performed. In order to generate the image signal of the virtual frame using the motion vector
A motion vector setting step for setting a first motion vector;
Assuming that the first motion vector is a motion vector of a pixel on the virtual frame, the position on the mth frame and the nth frame corresponding to the pixel on the virtual frame is determined using the motion vector. The gradient is calculated using a pixel block based on the determined positions on the m-th frame and the n-th frame, and is a displacement of the motion from the first motion vector. A motion vector calculating step for obtaining a motion vector of 2;
A vector addition step of adding the first to second motion vectors to obtain a motion vector of a pixel on the virtual frame;
In order to cause a computer to execute an interpolation step of generating an image signal of the virtual frame by performing an interpolation process using the image signals of the m-th frame and the n-th frame based on the motion vector obtained in the vector addition step A computer-readable recording medium on which the program is recorded.

Images