JP4931884B2 - Frame rate conversion apparatus, frame rate conversion method, and frame rate conversion program - Google Patents

Description

  The present invention relates to a frame rate conversion apparatus, a frame rate conversion method, and a frame rate conversion program for converting a video frame rate to an arbitrary frame rate.

  2. Description of the Related Art Conventionally, as one of techniques related to digital video production, there is a technique for converting video shot with a film or the like and video signals recorded with the same number of frames into various frame rates. This technique is known from Patent Document 1 and the like. In particular, when converting a progressive video signal system video composed of 24 frames / second into a progressive video signal system video composed of 60 frames / second and recording it, conversion using a conversion system called 2: 3 pull-down is performed. Is generally used (see, for example, Patent Document 1).

  In recent years, in order to improve moving image performance, a combination of a plurality of frames included in a video signal and an interpolation frame generated inside the apparatus using a motion vector of the input video signal is used to create a new frame sequence. A frame rate conversion process for generating a signal is performed (see, for example, Patent Document 2).

  In recent years, with the advancement of digital signal technology, communication, broadcasting, recording media [CD (Compact Disc), DVD (Digital Versatile Disc)], medical images, printing, etc. targeted for video (moving images), images or audio This field has made remarkable progress as a multimedia industry or IT (Information Technology). One of the digital signal technologies for video, images, and audio is compression coding that reduces the amount of information, but the signal theory is typically the Shannon sampling theorem, and more recently the wavelet transform theory. Etc. Also, for example, music CDs use linear PCM (Pulse Code Modulation) without compression, but signal theory is also Shannon's sampling theorem.

  Conventionally, MPEG is known as a compression technique for moving images such as video and animation images, and is used in fields such as adoption of the MPEG-2 system in digital broadcasting and DVD, Internet streaming of third-generation mobile phones, and mobile communication. In recent years, digital compression technology for video signals has become very familiar due to the adoption of the MPEG-4 system and the like. The reasons behind this are an increase in the capacity of storage media, an increase in network speed, an increase in processor performance, and a large scale and low price of system LSI. As described above, an environment for supporting a video application system that requires digital compression has been steadily prepared.

  MPEG2 (ISO (International Organization for Standardization) / IEC (International Electrotechnical Commition) 13818-2) is a method defined as a general-purpose image encoding method, so that both the interlaced scanning method and the sequential scanning method can be supported. It is defined so as to be compatible with both standard resolution images and high-definition images. MPEG2 is currently widely used in a wide range of applications for professional use and consumer use. In MPEG2, for example, image data of a standard resolution of 720 × 480 pixels and interlaced scanning can be compressed to a bit rate of 4 to 8 [Mbps], and a high resolution image of 1920 × 1088 pixels and interlaced scanning can be used. Data can be compressed to a bit rate of 18 to 22 [Mbps], and a high compression rate can be ensured with high image quality.

  In general, in encoding of moving images, the amount of information is compressed by reducing redundancy in the time direction and the spatial direction. Therefore, in inter-picture predictive coding for the purpose of reducing temporal redundancy, motion is detected and a predicted image is created in units of blocks with reference to the forward or backward picture, and the resulting predicted image and the encoded image are encoded. Encoding is performed on the difference value from the target picture. Here, a picture is a term representing a single screen, which means a frame in a progressive image and a frame or field in an interlaced image. Here, an interlaced image is an image in which one frame is composed of two fields having different times. In interlaced image encoding and decoding processing, one frame may be processed as a frame, processed as two fields, or processed as a frame structure or a field structure for each block in the frame. it can.

JP 2003-284007 A JP 2008-167103 A

  A conventional AD conversion / DA conversion system based on Shannon's sampling theorem handles a signal whose band is limited by the Nyquist frequency. At this time, in the DA conversion, a function (regular function) for reproducing a signal in a limited band is used to reproduce a signal that has become discrete by sampling into a continuous wave.

  One of the inventors of the present application has found that various properties of images such as video (moving images), character figures and natural images, or signals such as audio can be classified using a fluency function. According to this theory, the regular function based on Shannon's sampling theorem is one of the fluency functions, and only fits one of the various properties of the signal. Therefore, if a signal having various properties is handled only by the regular function based on Shannon's sampling theorem, there is a risk of limiting the quality of the reproduced signal after DA conversion.

  The wavelet transformation theory, which is one of the above fluency function spaces, represents a signal using a mother wavelet that decomposes the object by resolution, but the optimal mother wavelet is not always given to the signal, and D- This may limit the quality of the reproduction signal after A conversion.

  Here, the fluency function is a function categorized by the parameter m (m is a positive integer from 1 to ∞). m represents that the function can be continuously differentiated only (m−2) times. Incidentally, since the regular function can be differentiated any number of times, m is ∞. Furthermore, the fluency function is composed of (m−1) -th order functions. In particular, the fluency DA function in the fluency function is given a numerical value at the k-th sample point kτ of interest, where τ is the sample interval. It is a function that becomes 0 at other sample points.

  All signal properties can be classified by a fluency function having a parameter m, and are classified by the parameter m. For this reason, the fluency information theory using the fluency function includes the Shannon sampling theorem and wavelet transformation theory, which only represent some of the properties of conventional signals, and is a theoretical system that represents the entire signal. Positioned. By using such a function, it is expected that a high-quality reproduction signal that is not band-limited by the Shannon sampling theorem is obtained over the entire signal after the DA conversion.

  By the way, conventionally, it has been required to convert the 24 frame rate of a movie to 30 frames of video, to increase the frame rate of TV images for higher definition, or to convert it to a mobile frame rate. However, a method of generating a new frame by frame thinning or internal interpolation of preceding and following frames has become the mainstream.

  However, the conventional method of generating a new frame by thinning out the frames or performing internal interpolation of the preceding and following frames has problems such as that the motion of the video is not smooth and the video is not linear.

  Accordingly, an object of the present invention is to provide a frame rate conversion device, a frame rate conversion method, and a frame rate conversion program that can be reproduced with clear and smooth motion even if the number of frames is increased or decreased in view of the conventional problems as described above. There is to do.

  Other objects of the present invention and specific advantages obtained by the present invention will become more apparent from the description of embodiments described below.

  In video, generally there are many scenes where similar scenes continue before and after a frame, and this feature is used to increase the frame rate by using a plurality of pieces of information to improve the image quality. A high-quality interpolated frame is constructed by estimating local corresponding points between a plurality of frames and interpolating corresponding image points.

  In the present invention, video correspondence points between frames are tracked, the time transition is expressed as a function, and the number of frames is increased or decreased by generating a frame under function interpolation by the ratio between the original frame and the number of frames to be converted. Even get clear and smooth video signals.

That is, the present invention is a frame rate conversion apparatus, wherein for a plurality of pixels in a reference frame, the gray value at each pixel point is represented by a continuous function of position, and the above function approximation in a plurality of image frames having different times is provided. A corresponding point estimation processing unit that estimates a position that gives the maximum degree of correlation of image density as a corresponding point, and a tone value that indicates the density of neighboring pixel points for the estimated shade of each corresponding position in each image frame With respect to a plurality of pixels in the reference frame with respect to an interpolation frame generated at a frame rate ratio to be converted between each image frame and a first gradation value generation processing unit that obtains a gradation value at a corresponding position Approximate the shade on the locus of the corresponding point from the estimated gradation value of each corresponding position in each image frame with a fluency function, and convert the function into an interpolated frame. A tone value of each pixel in the vicinity of the corresponding position in the interpolation frame is generated from a second tone value generation processing unit that obtains each tone value of the corresponding position in the interpolation frame and the tone value of each corresponding position in the interpolation frame. And a third gradation value generation processing unit.

The present invention is also a frame rate conversion method, wherein for a plurality of pixels in a reference frame, the gray value at each pixel point is represented by a continuous function of position, and the above function approximation in a plurality of image frames having different times is provided. A corresponding point estimation processing step for estimating a position that gives the maximum degree of correlation of the image shade as a corresponding point, and a tone value indicating the shade of each neighboring pixel point for the shade of each corresponding position estimated in each image frame A first gradation value generation processing step for obtaining a gradation value at a corresponding position , and an interpolation frame generated at a frame rate ratio to be converted between the image frames, with respect to a plurality of pixels in the reference frame , the density of the corresponding point trace from tone values of the corresponding positions estimated at the respective image frames is approximated by a fluency function, interpolation from the function off A second tone value generation processing step of determining the gradation values of the corresponding position in the over-time, from the gray scale value of each corresponding position in the interpolation frame, the gradation value of each pixel adjacent corresponding position in the interpolation frame And a third gradation value generation processing step to be generated.

Furthermore, the present invention is a frame rate conversion program executed by a computer provided in the frame rate conversion apparatus, wherein for a plurality of pixels in a reference frame, the gray value of each pixel point is represented by a continuous function of position, and time The corresponding point estimation processing unit that estimates the position that gives the maximum correlation of the image shades obtained by the above function approximation in a plurality of image frames having different values as corresponding points, and the shades of the corresponding positions estimated in the respective image frames, respectively A first gradation value generation processing unit that obtains a gradation value at a corresponding position from gradation values indicating the density of neighboring pixel points , and an interpolation frame that is generated at a frame rate ratio to be converted between the image frames. , Corresponding points from a plurality of pixels in the reference frame based on the gradation values of the corresponding positions estimated in the image frames From the tone value of each corresponding position in the interpolation frame, a second tone value generation processing unit that approximates the shading on the trace with a fluency function and obtains each tone value of the corresponding position in the interpolation frame from the function, The computer is caused to function as a third gradation value generation processing unit that generates a gradation value of each pixel near the corresponding position in the interpolation frame.

The present invention is a frame rate conversion apparatus, comprising: a first function approximating unit that approximates a density distribution of a plurality of pixels in a reference frame by a continuous function of pixel positions; and the first function approximating unit. a function which is approximated, performs correlation calculation between the function of the gray distribution of the plurality of image frames having different times, the respective positions which gives the maximum value and the corresponding point positions corresponding in the plurality of image frames a corresponding point estimating unit, the horizontal direction of the corresponding point positions in each image frame estimated by the corresponding point estimating means from the origin of each image frame, the coordinates of the distance in the vertical direction, a plurality of images having different said time horizontal position of the coordinate point in the frame, and converts the respective change in the vertical position in a time-series signal, a second that function approximating the time series signal The number approximating means, the approximated function by the second function approximation means, an interpolation frame at any time between the plurality of image frames, corresponding in an interpolation frame corresponding to the corresponding point position of the image frame Using the position as the corresponding point position, the gray value at the corresponding point position of the interpolation frame is obtained by interpolation with the gray value at the corresponding point of the image frame , and the first value is adjusted to the gray value of the corresponding point of the interpolation frame. A function approximation is applied to obtain a density distribution in the vicinity of the corresponding point, and third function approximating means for converting the gray value in the vicinity of the corresponding point into the gray value of the pixel point in the interpolation frame is provided.

The present invention is also a frame rate conversion method, comprising: a first function approximating step for approximating a density distribution of a plurality of pixels in a reference frame by a continuous function of pixel positions; and the first function approximation. Correlation calculation is performed between the function approximated in steps and the function of the gray distribution in a plurality of image frames at different times, and the corresponding point positions corresponding to the positions in the plurality of image frames that give the maximum value are calculated. multiple differing the corresponding point estimating step, the corresponding point positions in each image frame that is estimated by the corresponding point estimating step is coordinated with the horizontal direction, the vertical distance from the origin of each image frame, the time to horizontal position, and then converts each change in vertical position in a time-series signal, function of the time series signals of the coordinate points in the image frame A second function approximation step of similar, the approximated function by the second function approximation step, the interpolation frame at any time between the plurality of image frames, the interpolation corresponding to the corresponding point position of the image frame The corresponding position in the frame is taken as the corresponding point position, and the gray value at the corresponding point position of the interpolated frame is obtained by interpolation with the gray value at the corresponding point of the image frame , and is adjusted to the gray value of the corresponding point in the interpolated frame. And applying the first function approximation to obtain a gray level distribution near the corresponding point, and a third function approximation step for converting the gray value near the corresponding point into the gray value of the pixel point in the interpolation frame. It is characterized by.

Furthermore, the present invention is a frame rate conversion program executed by a computer provided in the frame rate conversion apparatus, wherein a first function for approximating a density distribution of a plurality of pixels in a reference frame by a continuous function of pixel positions . Each position that gives the maximum value by performing a correlation operation between the function approximating means, the function approximated by the first function approximating means, and the function of the grayscale distribution in a plurality of image frames at different times. Corresponding point estimation means for the corresponding point positions in the plurality of image frames , and the corresponding point position in each image frame estimated by the corresponding point estimation means in the horizontal and vertical directions from the origin of each image frame . was coordinated with the distance, horizontal position of the coordinate points in a plurality of image frames having different said time, and the vertical way It converts the respective change in position in a time series signal, and a second function approximation unit for function approximating the time series signals, the approximated function by the second function approximation means, among the plurality of image frames the interpolation frame at any time, the corresponding position in the interpolation frame corresponding to the corresponding point position of the image frame to a corresponding point position, the gray value at the corresponding point position of the interpolation frame, at the corresponding point of the image frame Interpolating with the gray value, applying the first function approximation to the gray value of the corresponding point of the interpolated frame to obtain the gray distribution near the corresponding point, and calculating the gray value near the corresponding point as the interpolated frame The computer is caused to function as a third function approximating means for converting to the gray value of the pixel point in FIG.

  In the present invention, video correspondence points between frames are tracked, the time transition is expressed as a function, and a function-interpolated frame is generated by the ratio between the original frame and the number of frames to be converted. A clear and smooth video signal can be obtained.

  Therefore, according to the present invention, it is possible to display an image with clear and smooth movement at a frame rate suitable for the display.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Needless to say, the present invention is not limited to the following examples, and can be arbitrarily changed without departing from the gist of the present invention.

  The frame rate conversion apparatus 1 according to the present invention is configured as shown in FIG. 1, for example.

  The frame rate conversion apparatus 1 inserts an interpolated frame between original frames as shown in FIGS. 2A and 2B, for example, so that the low frame rate shown in FIG. In this example, 30 frames / second) is converted to a high frame rate movie (60 frames / second in this example) shown in FIG. , And a computer that functions as the first gradation value generation processing unit 3, the second gradation value generation processing unit 4, and the third gradation value generation processing unit 5.

  In the frame rate conversion apparatus 1, the corresponding point estimation processing unit 2 estimates each corresponding point in a plurality of image frames at different times for a plurality of pixels in the reference frame.

  In addition, the first tone value generation processing unit 3 obtains each tone value from the tone value indicating the density of the neighboring pixels for each corresponding point in each image frame estimated by the corresponding point estimation processing unit 2. .

  In addition, the second tone value generation processing unit 4 calculates, for a plurality of pixels in the reference frame, the shading on the locus of corresponding points from the estimated tone values of the corresponding points in the image frames. And the tone value of each corresponding point in the interpolation frame is obtained from the function.

  Further, the third gradation value generation processing unit 5 generates the gradation value of each pixel in the interpolation frame from the gradation value of each corresponding point in the interpolation frame.

  The frame rate conversion apparatus 1 performs estimation by performing corresponding point estimation processing according to the procedure of steps S1 to S4 shown in the flowchart of FIG. 3 by executing a video signal conversion program read from a storage unit (not shown) by a computer. Using the gradation value of the corresponding point, the gradation value of the corresponding point of the interpolation frame is generated by uniform interpolation, and the gradation value of the corresponding point of the interpolation frame is generated by non-uniform interpolation. Execute.

  That is, in this frame rate conversion apparatus 1, first, as shown in FIG. 4A, an image frame at time t = k is set as a reference frame F (k), and a plurality of pixels in the reference frame F (k) is obtained. For Pn (k), the motion vectors in the image frame F (k + 1) at time t = k + 1, the image frame F (k + 2) at time t = k + 2,... Then, corresponding point estimation processing for estimating the corresponding points Pn (k + 1), Pn (k + 2),... P (k + m) in each image frame (k + 1), (k + 2),... F (k + m) is performed. (Step S1).

  Next, for each corresponding point Pn (k + 1), Pn (k + 2),... P (k + m) in each image frame (k + 1), (k + 2),... F (k + m) estimated in step S1 above, As shown in FIG. 4B, a first gradation value generation process is performed to obtain each gradation value from gradation values indicating the shades of neighboring pixels (step S2).

  Next, with respect to the plurality of pixels Pn (k) in the reference frame F (k), as shown in FIG. 4C, the corresponding points Pn (k + 1), Pn ( k + 2),... P (k + m), each tone value, that is, the shade on the corresponding point trajectory of each image frame (k + 1), (k + 2),... F (k + m) is approximated by a fluency function. A second tone value generation process is performed for obtaining each tone value of the corresponding point in the interpolated frame between the image frames (k + 1), (k + 2),... F (k + m) from the function (step S3).

  In the next step S4, as shown in FIG. 4D, from the gradation value of each corresponding point in the interpolation frame F (k + 1/2) generated by the second gradation value generation process in step S3, A third gradation value generation process for generating a gradation value of each pixel in the interpolation frame F (k + 1/2) at time t = k + 1/2 by non-uniform interpolation is performed (step S4).

  Here, in a moving image composed of a plurality of frames, the position of the moving partial image on the frame differs for each frame. In addition, pixel points on one frame do not always move to pixel points at different positions on other frames, and usually correspond to pixels. That is, when one natural image is continuous information, it represents pixel information at different positions on the two frames. In particular, when a new frame image is generated by interpolation between frames, the pixel information on the original frame and the pixels on the new frame are almost all different. For example, when two frame images as shown in FIGS. 5A and 5B are overlapped at the same point, the relationship between the pixel points of each frame (here, rough for explanation) is The relationship is as shown in FIG. That is, the image shifts. In order to obtain the gray value of the grid point (pixel point without a mark) of the first frame using these two frame images, non-uniform interpolation processing is required.

For example, as shown in FIG. 6, the image interpolation process for determining the value of the position of the pixel u (τ _x , τ _y ) newly generated when the resolution of the image is converted is the original pixel u (x _i , y _j ) And the interpolation function h (x).

  Then, the same partial image is made to correspond using a plurality of frame images, and uniform interpolation is performed from the pixel information in the horizontal (vertical) direction near the desired corresponding point using a uniform interpolation function as shown in FIG. The obtained interpolation information for each frame, that is, for example, as shown in FIG. 8, each interpolation pixel value x of frame 1 and frame 2 is shown as pixel information in the vertical (horizontal) direction, as shown in FIG. Using such an unequal interpolation function, unequal interpolation is performed based on the amount of frame shift, and pixel information at a desired position ◯ in frame 1 is determined as shown in FIG.

  In this way, by tracking the video corresponding points between frames, expressing the time transition as a function, and generating a function interpolation frame with the ratio of the original frame and the number of frames to be converted, the number of frames can be increased or decreased. A clear and smooth motion video signal can be obtained, and a clear and smooth motion video display can be performed at a frame rate suitable for the display.

  A conventional high-quality method of generating a halftone value of a corresponding point by half-matching by block matching using motion information obtained by half-precision motion estimation by generating an interpolated frame F (k + 1/2) by uniform interpolation. In the frame rate conversion process, a part having a motion is deteriorated in the inserted interpolated frame image. Like the frame rate conversion apparatus 1, the gradation value of the corresponding point estimated by performing the corresponding point estimation process is used. In the high frame rate processing that generates the gradation value of the corresponding point of the interpolated frame by uniform interpolation and further generates the gradation value of the corresponding point of the interpolated frame by non-uniform interpolation, the moving part deteriorates. It was possible to achieve a higher frame rate.

  Here, the frame rate conversion apparatus 1 may have a function of performing the high frame rate processing as described above and performing an enlargement interpolation process using a two-frame image. For example, as shown in FIG. 9, for example, the function of performing the enlargement interpolation process using the two-frame image is configured by an input data control circuit 51, an output synchronization signal generation circuit 52, an SRAM 53, an SRAM selection unit 54, and an image processing module 55. This is realized by the enlarged interpolation processing device 50.

  In the enlargement interpolation processing device 50, the input data control circuit 51 performs control to sequentially input the input image supplied together with the horizontal synchronization signal and the vertical synchronization signal, that is, image information of each pixel, to the SRAM selection unit 54.

  The output synchronization signal generation circuit 52 generates an output side synchronization signal based on the supplied horizontal synchronization signal and vertical synchronization signal, outputs the generated output side synchronization signal, and supplies it to the SRAM selection unit 54.

  The SRAM selection unit 54 is configured as shown in FIG. 10, for example, and corresponds to the memory selection signal supplied from the control signal switching circuit 54A based on the write control signal and the read control signal generated in the supplied synchronization signal. The input data input via the input data control circuit 51 is stored in the SRAM 53 frame by frame by the write data selection unit 54B and the read data selection unit 54C that perform the above operations, and two frames of images are simultaneously output to the output synchronization signal generation circuit. The data is read out in synchronization with the output side synchronization signal generated by 52.

  Further, the image processing module 55 is configured as shown in FIG. 11, for example, which performs image interpolation processing based on inter-frame information.

  That is, the image processing module 55 receives the window setting unit 55A, the first equal interpolation processing unit 55B, and the second equal interpolation from which two frames of image information simultaneously read from the SRAM 53 via the SRAM selection unit 54 are input. A shift amount estimation unit 55D to which pixel information extracted from the image information of the two frames is input by the processing unit 55C, the window setting unit 55A, the shift amount vector estimated by the shift amount estimation unit 55D, and the second Information on the pixel interpolated by the uniform interpolation processing unit 55C, information on the pixel corrected by the deviation correction unit 55E, and information on the pixel interpolated by the first interpolation processing unit 55B. The input non-uniform interpolation processing unit 55F is included.

  In the image processing module 55, as shown in FIGS. 12A and 12B, a predetermined point (p) is set by the window setting unit 55A for two frame images f and g input via the SRAM selection unit 54. , Q), the window of one frame image g is shifted by the shift amount (τx, τy) by the shift amount estimation unit 55D, and the inner product is calculated by the pixel value of the relative position (x, y) in the window. An operation is performed and the value is set as a cross-correlation value Rpq (τx, τy).

  Then, the shift amount (τx, τy) that maximizes the cross-correlation value Rpq (τx, τy) around the point (p, q) is extracted by changing the shift amount (τx, τy).

  The cross-correlation values Rpq (τx, τy) can also be obtained by performing Fourier transform on the pixel data in the windows of the two frame images f and g, respectively.

  In the enlargement interpolation processing device 50, enlargement interpolation processing is performed according to the procedure as shown in the flowchart of FIG.

  That is, in the image processing module 55, when the two frame images f and g are read from the SRAM 53 via the SRAM selection unit 54 (step A), the two frame images f and g are obtained by the correlation calculation processing in the deviation amount estimation unit 55D. The shift amount (τx, τy) is calculated (step B).

  Then, the interpolated pixel value obtained by the uniform interpolation for the image f of the frame 1 is calculated by the first uniform interpolation processing unit 55B, thereby expanding in the horizontal direction or the vertical direction (step C).

  Further, the interpolation pixel value obtained by uniform interpolation for the image g of frame 2 is calculated by the second uniform interpolation processing unit 55C, thereby expanding in the horizontal direction or the vertical direction (step D).

  Further, the pixel value at the pixel position obtained by moving the enlarged image of frame 2 by the amount of displacement relative to frame 1 is calculated by the displacement amount correction unit 55E (step E).

  Then, in the non-uniform interpolation processing unit 55F, non-uniform interpolation is performed on the pixel value at the position to be obtained in frame 1 from a total of four pixel values, that is, two interpolated pixel values in frame 1 and two pixel values in the moving position of frame 2. Then, enlargement calculation is performed in the vertical direction or horizontal direction (step F), and the interpolation calculation result of frame 1 is output as an enlarged image (step G).

  The frame rate conversion apparatus 110 having a function of performing such an enlargement interpolation process is configured as shown in FIG. 14, for example.

  The frame rate conversion apparatus 110 includes a computer that functions as a first function approximation processing unit 111, a corresponding point estimation processing unit 112, a second function approximation processing unit 113, and a third function approximation processing unit 114.

  The first function approximation processing unit 111 performs a first function approximation process for approximating the density distribution of a plurality of pixels in the reference frame.

  Corresponding point estimation processing unit 112 performs a correlation operation with functions of the gray distribution in the plurality of reference frames having different times approximated by the first function approximating unit 111, and provides each position that gives the maximum value. Corresponding point estimation processing is performed with corresponding point positions corresponding to the plurality of reference frames.

  The second function approximation processing unit 113 coordinates the corresponding point position in each reference frame estimated by the corresponding point estimation unit 112 by the horizontal and vertical distances from the origin of the reference frame, and changes the time. Each change in the horizontal position and the vertical position of the coordinate point in a plurality of reference frames is converted into a time series signal, and second function approximation is performed to approximate the time series signal of each reference frame.

  The third function approximation processing unit 114 corresponds to the corresponding point position of the reference frame for the interpolated frame at any time between the plurality of reference frames by the function approximated by the second function approximation unit 113. The corresponding position in the interpolation frame is set as the corresponding point position, and the gray value at the corresponding point position in the interpolation frame is obtained by interpolation with the gray value at the corresponding point in the reference frame. At the same time, the first function approximation is applied to obtain a gray level distribution near the corresponding point, and a third function approximation process for converting the gray value near the corresponding point into the gray value of the pixel point in the interpolation frame is performed.

  In the frame rate conversion apparatus 110, the first function approximation processing unit 111 performs function approximation of the density distribution of a plurality of pixels in the reference frame, and the corresponding point estimation processing unit 112 performs the first function approximation processing. The correlation calculation is performed with the function of the gray distribution in the plurality of reference frames having different times approximated by the unit 111, and each position giving the maximum value is set as the corresponding point position corresponding to the plurality of reference frames, The second function approximation processing unit 113 coordinates the corresponding point position in each reference frame estimated by the corresponding point estimation processing unit 112 with the horizontal and vertical distances from the origin of the reference frame, and changes the time. Convert the changes in the horizontal position and vertical position of the coordinate point in multiple reference frames to a time-series signal. The time series signal of the reference frame to function approximation. Then, with the function approximated by the second tone value generation processing unit 113 by the third function approximation processing unit 114, the corresponding point of the reference frame for the interpolation frame at any time between the plurality of reference frames. The corresponding position in the interpolation frame corresponding to the position is set as the corresponding point position, and the gray value at the corresponding point position of the interpolation frame is obtained by interpolation with the gray value at the corresponding point of the reference frame. By applying the first function approximation in accordance with the gray level value of the image, the gray level distribution in the vicinity of the corresponding point is obtained, and the gray level value in the vicinity of the corresponding point is converted into the gray level value of the pixel point in the interpolation frame. A high frame rate processing is performed together with the processing.

  The present invention is applied to, for example, a video signal conversion system 100 configured as shown in FIG. 15, and the frame rate conversion device 1 is mounted on the video signal conversion system 100 as the high frame rate processing unit 40.

  The video signal conversion system 100 includes a preprocessing unit 20 that performs noise removal processing on image information input from an image input unit 10 such as an imaging device, and image information that has been subjected to noise removal processing by the preprocessing unit 20 is input. A compression encoding processing unit 30 that compresses and encodes the input image information, a high frame rate processing unit 40 that increases the image information compressed and encoded by the compression encoding processing unit 30, and the like. .

  The pre-processing unit 20 in the video signal conversion system 100 performs a filtering process to remove noise such as blur and camera shake included in the input image information by an image tensor calculation technique and a blur function adaptive correction process technique. Then, the output of the deterioration model 21 of the blur function H (x, y) to which the true input image f (x, y) is input by the system model as shown in FIG.

Is added with noise n (x, y) to obtain an observed image g (x, y), and estimated by using the observed image g (x, y) as an input by a restoration system model as shown in FIG. Images

It consists of the inverse filter 22 which obtains.

  The preprocessing unit 20 performs filtering processing to be removed by an image tensor calculation technique and a blur function adaptive correction technique, and evaluates an original image using characteristics of a Kronecker product.

  The Kronecker product is defined as:

When A = [a _ij ] is an mn matrix and B = [b _ij ] is an st matrix, the Kronecker product

Is an ms × nt matrix as follows.

  here,

Represents the Kronecker product operator.

  The basic properties of the Kronecker product are as follows.

  here,

Is an operator indicating an operation of generating a column vector by extending the matrix in the column direction.

  In the image model in the preprocessing unit 20, it is assumed that an unknown true input image f (x, y) exists, and the output of the degradation model 21 is output.

An observation image g (x, y) obtained by adding noise n (x, y) to can be expressed by the following equation (1).

  here,

Represents a deteriorated image obtained by this image system, and n (x, y) is added noise. And degraded image

Is represented by the following equation (2).

  Here, h (x, y; x ′, y ′) represents the impulse response of the deteriorated system.

  Since the image used is a discrete quantity, the image model of the input image f (x, y) can be rewritten as shown in Expression (3).

Here, H _k ^(x) H _l ^(y) becomes a point spread function (PSF) H of the degradation model by expressing it in matrix form as in the following equation (4).

  The characteristics of the inverse filter 22 are determined by a learning process according to the procedure shown in the flowchart of FIG.

  That is, in the learning process, first, the observation image g (x, y) is read as the input image g (step S11a),

As image g _E (step S12a),

Singular value decomposition (SVD) is performed (step S13a).

  Further, a point spread function (PSF) H of the degradation model is read (step S11b),

A degradation model represented by the Kronecker product is constructed (step S12b), and a singular value decomposition (SVD) of the function H of the degradation model is performed (step S13b).

  Where the system equation g is

Can be rewritten.

  And

As a new image g _KPA is calculated (step S14).

And about the calculated new image g _KPA ,

(Step S15), and for the obtained f _K ,

It is determined whether or not the test condition is satisfied (step S16).

  Here, k is a repetition number, and ε and c are thresholds for determination, respectively.

The determination result in step S16 is Faise, i.e., when f _K obtained in step S15 does not satisfy the above test conditions, the function H of the deterioration model,

(Step S17), the process returns to step S13b, singular value decomposition (SVD) is performed on the function HK _{+ 1} obtained in step S16, and the processes from step S13b to step S17 are performed. repeatedly performs the determination result in step S16 is True, that is, when f _K obtained in the step S15 is the test condition is satisfied, the f _K obtained in step S15

(Step S18), the learning process for one input image g is terminated.

  The characteristics of the inverse filter 22 are determined by performing the learning process on a large number of input images g.

  That is, here, h (x, y) * f (x, y) is represented as Hf, and the system equation is

And also

As, by approximating the f, and derives a new image g _E of interest as follows.

Here, E indicates prediction. The new image g _E is configured as preservation and enhancement of edge details of the original image.

The new image g _E

As obtained. Here, _CEP and _CEN are edge preservation and edge enhancement operators, respectively.

And select a simple Laplacian kernel C _EN = ∇ ² f and a Gaussian kernel C _EP with control parameters β and γ,

And

  And

  And reconstruct the minimization problem from the following singular value decomposition (SVD)

Function H of the above degradation model

Estimate and use.

  Like the pre-processing unit 20 in the video signal conversion system 100, a filtering process is performed to remove noise such as blur and camera shake included in the input image information by an image tensor calculation technique and a blur function adaptive correction process technique. Thus, noise can be removed and image sharpening and edge enhancement can be performed.

  The video signal conversion system 100 compresses and encodes the image information subjected to the noise removal processing by the preprocessing unit 20 by the compression encoding processing unit 30, and converts the compression encoded image information to the frame rate conversion processing unit 40. To increase the frame rate.

  The compression coding processing unit 30 in the video signal conversion system 100 performs compression coding processing based on the fluency theory. As shown in FIG. 19, the first functioning processing unit 31 and the second functioning processing are performed. Unit 32, and an encoding processing unit 33 for describing and encoding each piece of image information functioned by the first functioning processing unit 31 and the second functioning processing unit 32 in a predetermined format.

  The first function processing unit 31 includes a corresponding point estimation unit 31A that performs corresponding point estimation between a plurality of frame images on the image information that has been subjected to noise removal processing by the preprocessing unit 20, and the corresponding point estimation unit. It comprises a motion function processing unit 31B for converting the image information of the motion part using the image information of the corresponding point of each frame image estimated by 31A.

  The corresponding point estimation unit 31A is configured as shown in FIG. 20, for example.

  That is, the corresponding point estimation unit 31A includes a first partial region extraction unit 311 that extracts a partial region of a frame image and other consecutive frame images similar to the partial region extracted by the first partial region extraction unit 311. A second partial region extraction unit 312 that extracts the partial regions of the first partial region extraction unit 311 and the partial regions extracted by the first partial region extraction unit 311 and the second partial region extraction unit 312 to the same ratio, A function approximating unit 313 for expressing and outputting the shade of each image by a piecewise polynomial according to the fluency theory, a correlation value calculating unit 314 for calculating a correlation value of the output of the function approximating unit 313, and the correlation value calculating unit 314 The shift amount calculation unit 315 calculates the position shift of the image that gives the maximum correlation value calculated by the above and outputs the calculated value as the shift amount of the corresponding point.

  In the corresponding point estimation unit 31A, the first partial region extraction unit 311 extracts the partial region of the frame image as a template, and other continuous similarities to the partial region extracted by the first partial region extraction unit 311. A partial region of the frame image is extracted by the second partial region extraction unit 312, and each partial region extracted by the first partial region extraction unit 311 and the second partial region extraction unit 312 by the function approximation unit 313 is extracted. The image is converted to the same ratio, and the density of each converted image is expressed as a function with a piecewise polynomial.

  The corresponding point estimation unit 31A captures the lightness and darkness of the image as a continuous change state, and estimates the corresponding point of the image by fluency information theory. The corresponding point estimation unit 31A and the second partial region extraction unit 311 It consists of a partial area extraction unit 312, a function approximation unit 313, a correlation value calculation unit 314, and a deviation amount calculation unit 315.

  In the corresponding point estimation unit 31A, the first partial region extraction unit 311 extracts a partial region of the frame image from the input image.

  The second partial region extraction unit 312 extracts partial regions of other continuous frame images similar to the partial region extracted by the first partial region extraction unit 311.

  The function approximating unit 313 converts the partial regions extracted by the first partial region extracting unit 311 and the second partial region extracting unit 312 into the same ratio, and the density of each converted image is converted to a fluency theory. According to, function representation with piecewise polynomial is output.

  The correlation value calculation unit 314 calculates the correlation value of the output of the function approximation unit 313.

  Further, the deviation amount calculation unit 315 calculates the positional deviation of the image that gives the maximum correlation value calculated by the correlation value calculation unit 314, and outputs the calculated value as the deviation amount of the corresponding point.

  In the corresponding point estimation unit 31, the first partial region extraction unit 311 extracts a partial region of the frame image as a template, and is continuous similar to the partial region extracted by the first partial region extraction unit 311. The partial areas of the other frame images are extracted by the second partial area extraction unit 312, and the respective parts extracted by the function approximation unit 313 by the first partial region extraction unit 311 and the second partial region extraction unit 312. The area is converted to the same ratio, and the density of each converted image is expressed as a function with a piecewise polynomial.

Here, it is assumed that the images f ₁ (x, y) and f ₂ (x, y) belong to the space S ^(m) (R ² ), and φm (t) is (m−2) th order It is expressed as the following equation (5) by a piecewise polynomial,

When the space S ^(m) (R ² ) is expressed as the following equation (6),

The correlation function c (τ ₁ , τ ₂ ) between frames can be expressed as the following equation (7).

And the above assumption, ie,

From Equation (7), the frame correlation function can be expressed by the following Equation (8).

That is, the correlation function c (τ _1, τ ₂₎ between the frame belongs to the space ^{S (2m) (R 2)} for performing 2m following interpolation as shown in FIG. 21, space performs the 2m order interpolation ^{S ( 2m)} (sampling frequency [psi _2m of _{^{R 2) (τ 1, τ}} 2) is present uniquely, the correlation function c (tau ₁ between the frame, tau _2), at the following equation (9) expressed.

  From equation (8), a (2m-1) th order piecewise polynomial function can be constructed to interpolate the correlation surface.

  That is, the block-based motion vector estimation approach appropriately obtains an initial estimate of the separate block motion vectors in equation (7), and then applies equation (8) to obtain real motion of arbitrary accuracy. .

  The general form of the separable correlation surface interpolation function is expressed by equation (10).

Here, Ck and dl are interpolation coefficients, and M _2m (x) = φ _{2 m (} x + 2) · φ _m (x) is an (m−1) -order B-spline.

The correlation function c (τ ₁ , τ ₂ ) can be approximated by the following equation (11) by the appropriate stroke limit in the equation (10).

Here, K ₁ = [τ ₁ ] −s + 1, K ₂ = [τ ₂ ] + s, L ₁ = [τ ₂ ] −s + 1, L ₂ = [τ ₂ ] + s, and s is φ _m (x) Determine.

  Then, for example, when m = 2, a desired interpolation formula is obtained by substituting the following formula (12) into formula (11).

  The motion vector v is derived using the following equation (13).

The correlation function c (τ ₁ , τ ₂ ) can be reproduced using only integer point information, and the correlation value calculation unit 314 uses the correlation function c (τ ₁ , τ ₂ ) to calculate the function approximation unit. The correlation value of the output of 313 is calculated.

  Then, the deviation amount calculation unit 315 calculates the motion vector V by the equation (13) indicating the positional deviation of the image that gives the maximum value of the correlation value calculated by the correlation value calculation unit 314, and obtains the obtained motion vector V. Output as the deviation of the corresponding point.

  Here, FIG. 22 schematically shows how the motion vector V is determined by the corresponding point estimation by the corresponding point estimation unit 31A.

  That is, in the corresponding point estimation unit 31A, as shown in FIG. 22A, a partial area of the frame image (k) is extracted, and partial areas of other continuous frame images similar to the partial area are extracted. As shown in FIG.

The correlation between the frames is calculated using the correlation function c (τ ₁ , τ ₂ ) expressed by the following, and the motion is detected at the peak point of the correlation surface as shown in FIG. By obtaining the motion vector v by the above equation (13), the motion of the pixel in the frame image (k) is determined as shown in FIG.

  The motion vector of each block of the frame image (k) determined as described above is obtained by comparing the motion vector of each block of the same frame image (k) with that determined by the conventional block matching. It will change smoothly.

That is, for example, as shown in FIG. 23 (A), when frame 1 and frame 2 in which the subject rotates, quadruple enlargement is performed by 2-frame corresponding point estimation and non-uniform interpolation. As shown in (B1) and (C1), the motion vector estimated at the estimated corresponding point by the conventional block matching has a portion where the change is not smooth. However, the corresponding point estimating unit 31A having the above-described configuration The change in the motion vector estimated at the estimated corresponding point is generally smooth as shown in (B2) and (C2) of FIG. In addition, the amount of calculation with an accuracy of 1 / N is N ² in the present method, while N ² in the conventional method.

  Then, the motion function processing unit 31B functions the image information of the motion part using the motion vector V obtained by the corresponding point estimation in the corresponding point estimation unit 31A.

  That is, in the motion function processing unit 31B, when the corresponding point of the partial moving image is estimated for each reference frame, the movement amount, that is, the shift amount of the corresponding point corresponds to the change in the coordinate position x, y of the frame. As shown in FIG. 24, when the origin of the frame is taken at the upper left corner, for example, the movement of the image of each frame as shown in FIG. 25A is shown in FIGS. As described above, the movement of the X coordinate and the Y coordinate of each frame is expressed, and the change of the movement of each of the X coordinate and the Y coordinate is approximated by a function to make a function. Then, as shown in FIG. 26, motion compensation is performed by interpolating with the function and estimating the position between frames.

  The second functioning processing unit 32 encodes an input image by fluency functioning processing that approximates contour, shading, and interframe information based on the fluency information theory, and includes an automatic region classification processing unit. 32A, an outline function approximation processing unit 32B, a tone function conversion processing unit 32C, a frequency function approximation processing unit 32D, and the like.

  Based on the fluency information theory, the area automatic classification processing unit 32A uses a piecewise plane area (m ≦ 2), a piecewise curved surface area (m = 3), a piecewise spherical area (m = ∞), an irregularity. Classify into regions (m ≧ 4).

  In the fluency information theory, signals are classified into classes specified by the order m in the concept of signal space.

The signal space ^m S is represented by a (m−1) th order piecewise polynomial with a variable that can be continuously differentiated (m−2) times.

It has been proved that the signal space ^m S is equal to that of the step function when m = 1, and equal to that of the Fourier power function when m = ∞. The fluency model is a model that clarifies the relationship between signals belonging to the signal space ^m S and discrete-time signals by defining a fluency sampling function.

  The contour function approximation processing unit 32B includes an automatic contour classification processing unit 321 and a function approximation processing unit 322, and includes a piecewise plane area (m ≦ 2) and a piecewise curved surface area (m = 3) The straight line, arc, and quadratic curve included in the piecewise spherical area (m = ∞) are extracted by the contour automatic classification processing unit 321 and approximated by the function by the function approximation processing unit 322.

  The gradation function processing unit 32C is configured to perform the piecewise plane region (m ≦ 2), the piecewise curved surface region (m = 3), and the piecewise spherical region (m = ∞) classified by the region automatic classification processing unit 32A. Perform density function using fluency function.

  The frequency function approximation processing unit 32D performs frequency function approximation processing by DCT or the like on the irregular regions (m ≧ 4) classified by the region automatic classification processing unit 2, that is, regions that cannot be expressed by a polynomial.

  The second function processing unit 32 can express the shading and contour of an image using a number of multivariable fluency functions for each frame of the video.

  Then, the encoding processing unit 33 describes and encodes each image information functioned by the first functioning processing unit 31 and the second functioning processing unit 32 in a predetermined format.

  In the video signal conversion system 100, as described above, the image information input from the image input unit 10 such as the imaging device is subjected to noise removal processing by the preprocessing unit 20, and the noise removal processing is performed by the preprocessing unit 20. The applied image information is compression-encoded by the compression-encoding processor 30, and the high-frame-rate processor 40 using the frame rate conversion device 1 tracks video corresponding points between frames, and the time transition is a function. The function interpolation frame is generated by expressing the ratio of the original frame and the number of frames to be converted.

  That is, the video signal conversion system 100 expresses a contour or the like using a number of fluency functions for each video frame, and expresses a discrete frame sequence by a continuous function based on a piecewise polynomial in the time direction, thereby allowing an arbitrary frame rate. High-definition video can be played back.

  In fluency information theory, the signal space in the class specified by the order m is classified based on the frequency that the signal is continuously differentiable.

For any m> 2, subspace spanned is expressed by a (m−1) -order piecewise polynomial that can be continuously differentiated only (m−2) times.
The sampling function ψ (x) of the (m = 3) class is expressed by the following equation (14) by linear combination of second-order piecewise polynomials that can be continuously differentiated only once.

  Here, φ (x) is expressed by the following equation (15).

  Since ψ (x) is a sampling function, a section function can be obtained by a sampling sequence and a convolution operation.

Here, when τ = 1, the equation (14) can be expressed as a piecewise polynomial given by the following equation (16).

  For example, (m = 3) class of unequal fluency interpolation functions

Is a function as shown in FIG.

  Non-uniform interpolation fluency function

Is composed of 8 partial polynomials of frequency 2, and the unequal interpolation fluency function of (m = 3) class is an unequal interval specified by s ₁ (x) to S ₈ (x) as shown in FIG. The component is given by the following equation (17).

  here,

It is.

  Here, FIG. 28 shows an example of high resolution interpolation.

  FIG. 29 shows a specific example of a pixel structure for interpolation.

  In FIG. 29, the pixel of Frame_1 is a different motion vector that changes the pixel in Frame_2.

have.

  FIG. 28 illustrates the concept of one-dimensional image interpolation from two consecutive frames.

  The motion estimation is based on an all search block matching algorithm whose block size and search window size are known.

The high resolution frame pixel is represented by f (τ _x , τ _y ), and the pixel structure is as shown in one example of the high resolution interpolation approach of FIG.

In the first step, two consecutive frames are obtained from the video sequence and represented by f ₁ (x, y) and f ₂ (x, y).

  In the second step, an initial estimation of the motion vector is performed.

The initial estimation of the motion vector is performed.

  here,

It is.

  In equation (18):

Represents the average of the search window, and

Represents the average of the current blocks in the matching.

  In the third step, all pixels using equations (12) and (17)

For the motion vector from the second step

A motion vector is obtained from one nearby pixel.

  In the fourth step, uniform horizontal interpolation is performed as follows.

  In the fifth step, non-uniform vertical interpolation using the pixels obtained in the fourth step is executed according to equation (20).

  The fourth and fifth steps are repeated for all pixels in the high resolution image.

  In moving picture coding based on the fluency theory, a signal space suitable for the original signal is selected, and the function is converted into a highly compressed image while preserving sharpness.

  A motion vector can be obtained with arbitrary accuracy by accurately determining a function space to which a correlation function between frames belongs based on the fluency theory.

  In moving picture coding based on the fluency theory, a signal space suitable for the original signal is selected, and the function is converted into a highly compressed image while preserving sharpness.

  In this way, by tracking the video corresponding points between frames, expressing the time transition as a function, and generating a function interpolation frame with the ratio of the original frame and the number of frames to be converted, the number of frames can be increased or decreased. A clear and smooth motion video signal can be obtained, and a clear and smooth motion video display can be performed at a frame rate suitable for the display.

  For example, as shown in FIG. 30A, when generating a frame at an arbitrary time between the frame k and the frame k + 1, the interpolation frame F (k + 1/2) is generated by equal interpolation, and the ½ precision is obtained. In the conventional high frame rate processing in which the gradation value of the corresponding point is generated with the half accuracy by block matching using the motion information obtained by the motion estimation, as shown in (B1) and (C1) of FIG. In the interpolated frame image to be inserted, the moving portion is deteriorated, but as in the high frame rate processing unit 40, uniform interpolation is performed using the gradation value of the corresponding point estimated by performing the corresponding point estimation process. 30 generates the gradation value of the corresponding point of the interpolation frame, and further generates the gradation value of the corresponding point of the interpolation frame by non-uniform interpolation, (B2) and (C2) in FIG. As shown, it could be higher frame rate without the parts of the motion deteriorates.

  In the video signal conversion system 100, image information input from the image input unit 10 such as an imaging device is subjected to noise removal processing by the preprocessing unit 20, and the image information subjected to noise removal processing by the preprocessing unit 20 is processed. The compression encoding processing unit 30 performs compression encoding, and the high frame rate processing unit 40 tracks video correspondence points between frames, expresses the time transition as a function, and compares the ratio between the original frame and the number of frames to be converted. By generating a function interpolation frame, the image information compression-encoded by the compression-encoding processor 30 is increased in frame rate, so that a video signal with clear and smooth motion can be obtained.

It is a block diagram which shows the structural example of a frame rate conversion apparatus. It is a figure which shows typically the frame rate increase process by the said frame rate conversion apparatus. It is a flowchart which shows the execution procedure of the high frame rate process by the said frame rate conversion apparatus. It is a figure which shows typically the content of the frame rate increase process by the said frame rate conversion apparatus. It is a figure where it uses for description of the non-uniform interpolation process in the said frame rate conversion apparatus. It is a figure which uses for description of the image interpolation process which determines the value of the pixel position newly produced | generated when converting the resolution of an image. It is a figure which shows the example of a uniform interpolation function and a nonuniform interpolation function. It is a figure which shows the content of the said image interpolation process typically. It is a block diagram which shows the structural example of an expansion interpolation processing apparatus. It is a block diagram which shows the structural example of the SRAM selection part in the said expansion interpolation processing apparatus. It is a block diagram which shows the structural example of the image processing block in the said expansion interpolation processing apparatus. It is a figure which shows typically two frame images input into the image processing module in the said expansion interpolation processing apparatus. It is a flowchart which shows the procedure of the expansion interpolation process in the said expansion interpolation processing apparatus. It is a block diagram which shows the structural example of the frame rate conversion apparatus which has an expansion interpolation process function. It is a block diagram which shows the structure of the video signal conversion system to which this invention is applied. It is a block diagram which shows the system model used in order to construct | assemble the pre-processing part in the said video signal conversion system. It is a block diagram which shows the restoration system model used in order to construct | assemble the pre-processing part in the said video signal conversion system. It is a flowchart which shows the procedure of each process of the characteristic of the inverse filter used for the said pre-processing part. It is a block diagram which shows the structure of the compression encoding process part in the said video signal conversion system. It is a block diagram which shows the structure of the corresponding point estimation part with which the said compression encoding process part was equipped. It is a figure where it uses for description of the space which performs 2m order interpolation to which the correlation function between frames belongs. It is a figure which shows typically the mode of determination of the motion vector by the corresponding point estimation by the said corresponding point estimation part. It is a figure which compares and shows the motion vector determined by the corresponding point estimation by the said corresponding point estimation part, and the motion vector determined by the conventional block matching. It is a figure where it uses for description of the origin of the frame image handled by the motion function processing part with which the said compression encoding process part was equipped. It is a figure which shows typically the motion of the image of each frame as a motion of the X coordinate of each frame, and a Y coordinate. It is a figure which shows typically the content of the estimation process of the position between frames. It is a figure which shows the unequal fluency interpolation function of a class (m = 3). It is a figure which shows the example of a high resolution interpolation approach. It is a figure which shows the specific example of the pixel structure for interpolation. It is a figure which compares and shows the intermediate | middle frame produced | generated by the said high frame rate process part, and the intermediate | middle frame produced | generated by the conventional method.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Frame rate conversion apparatus, 2 Corresponding point estimation process part, 3 1st gradation value generation process part, 4 2nd gradation value generation process part, 5 3rd gradation value generation process part 5, 10 Image input , 20 preprocessing unit, 21 degradation model, 22 inverse filter, 30 compression encoding processing unit, 40 high frame rate conversion processing unit, 100 video signal conversion system, 110 frame rate conversion device, 111 first function approximation processing unit , 112 corresponding point estimation processing unit, 113 second function approximation processing unit, 114 third function approximation processing unit

Claims (6)

For a plurality of pixels in the reference frame, the gray value at each pixel point is expressed by a continuous function of the position, and the position that gives the maximum correlation between the above-mentioned function approximate image gray levels in a plurality of image frames at different times A corresponding point estimation processing unit that estimates as
A first gradation value generation processing unit that obtains a gradation value at a corresponding position from a gradation value indicating a gradation of a neighboring pixel point for the estimated gradation at each corresponding position in each image frame;
For an interpolated frame generated at a frame rate ratio to be converted between the image frames, a corresponding point trajectory from a plurality of pixels in the reference frame from tone values estimated at corresponding positions in the image frames. A second gradation value generation processing unit that approximates the upper and lower shades with a fluency function, and obtains each gradation value of the corresponding position in the interpolation frame from the function;
A frame rate conversion comprising: a third gradation value generation processing unit configured to generate a gradation value of each pixel in the vicinity of the corresponding position in the interpolation frame from the gradation value of each corresponding position in the interpolation frame. apparatus. For a plurality of pixels in the reference frame, the gray value at each pixel point is expressed by a continuous function of the position, and the position that gives the maximum correlation between the above-mentioned function approximate image gray levels in a plurality of image frames at different times Corresponding point estimation processing step to be estimated as
A first gradation value generation processing step for obtaining a gradation value at a corresponding position from a gradation value indicating the gradation of a neighboring pixel point for the estimated gradation at each corresponding position in each image frame;
For the interpolated frames generated at the frame rate ratio to be converted between the above image frames,
For a plurality of pixels in the reference frame, the gray level on the corresponding point locus is approximated by a fluency function from the gradation value of each corresponding position estimated in each image frame, and each level of the corresponding position in the interpolation frame is calculated from the function. A second tone value generation processing step for obtaining a tone value;
And a third gradation value generation processing step for generating a gradation value of each pixel in the vicinity of the corresponding position in the interpolation frame from the gradation value of each corresponding position in the interpolation frame. Method. A frame rate conversion program executed by a computer provided in the frame rate conversion apparatus,
For a plurality of pixels in the reference frame, the gray value at each pixel point is expressed by a continuous function of the position, and the position that gives the maximum correlation between the above-mentioned function approximate image gray levels in a plurality of image frames at different times A corresponding point estimation processing unit that estimates as
A first gradation value generation processing unit that obtains a gradation value at a corresponding position from a gradation value indicating a gradation of a neighboring pixel point for the estimated gradation at each corresponding position in each image frame;
For the interpolated frames generated at the frame rate ratio to be converted between the above image frames,
For a plurality of pixels in the reference frame, the gray level on the corresponding point locus is approximated by a fluency function from the gradation value of each corresponding position estimated in each image frame, and each level of the corresponding position in the interpolation frame is calculated from the function. A second tone value generation processing unit for obtaining a tone value;
As a third gradation value generation processing unit that generates the gradation value of each pixel near the corresponding position in the interpolation frame from the gradation value of each corresponding position in the interpolation frame,
A frame rate conversion program for causing the computer to function. First function approximating means for approximating the density distribution of a plurality of pixels in the reference frame with a continuous function of pixel positions;
Correlation calculation is performed between the function approximated by the first function approximating means and the function of the density distribution in the plurality of image frames at different times, and each position that gives the maximum value is set to the plurality of image frames. Corresponding point estimation means for corresponding corresponding point positions in
The corresponding point position in each image frame estimated by the corresponding point estimation means is coordinated by the horizontal and vertical distances from the origin of each image frame, and the coordinate point in the plurality of image frames having different times is used. A second function approximation means for converting each change in the horizontal position and the vertical position into a time-series signal and approximating the time-series signal with a function;
With respect to the interpolation frame at any time between the plurality of image frames, the corresponding position in the interpolation frame corresponding to the corresponding point position of the image frame is determined by the function approximated by the second function approximation means. The gray value at the corresponding point position of the interpolated frame is obtained by interpolation with the gray value at the corresponding point of the image frame, and the first function approximation is applied according to the gray value of the corresponding point of the interpolated frame. A frame rate conversion device comprising: a third function approximating unit that obtains a gray level distribution in the vicinity of the corresponding point and converts a gray level value in the vicinity of the corresponding point into a gray level value of a pixel point in the interpolation frame. A first function approximating step for approximating the density distribution of a plurality of pixels in the reference frame with a continuous function of pixel positions;
A correlation operation is performed between the function approximated in the first function approximation step and the function of the gray distribution in a plurality of image frames at different times, and each position giving the maximum value is defined as the plurality of image frames. A corresponding point estimation step as a corresponding point position in FIG.
The corresponding point position in each image frame estimated in the corresponding point estimation step is coordinated by the horizontal and vertical distances from the origin of each image frame, and the coordinate points of the plurality of image frames having different times are coordinated. A second function approximation step of converting each change in the horizontal position and the vertical position into a time series signal and approximating the time series signal to a function;
With respect to the interpolation frame at any time between the plurality of image frames, the corresponding position in the interpolation frame corresponding to the corresponding point position of the image frame is determined by the function approximated in the second function approximation step. The gray value at the corresponding point position of the interpolated frame is obtained by interpolation with the gray value at the corresponding point of the image frame, and the first function approximation is applied according to the gray value of the corresponding point of the interpolated frame. A frame rate conversion method comprising: a third function approximating step for obtaining a gray level distribution in the vicinity of the corresponding point and converting the gray level value in the vicinity of the corresponding point into a gray level value of the pixel point in the interpolation frame. A frame rate conversion program executed by a computer provided in the frame rate conversion apparatus,
First function approximating means for approximating the density distribution of a plurality of pixels in the reference frame with a continuous function of pixel positions;
Correlation calculation is performed between the function approximated by the first function approximating means and the function of the density distribution in the plurality of image frames at different times, and each position that gives the maximum value is set to the plurality of image frames. Corresponding point estimation means for corresponding corresponding point positions in
The corresponding point position in each image frame estimated by the corresponding point estimation means is coordinated by the horizontal and vertical distances from the origin of each image frame, and the coordinate point in the plurality of image frames having different times is used. A second function approximation means for converting each change in the horizontal position and the vertical position into a time-series signal and approximating the time-series signal with a function;
With respect to the interpolation frame at any time between the plurality of image frames, the corresponding position in the interpolation frame corresponding to the corresponding point position of the image frame is determined by the function approximated by the second function approximation means. The gray value at the corresponding point position of the interpolated frame is obtained by interpolation with the gray value at the corresponding point of the image frame, and the first function approximation is applied according to the gray value of the corresponding point of the interpolated frame. As a third function approximating means for obtaining a density distribution in the vicinity of the corresponding point and converting the gray value in the vicinity of the corresponding point into a gray value of the pixel point in the interpolation frame,
A frame rate conversion program for causing the computer to function.

Images