JP5118087B2 - Frame rate conversion method, frame rate conversion device, and frame rate conversion program - Google Patents

Description

  The present invention relates to a frame rate conversion method and apparatus for converting a frame of a high frame rate video signal into a low frame rate video signal by downsampling based on filtering, and a frame rate conversion program used for realizing the frame rate conversion method. About.

  In recent years, there are growing expectations for super-high-quality images such as large-screen sports images and digital cinema that are full of realism. In response to this, research on high-quality video has been vigorously conducted.

  The following four elements are necessary to realize super high-quality video. That is, spatial resolution, pixel value depth, color reproducibility, and temporal resolution. In response, the former three elements are being studied in applications such as digital cinema and natural vision projects. In addition, improvement of time resolution, that is, an increase in video frame rate, which is indispensable for expressing a natural movement of a subject, has been studied.

  The upper limit of the frame rate of the video input / output system is asymmetric. At present, a high-speed camera capable of capturing a high frame rate video exceeding 1000 [frame / sec] is available as an imaging system. However, video captured by such a high-speed camera is used for slow playback applications. On the other hand, since the upper limit of the current display is about 120 [fps], a video source shot with a high-speed camera needs to be thinned in a display form for real-time playback.

  Normally, as shown in FIG. 17, downsampling is performed such that the frame time intervals after downsampling are equal (see, for example, Patent Document 1).

  However, in the simple frame thinning process, image quality degradation due to temporal aliasing (motion blur) becomes a problem. In order to avoid such a problem, band limiting filtering in the time axis direction is necessary. On the other hand, in the case of an encoder using motion-compensated interframe prediction, the reduction in aliasing in the time direction is not directly linked to the reduction in prediction error. In other words, there is room for optimization regarding the downsampling filter in the time direction from the viewpoint of encoding efficiency.

  In the case of a conventional video signal with a frame rate of 30 fps or 60 fps, it is difficult to approximate the characteristics of the filter with high accuracy because sufficient samples (that is, frames) for filtering cannot be secured. For example, when a 30 fps video signal is generated by filtering a 60 fps video signal, the number of frames to be filtered is limited to 2 frames under conditions that do not allow duplication of the filtering target frame.

  On the other hand, in the case of a high frame rate video signal, the degree of freedom in filter design is expanded. For example, when a video signal of 62.5 fps is generated by filtering a video signal of 1000 fps, 16 frames can be targeted for filtering even under conditions that do not allow overlapping of the filtering target frames.

  Therefore, when a high frame rate video signal is input and a low frame rate video signal is obtained by filtering, the degree of freedom in filtering design is increased.

  However, the conventional technology does not consider this point at all. Thus, when a high frame rate video signal is input and a low frame rate video signal is obtained by filtering, there is room for studying a filtering method for optimal frame rate conversion from the viewpoint of coding efficiency.

JP 2004-201165 A

  Against this background, the present inventor “Yukihiro Bando, Noriyuki Takamura, Hitoshi Uekura, Yoshiyuki Yashima:“ A Study on the Downsampling Method of High Frame Rate Video Signals ”, Image Coding Symposium, p5-02 , 2008 "examined the method of selecting a frame after downsampling so as to minimize the motion compensation prediction error power.

  However, in this method, the frame rate is converted by a down-sampling method based on frame selection, and the frame rate is converted by a down-sampling method based on filtering that can solve image quality degradation in the time direction. Not a translation.

  Further, in this method, frames to be selected sequentially are set, and minimization of motion compensation prediction error power in the entire sequence is not guaranteed, and there is room for improvement from that point.

  The present invention has been made in view of such circumstances, and in video encoding processing using a low frame rate video signal obtained by downsampling for a high frame rate video signal as input, by downsampling based on filtering. By generating a low frame rate video signal, image quality degradation due to temporal aliasing is prevented, and by generating the low frame rate video signal in consideration of coding efficiency for the entire sequence, downsampling is performed. An object is to establish a new frame rate conversion technique that realizes the improvement of the encoding efficiency of the later low frame rate video signal.

[1] Basic concept of the present invention For a high frame rate video signal subject to frame rate conversion, the frame interval is δ _t , and the time t = jδ _t (j = 0, 1,...) The pixel value at the position x in the frame is represented as f (x, t) (x = 0, 1,..., X−1).

Consider a case where the number of frames is converted to 1 / M by down-sampling the pixel signal f (x, t). A frame rate ratio M before and after downsampling is called a downsampling ratio. That is, this conversion assumes that the frame rate is converted from “1 / δ _t ” to “1 / Mδ _t ”.

  In the following, a one-dimensional signal will be described as an example for the sake of simplicity, but the same discussion can be easily extended to a two-dimensional signal.

  Frame rate conversion methods include frame thinning and simple averaging. In frame decimation, a frame represented by the following equation (1) is a frame after downsampling.

  In simple averaging, a frame represented by the following expression (2) is a frame after downsampling.

  However, neither the frame decimation method nor the simple averaging method is a method assuming moving image encoding with motion compensation, and is not an optimal method from the viewpoint of the encoding efficiency of a video signal after downsampling.

Therefore, in the present invention, in order to improve the coding efficiency of the video signal after down-sampling while preventing image quality deterioration due to time-direction aliasing (motion click), the following expression (3 ), The filter coefficient W _i having the tap length 2Δ + 1 (i is a variable indicating the frame position of downsampling)
W _i = (ω _i [−Δ], ...., ω _i [Δ])
As shown in FIG. 1, a frame after downsampling is obtained by performing filtering using.

Here, it is assumed that Σω _i [j] = 1. However, (SIGMA ₎ is the sum total about j =-(DELTA _{) i- (} DELTA) _i . Δ _i is a parameter given from the outside for each frame (may not depend on i).

  If the frame position of downsampling is the head frame position or the last frame position, 2Δ + 1 frames to be filtered may not be aligned, but in this case, they exist as filtering objects. The filter coefficient for the frame is determined by proportional calculation (proportional calculation so that the sum of the filter coefficients is 1).

At this time, in the present invention, in order to realize improvement in the encoding efficiency of the video signal after downsampling, the encoding efficiency of all the frames of the low frame rate video signal after downsampling is considered. The filter coefficient W _i is determined.

Next, a method determining the filter coefficients W _i is a characteristic process in the present invention.

[I] “Determination of Filter Coefficient (Part 1)”
As a set of filter coefficients that can be taken as filter coefficients W _i = (ω _i [−Δ],..., Ω _i [Δ]), N kinds of coefficients ψ _n = {ψ _n [−Δ],. .., ψ _n [Δ]} (n = 0,..., N−1). In the following, it is assumed that ~ Ψ = (Ψ ₀ , ...., Ψ _N-1 ). In the following notation, the symbol “˜X” (where X is a letter) indicates a symbol attached to “X”.

  As shown in FIG. 2, at each frame position set at equal intervals, an appropriate filter coefficient is selected from the N patterns. Hereinafter, the filter coefficient selection method will be described in detail.

Signal _{^ f (x, iMδ t,} W i) sections of the size S B [k] (k = 0, 1, .... , K-1) is divided into, each interval B [k] (k = When motion compensation (estimated displacement d _i = (d _i [0],..., D _i [K−1]) is performed in units of 0, 1,..., K−1), The prediction error power after motion compensation in the section can be expressed as the following equation (4): In the following notation, the symbol ^ in “^ X” (where X is a letter) is “X”. The symbol attached to is shown.

  Here, in Expression (4), the prediction error power after motion compensation is obtained with respect to a frame (frame serving as a reference frame) generated by the previous filtering.

When W _i and W _i-1 are given, the estimated displacement d [k] (k = 0, k) is set so as to minimize σ _i ² [W _i , W _i-1 ] according to the equation (4). 1,..., K-1) and d [k, _Wi , _Wi-1 ] (k = 0, 1,..., K-1). At this time, the accuracy of the estimated displacement d [k] is given from the outside (eg, integer pixel accuracy, 1/2 pixel accuracy, 1/4 pixel accuracy, etc.).

Here, it should be noted, is a method of setting the W _i. If W _i is set so as to minimize σ _i ² [W _i , W _i-1 ], the prediction error power Σσ _i ² [W _i , W _i-1 ] in the entire sequence is set.
However, Σ is the sum of i = 1 to (J / M−1), and J does not mean that the number of frames is minimized. This is because selection of W _i is because also affects sigma _i ² [W _{i + 1,} W _i].

  Therefore, the parameters to be obtained are J / M parameters satisfying the following expression (5).

Since each W _i is selected from N candidates included in Ψ, the possible combinations of ^{J / M} parameters (W ₀ ,..., W _{J / M-1} ) are N ^{J / M} Of these, it is difficult in terms of computational complexity to search for an optimal combination (W ^* ₀ ,..., W ^* _{J / M-1} ) from among them.

Accordingly, in the present invention, focusing on the fact that σ _i ² [W _i , W _i-1 ] depends only on the result of the immediately preceding frame, the optimal solution is calculated as follows.

First, for all of the filter coefficient set ~Ψ which may take as W _i, the optimal _{W i-1, ...., Σσ} i 2 in the case of using the W ₀ [W _i, W _i-1] ( However, Σ is defined as S _i (W _i ) as the sum of i = 1 to _i .

Here, focusing on the fact that σ _i ² [W _i , W _i-1 ] depends only on the result of the previous frame, S _i (W _i ) uses S _i-1 (W _i-1 ), As shown in FIG. 3, it is expressed as the following formula (6). Note that i = 1,..., (J / M−1).

S _i (W _i ) (W _i = Ψ ₀ ,..., Ψ _N−1 ) calculated at this time is stored in a memory and used in the calculation of S _{i + 1} (W _{i + 1} ). And Further, W _i-1 giving the minimum value of the equation (6) is set as ^ W _i-1 (W _i ) as shown in FIG. This ^ W _i-1 (W _i ) (W _i = Ψ ₀ ,..., Ψ _N-1 ) is all stored in the memory. Here, the symbol ^ in “^ X” (X is a letter) indicates a symbol attached to “X”.

  The minimization problem of equation (5) can be expressed as equation (7) below.

When W _{J / M-1} that minimizes the equation (7) is set as W ^* _{J / M-1} , this W ^* _{J / M-1} can be expressed by the following equation (8).

Since the optimal solution for W ^* _{J / M-1} is stored in the memory as ^ W _{J / M-2} (W _{J / M-1} ), W ^* _{J / M-2} = ^ W _{J / M-2} (W _{J / M-1} ).

Hereafter, the same reference process is
W ^* _{J / M-3} = ^ W _{J / M-3} (W _{J / M-2} )
...
_{^{W * 0 = ^ W 0 (}} W 1)
Repeat as.

In this way, according to “determination of filter coefficients (part 1)”, the optimum combination of filter coefficients (W ^* ₀₎ used at each frame position i set at equal intervals without depending on the round robin search. , ...., W ^* _{J / M-1} ).

  In the above description, the filter coefficient is selected so as to minimize the motion compensation prediction error power, but the encoding is performed by a specific encoder (for example, an H.264 compliant encoder). The filter coefficient may be selected so as to minimize the Lagrangian cost (encoding cost: J = D + λR) of the weighted sum of the encoding distortion D and the code amount R. Here, λ is a parameter for adjusting the weight of the coding distortion D and the code amount R, and is given from the outside.

[Ii] “Determination of filter coefficients (2)”
In “Determination of Filter Coefficient (Part 1)”, the filtering represented by Expression (3) using the filter coefficient W _i = (ω _i [−Δ],..., Ω _i [Δ]) is performed. The frame after downsampling is obtained by the above, but in the “determination of filter coefficient (part 2)”, the frame after downsampling is obtained by performing the filtering represented by the following equation (9).

Here, it is assumed that Σω _i [j] = 1. However, (SIGMA ₎ is the sum total about j =-(DELTA _{) i- (} DELTA) _i . Δ _i is a parameter given from the outside for each frame (may not depend on i).

The difference from Equation (3) is that in Equation (9), a parameter p _i for correcting the position of the filter processing is added. This parameter p _i takes the values p _i = 0, 1,..., P−1. Note that the value of p _i can also be defined to take a positive or negative value around 0.

  Usually, as shown in FIG. 17, the frame time intervals after down-sampling are performed at equal intervals. This is based on the assumption that jerkiness (image quality degradation) occurs in non-uniformly spaced time sampling. This assumption is correct when the downsampling target is a low frame rate video signal. However, this is not the case when the object of downsampling is a high frame rate video signal. If the target of downsampling is a high frame rate video signal, even if the frames are not arranged at equal intervals, if the deviation of the equal intervals is within a certain threshold, visual jerkiness (image quality degradation) occurs. Because it does not.

Therefore, in “determination of filter coefficient (part 2)”, a parameter p _i for correcting the position of the filter processing is added.

That is, as a set of filter coefficients that can be taken as filter coefficients W _i = (ω _i [−Δ],..., Ω _i [Δ]), N types of coefficients ψ _n = {ψ _n [−Δ], .., ψ _n [Δ]} (n = 0,..., N−1), and P positions are selectable as positions where these filter coefficients are applied.

  As illustrated in FIG. 4 illustrating P = 3, an appropriate filter coefficient is selected from the N × P ways at each frame position set at equal intervals. Hereinafter, the filter coefficient selection method will be described in detail.

Signal _{^ f (x, iMδ t,} W i, p i) section B of the size S [k] is divided (k = 0, 1, .... , K-1) , each interval B [k] Motion compensation (estimated transition amount d _i = (d _i [0],..., D _i [K−1]) is performed in units of (k = 0, 1,..., K−1). In this case, the prediction error power after motion compensation in the section can be expressed as the following equation (10).

  Here, in Equation (10), the prediction error power after motion compensation is obtained with respect to a frame (frame serving as a reference frame) generated by the previous filtering.

When W _i , W _i-1 , p _i , and p _i-1 are given, σ _i ² [W _i , W _i-1 , p _i , p _i-1 ] is minimized according to this equation (10). And the estimated displacement d [k] (k = 0, 1,..., K−1) is set so as to be converted into d [k, W _i , W _i−1 , p _i , p _i−1. ] (K = 0, 1,..., K−1). At this time, the accuracy of the estimated displacement d [k] is given from the outside (eg, integer pixel accuracy, 1/2 pixel accuracy, 1/4 pixel accuracy, etc.).

What should be noted here is the setting method of W _i and p _i . If W _i and p _i are set so as to minimize σ _i ² [W _i , W _i-1 , p _i , p _i-1 ], the prediction error power Σσ _i ² [W _i , W _i-1 , p _i , p _i-1 ]
However, Σ is the sum of i = 1 to (J / M−1), and J does not mean that the number of frames is minimized. This is because the selection of W _i , p _i also affects σ _i ² [W _i , W _i-1 , p _i , p _i-1 ].

  Therefore, the parameters to be obtained are 2 × (J / M) parameters that satisfy the following expression (11).

Since each W _i is selected from N candidates included in ~ Ψ and each p _i is selected from P candidates, 2J / M parameters (W ₀ ,..., W _{J / M-1} , p ₀ ,..., P _{J / M-1} ) are possible combinations (PN) ^{J / M} , and the optimum combination (W ^* ₀ , ... , W ^* _{J / M−1} , p ^* ₀ ,..., P ^* _{J / M−1} ) is intensively searched.

Therefore, in the present invention, paying attention to the fact that σ _i ² [W _i , W _i-1 , p _i , p _i-1 ] depends only on the result of the immediately preceding frame, the optimal solution is calculated as follows.

First, the parameter set can take as all filter coefficient set ~Ψ and p _i which can take as _{W i (p i = 0,} 1, ...., P-1) with respect to the optimal W _{i-1, ...., W 0, p i-} 1, ...., Σσ i 2 in the case of using p _{0 [W i, W i-1,} p i, p i-1 ] (where, sigma is i = 1 to _i ) is defined as S _i (W _i , p _i ).

Here, focusing on the fact that σ _i ² [W _i , W _i−1 , p _i , p _i−1 ] depends only on the result of the immediately preceding frame, S _i (W _i , p _i ) becomes S _{i−. Using 1} (W _i-1 , p _i-1 ), as shown in FIG. Note that i = 1,..., (J / M−1).

S _i (W _i , p _i ) calculated at this time is stored in a memory and used in the calculation of S _{i + 1} (W _{i + 1} , p _{i + 1} ). Further, as shown in FIG. 5, W _i-1 and p _i-1 that give the minimum value of the equation (12) are represented as ^ W _i-1 (W _i , p _i ) and ^ p _{i-1, respectively.} Let (W _i , p _i ). ^ W _i-1 (W _i , p _i ) and ^ p _i-1 (W _i , p _i ) (W _i = Ψ ₀ ,..., Ψ _N−1 , p _i = 0, 1, .., P-1) are all stored in the memory.

  The minimization problem of Equation (11) can be expressed as Equation (13) below.

When W _{J / M-1} and p _{J / M-1} that minimize the equation (13) are set as W ^* _{J / M-1} and p ^* _{J / M-1} , respectively, this W ^* _{J / M -1} and p ^* _{J / M-1} can be expressed by the following formula (14).

The optimal solutions for W ^* _{J / M-1} and p ^* _{J / M-1} are ^ W _{J / M-2} (W _{J / M-1} , p _{J / M-1} ), ^ p _{J / M-2} Since it is stored in the memory as (W _{J / M-1} , p _{J / M-1} ), W ^* _{J / M-2} = ^ W _{J / M-2} (W _{J / M −1} , p _{J / M−1} ), p ^* _{J / M−2} = ^ p _{J / M−2} (W _{J / M−1} , p _{J / M−1} ).

Hereafter, the same reference process is
W ^* _{J / M-3} = ^ W _{J / M-3} (W _{J / M-2} , p _{J / M-2} )
p ^* _{J / M-3} = ^ p _{J / M-3} (W _{J / M-2} , p _{J / M-2} )
...
_{^{W * 0 = ^ W 0 (}} W 1, p 1)
_{^{p * 0 = ^ p 0 (}} W 1, p 1)
Repeat as.

In this way, according to “determination of filter coefficient (part 2)”, when the vicinity of each frame position i is also included in the filter processing target, it is equal intervals without depending on the brute force search. The optimal combination of filter coefficients (W ^* ₀ ,..., W ^* _{J / M-1} ) and the optimal combination of filtering positions (p ^* ₀ ,. , p ^* _{J / M-1} ).

  In the above description, the filter coefficient is selected so as to minimize the motion compensation prediction error power, but the encoding is performed by a specific encoder (for example, an H.264 compliant encoder). The filter coefficient may be selected so as to minimize the Lagrangian cost (encoding cost: J = D + λR) of the weighted sum of the encoding distortion D and the code amount R. Here, λ is a parameter for adjusting the weight of the coding distortion D and the code amount R, and is given from the outside.

[2] Configuration of the Present Invention Next, the configuration of the present invention will be described.

  The frame rate conversion apparatus of the present invention selects filter coefficients from a set of filter coefficients prepared in advance at equally spaced frame positions, and filters a plurality of frames of a high frame rate video signal using the filter coefficients. In order to execute the downsampling, (1) at a frame position that is a predetermined number before each of a plurality of frames generated when each filter coefficient is used at the frame positions at equal intervals. A value indicating the coding efficiency of the low frame rate video signal generated by downsampling between each of the generated multiple frames is calculated, and based on the value, the coding efficiency display value from the first frame position is calculated. Filterer used to generate frames with connections that minimize the sum And (2) a code from a plurality of frames generated at a frame position at which the execution by the execution unit is completed. Detecting means for detecting a filter coefficient used for generating a frame having the minimum value of the total sum by detecting a frame having the minimum value of the sum of the display efficiency display values, and (3) a filter coefficient detected by the detecting means And selecting means for selecting a filter coefficient to be used at each frame position by tracing the filter coefficient specified by the execution means from the starting point.

  Here, specifically, the execution means obtains an addition value of the calculated encoding efficiency display value and the sum of the encoding efficiency display values of the frames at the frame position before the specified number as the calculation target. By specifying the minimum value among them, the filter coefficient at the frame position before the specified number used to generate a frame having a connection that minimizes the sum of the encoding efficiency display values from the head frame position. And the added value is set as the sum of the encoding efficiency display values of the original frame.

  When this configuration is adopted, there may be provided setting means for setting position correction parameters for correcting the frame positions at equal intervals. In this case, the execution means at the frame position corrected by the position correction parameters. The processing is executed in such a manner that a plurality of generated frames are included in frames generated at equally spaced frame positions.

  Further, the execution means may calculate the motion compensation prediction error power as the coding efficiency display value or the coding cost obtained when coding using a predetermined encoder.

  The frame rate conversion method of the present invention realized by the operation of each of the above processing means can also be realized by a computer program. This computer program can be provided by being recorded on an appropriate computer-readable recording medium. The present invention is realized by being provided via a network, installed when the present invention is implemented, and operating on a control means such as a CPU.

  In the frame rate conversion apparatus according to the present invention configured as described above, the execution unit performs the processing between each of the plurality of frames generated at the previous frame position in accordance with the encoding process of the low frame rate video signal. In the case of adopting a configuration for calculating a value indicating the encoding efficiency of the low frame rate video signal generated by downsampling, the execution means has one path for the filter coefficient specifying process, and the execution means Since the end of the execution of repetition is only the final frame position, the frame rate conversion apparatus of the present invention is configured as follows.

  That is, the frame rate conversion apparatus of the present invention selects a filter coefficient from a set of filter coefficients prepared in advance at equally spaced frame positions, and filters a plurality of frames of a high frame rate video signal using the filter coefficient. In order to perform this, (1) one frame before each of a plurality of frames generated when each filter coefficient is used at equally spaced frame positions. A value indicating the coding efficiency of the low frame rate video signal generated by downsampling is calculated with each of the plurality of frames generated at the position, and the coding efficiency display from the head frame position is calculated based on the calculated value. Used to generate a frame with a connection that minimizes the sum of values An execution means for repeatedly executing the filter coefficient at the previous frame position until the final frame position; and (2) the sum of the encoding efficiency display values from a plurality of frames generated at the final frame position. Detecting means for detecting a filter coefficient used to generate a frame having the minimum value of the sum by detecting a frame having the minimum value; and (3) specifying an execution means starting from the filter coefficient detected by the detecting means. By selecting the filter coefficient to be used at each frame position by tracing the above-described filter coefficient, a selection means for selecting the filter coefficient is provided.

  In the present invention, when frame rate conversion is performed on a frame of a high frame rate video signal to a low frame rate video signal by downsampling based on filtering, the encoding efficiency for the entire sequence of the low frame rate video signal after downsampling is performed. The frame rate conversion process is executed in consideration of the above.

  Thus, according to the present invention, when a video signal having a desired frame rate obtained by down-sampling based on filtering for a high frame rate video signal is encoded, the video obtained by thinning out even frame intervals. The amount of code can be kept lower than that of the signal, and the amount of code can be kept lower than that of the video signal obtained by thinning out unequal frame intervals without considering the coding efficiency for all sequences. It becomes like this.

  According to the present invention, since the frame of the high frame rate video signal is converted into the low frame rate video signal by downsampling based on filtering, it is possible to prevent image quality deterioration due to temporal aliasing. become.

It is explanatory drawing of the frame rate conversion process based on filtering. It is explanatory drawing of the selection process of a filter coefficient. It is explanatory drawing of the calculation process of S _i (W _i ) calculated in the present invention. It is explanatory drawing of the parameter which correct | amends the position of a filter process. It is explanatory drawing of the calculation process of S _i (W _i , p _i ) calculated in the present invention. It is an apparatus block diagram of the frame rate conversion apparatus of this invention. It is explanatory drawing of the data structure of a filter coefficient memory | storage part. It is an apparatus block diagram of the frame rate conversion apparatus of this invention. It is an apparatus block diagram of the frame rate conversion apparatus of this invention. It is a flowchart which a downsampling execution part performs. It is a flowchart which a downsampling execution part performs. It is a flowchart which a downsampling execution part performs. It is a flowchart which a downsampling execution part performs. It is a flowchart which a downsampling execution part performs. It is a flowchart which a downsampling execution part performs. It is explanatory drawing of the experimental result performed in order to verify the effectiveness of this invention. It is explanatory drawing of the frame rate conversion process by a prior art.

  Hereinafter, the present invention will be described in detail according to embodiments.

  FIG. 6 shows an example of the device configuration of the frame rate conversion device 1 of the present invention.

  As shown in this figure, the frame rate conversion apparatus 1 of the present invention includes a high frame rate video file 10 storing a high frame rate video signal to be subjected to frame rate conversion processing, and a low frame rate subjected to frame rate conversion processing. A low frame rate video file 11 that stores a video signal, and a frame rate downsampling unit 12 that executes a frame rate conversion process based on filtering from a high frame rate video signal to a low frame rate video signal.

  The frame rate downsampling unit 12 includes a motion estimation unit 1200, a motion compensation prediction error calculation unit 120 that calculates motion compensation prediction error power, a filter coefficient storage unit 121 that stores a set of filter coefficients, and a motion compensation Using the motion compensation prediction error power calculated by the prediction error calculation unit 120 as an evaluation value, an optimum filter coefficient is selected from the set of filter coefficients stored in the filter coefficient storage unit 121, and the selected filter coefficient is used. A downsampling execution unit 122 that executes frame rate downsampling processing by performing filtering processing on a plurality of frames of a high frame rate video signal, and a work memory 123 that stores work data of the downsampling execution unit 122; Is provided.

  Here, when the filtering process is performed on 2Δ + 1 frames, the filter coefficient storage unit 121 stores a set of filter coefficients used by the downsampling execution unit 122 according to the data structure shown in FIG. Will do.

  Before describing the frame selection process performed by the downsampling execution unit 122, the process of the motion estimation unit 1200 included in the motion compensation prediction error calculation unit 120 will be described.

  The motion estimator 1200 receives the prediction target frame and the reference frame as input, and executes the processing shown below, so that the motion vector d [k] for each block in the prediction target frame, that is, Expression (4) or Expression Processing for estimating d [k] (k = 0, 1,..., K−1) in (10) is performed. Here, k is an index for identifying a block.

Processing: Obtains a motion vector d [k] that minimizes the term of Σ {xεB [k]} (prediction error sum of k-th block) in Equation (4) and Equation (10). It is selected from the values in the search range -D ≦ d [k] ≦ D−1 given in advance. The selection method is the case where the candidate vectors are used for all candidate vectors in the search range. The motion compensation prediction error calculation unit 120 calculates a prediction error sum and sets d [k] as a vector that minimizes the prediction error sum. The motion compensation prediction error calculation unit 120 uses the vector d [k] estimated by the motion estimation unit 1200, Based on Expression (4) and Expression (10), motion compensation prediction error power between the prediction target frame and the reference frame is calculated.

[1] First Embodiment Next, a first embodiment will be described.

[1-1] Device Configuration of Downsampling Execution Unit 122 FIG. 8 illustrates an example of the device configuration of the downsampling execution unit 122 configured to realize the first embodiment. Here, in the processing of the downsampling execution unit 122 described below, a case where motion compensation prediction error error power is used as a selection criterion for a filter coefficient used for filtering is shown.

  The downsampling execution unit 122 executes frame rate downsampling processing based on filtering. When the first embodiment is followed, as shown in FIG. 8, the downsampling execution unit 122 selects a processing frame position. Unit 1221, processing filter coefficient selection unit 1222, reference filter coefficient determination unit 1223, all frame position end determination unit 1224, cumulative prediction error minimized final filter coefficient detection unit 1225, and determined reference filter coefficient tracking unit 1226 A filtering execution unit 1227.

The input unit 1220 reads the frame number J of high frame rate video signal, the frame interval [delta] _t of the high frame rate video signal, and a down-sampling ratio M.

Processing frame position selecting unit 1221, down-sampling frame position defined by the equidistant downsampling based on the downsampling ratios M and frame interval [delta] _t (hereinafter, abbreviated as frame position) as selection, top frame The frame position up to the final frame position is sequentially selected as the processing frame position starting from the frame position immediately after the position.

  The processing filter coefficient selection unit 1222 sequentially selects filter coefficients from the filter coefficient set stored in the filter coefficient storage unit 121, and uses the selected filter coefficients to select the processing selected by the processing frame position selection unit 1221. A processing frame is generated by filtering a frame located in the vicinity of the frame position.

  The reference filter coefficient determination unit 1223 performs motion compensation between the processing frame generated by the processing filter coefficient selection unit 1222 and each reference frame generated in association with selection of each filter coefficient at the previous frame position. Calculate the prediction error power, obtain the sum of the calculated motion compensation prediction error power and the sum of the motion compensation prediction error power of the reference frame that is the calculation target (cumulative prediction error calculation function), By specifying the minimum value (minimum cumulative prediction error specifying function), a reference frame having a connection that minimizes the sum of motion compensation prediction error power from the head frame position is specified, and the specified reference frame The filter coefficient used for the generation of the filter coefficient selected for the filter coefficient selected by the processing filter coefficient selection unit 1222 ( Is determined as the irradiation filter coefficient).

  The all frame position end determination unit 1224 determines whether or not processing has been performed for all frame positions up to the final frame position, and determines that processing has not been performed for all frame positions. When instructing the unit 1221 to select the next frame position and determining that all the frame positions have been processed, the cumulative prediction error minimizing final filter coefficient detecting unit 1225 is instructed to execute the process. To do.

  The cumulative prediction error minimizing final filter coefficient detection unit 1225 has a minimum value among the sums of motion compensation prediction error powers calculated at the final frame position (calculated by the reference filter coefficient determination unit 1223). By detecting the frame, the final frame that minimizes the sum of motion compensation prediction error power from the head frame position is identified, and the filter coefficient used to generate the identified final frame is determined at the final frame position. It is detected as the optimum filter coefficient (final filter coefficient).

  The determined reference filter coefficient tracking unit 1226 traces the reference filter coefficient determined by the reference filter coefficient determination unit 1223 from the final filter coefficient detected by the cumulative prediction error minimized final filter coefficient detection unit 1225 as a starting point. Extract the optimal reference filter coefficients for each frame position defined by downsampling.

  The filtering execution unit 1227 executes the filtering process using the final filter coefficient detected by the cumulative prediction error minimizing final filter coefficient detection unit 1225 and the reference filter coefficient extracted by the determined reference filter coefficient tracking unit 1226, thereby increasing the filtering process. A frame rate conversion process from a frame rate video signal to a low frame rate video signal is executed.

[1-2] Process Performed by Downsampling Execution Unit 122 The flow of the frame rate downsampling process performed by the downsampling execution unit 122 configured as described above is shown below.

1. Captured image signal (high frame rate video signal subject frame rate conversion process), reads the number of frames J, the frame rate (used in order to calculate the frame interval [delta] _t) 2. 2. Read downsampling ratio M. The following processing is performed for i = 1,..., J / M-1. 4. Perform the following processing for W _i = Ψ ₀ ,..., Ψ _{N−1 W i-1 = Ψ 0,} ...., Ψ N-1 6 to perform the following processing on. The reference frame ^ f (x-d _{i [k], (i-1) Mδ t} , W i-1) to when _{^ f (x, iMδ t,} W i) for the motion compensated prediction error power (formula A motion vector that minimizes (4)) is obtained. The method of obtaining the motion vector is given from the outside. For example, a full search method is used that squeezes candidates within the search range. 6. Store motion-compensated prediction error power in the case of using the obtained motion vector in σ _i ² [W _i , W _i-1 ]. The value of σ _i ² [W _i , W _i-1 ] + S _i-1 (W _i-1 ) is calculated.
(However, when i = 1, the value of σ ₁ ² [W ₁ , W ₀ ] is calculated.)
8). sigma _i ² [W _i, W _{i-1] + S i-1 (W i} -1) (W i-1 = Ψ 0, ...., Ψ N-1) the minimum value among the S _i Store in (W _i )
(However, when i = 1, the minimum value in σ ₁ ² [W ₁ , W ₀ ] (W ₀ = Ψ ₀ ,..., Ψ _N−1 ) is set to S ₁ (W ₁ )
9. Store W _i-1 giving S _i (W _i ) in ^ W _i-1 (W _i ).
Ten. Storing S J _{/ M-1} a (W J _{/ M-1)} to W _{J / M-1} to minimize the W ^* _{J / M-1}
11． i = J / M-2,...
12． W ^* _i-1 = ^ W _i-1 (W _i ) is read
13. As the i-th frame after the _{down-sampling, ^ f (x, iMδ t} , W * i)
(X = 0,..., X−1) are stored.

In this way, the downsampling execution unit 122 first filters each frame located in the vicinity of the frame position defined by the equally spaced downsampling using each of the filter coefficients prepared in advance. As shown in FIG. 3, motion compensation is performed between each of the generated frames and each of the similarly generated frames corresponding to the selection of each filter coefficient at the previous frame position. Calculate the prediction error power, determine the sum of the calculated motion compensation prediction error power and the sum of the motion compensation prediction error power of the reference frame that is the calculation target, and specify the minimum value among them Thus, the process of obtaining the optimum filter coefficient at the previous frame position is repeated until the final frame position. It returns to run,
Subsequently, the optimum filter coefficient at the final frame position is detected by detecting the frame having the minimum value among the motion compensation prediction error powers calculated at the final frame position, and the detection is performed. The optimal filter coefficient is extracted for each of the frame positions defined by equally spaced down-sampling by tracing the previously specified filter coefficient toward the top frame position with the filter coefficient as a starting point,
Finally, the frame rate conversion process from the high frame rate video signal to the low frame rate video signal is executed by performing filtering at each frame position using the extracted optimum filter coefficients.

  Here, the downsampling execution unit 122 is not between the previous frame position but the frame position before the specified number before the frame position according to the encoding process of the low frame rate video signal. Then, the motion compensation prediction error power may be calculated.

  In this case, there are a plurality of paths for specifying the filter coefficient, and the execution of repetition is started from the beginning frame position and the following specified number of frame positions as the starting point, and the path is specified more than the final frame position. The repeated execution is performed for the frame position up to the previous frame position and the subsequent frame position.

  Then, a frame having the minimum value of the sum is generated by detecting a frame having the minimum value of the sum of the motion compensation prediction error power from a plurality of frames generated at the frame position where the execution of the repetition is completed. For each of the frame positions defined by equally spaced down-sampling, the filter coefficient used in the above is detected, and the filter coefficient specified above is traced to the head frame position starting from the detected filter coefficient. The optimum filter coefficient is extracted.

[2] Second Embodiment Next, a second embodiment will be described.

[2-1] Device Configuration of Downsampling Execution Unit 122 FIG. 9 illustrates an example of the device configuration of the downsampling execution unit 122 configured to realize the second embodiment. Here, in the processing of the downsampling execution unit 122 described below, a case where motion compensation prediction error error power is used as a selection criterion for a filter coefficient used for filtering is shown.

  The downsampling execution unit 122 executes a frame rate downsampling process based on filtering. In the case of following the second embodiment, as shown in FIG. 9, an input unit 1220α and a processing frame position selection are performed. Unit 1221α, processing position / filter coefficient selection unit 1222α, reference position / filter coefficient determination unit 1223α, end of all frame position determination unit 1224α, cumulative prediction error minimized final position / filter coefficient detection unit 1225α, and determination reference A position / filter coefficient tracking unit 1226α and a filtering execution unit 1227 are provided.

The input unit 1220α reads the frame number J of high frame rate video signal, the frame interval [delta] _t of the high frame rate video signal, and a down-sampling ratio M.

Processing frame position selecting unit 1221α the downsampling frame position defined by the equidistant downsampling based on the downsampling ratios M and frame interval [delta] _t (hereinafter, abbreviated as frame position) as selection, top frame The frame position up to the final frame position is sequentially selected as the processing frame position starting from the frame position immediately after the position.

The processing position / filter coefficient selection unit 1222α sequentially sets parameters for correcting the position of the filter processing (the aforementioned p _i : p _i = 0, 1,..., P−1) and the filter coefficient storage unit The filter coefficients are sequentially selected from the set of filter coefficients stored in 121, and are located in the vicinity of the processing frame position selected by the processing frame position selection unit 1221α using the set / selected processing position and the filter coefficient. A processing frame is generated by filtering the frame.

  The reference position / filter coefficient determination unit 1223α is generated in association with the processing frame generated by the processing position / filter coefficient selection unit 1222α and the setting / selection of each reference position / filter coefficient at the previous frame position. Calculate the motion compensation prediction error power between each of the reference frames, and calculate the sum of the calculated motion compensation prediction error power and the sum of the motion compensation prediction error power of the reference frame that is the calculation target (Cumulative prediction error calculation function), by specifying the minimum value among them (minimum cumulative prediction error specification function), a reference frame having a connection that minimizes the sum of motion compensation prediction error power from the start frame position , And the processing position / filter coefficient selection unit 1222 uses the reference position and filter coefficient used to generate the identified reference frame. α is determined and determined as the optimum reference position and filter coefficient (reference filter coefficient) for the selected processing position and filter coefficient.

  The all frame position end determination unit 1224α determines whether or not processing has been performed for all frame positions up to the final frame position, and determines that processing has not been performed for all frame positions. When instructing the unit 1221α to select the next frame position and determining that all the frame positions have been processed, the cumulative prediction error minimizing final position / filter coefficient detecting unit 1225α is executed. Instruct.

  The accumulated prediction error minimizing final position / filter coefficient detection unit 1225α includes the motion compensation prediction error power calculated at the final frame position (calculated by the reference position / filter coefficient determination unit 1223α). By detecting the frame having the minimum value, the final frame that minimizes the sum of the motion compensation prediction error power from the head frame position is identified, and the final position and filter used to generate the identified final frame Coefficients are detected as optimal final positions and filter coefficients (final filter coefficients) at the final frame position.

  The determined reference position / filter coefficient tracking unit 1226α starts from the final position detected by the cumulative prediction error minimizing final position / filter coefficient detection unit 1225α and the final filter coefficient, and the reference position determined by the reference position / filter coefficient determination unit 1223α. Then, by tracing the reference filter coefficient, the optimum reference position and reference filter coefficient for each of the frame positions defined by equal-sampling down sampling are extracted.

  The filtering execution unit 1227α uses the final position and final filter coefficient detected by the cumulative prediction error minimized final position / filter coefficient detection unit 1225α, and the reference position and reference filter coefficient extracted by the determined reference position / filter coefficient tracking unit 1226α. The frame rate conversion process from the high frame rate video signal to the low frame rate video signal is executed by using the filtering process.

[2-2] Process Performed by Downsampling Execution Unit 122 The flow of the frame rate downsampling process performed by the downsampling execution unit 122 configured as described above is shown below.

1. Captured image signal (high frame rate video signal subject frame rate conversion process), reads the number of frames J, the frame rate (used in order to calculate the frame interval [delta] _t) 2. 2. Read downsampling ratio M. The following processing is performed for i = 1,..., J / M-1. 4. Perform the following processing for W _i = Ψ ₀ ,..., Ψ _N−1 5. Perform the following processing for p _i = 0, ..., P-1. _{W i-1 = Ψ 0,} ...., Ψ N-1 7 which performs the following processing for. The following processing is performed for p _i-1 = 0,..., P-1. The reference frame ^ f (x-d _{i [k], (i-1) Mδ t} , W i-1, p i-1) to when _{^ f (x, iMδ t,} W i, p i) A motion vector that minimizes the motion compensation prediction error power (equation (10)) is obtained. The way to obtain the motion vector is given from the outside. For example, a full search method is used that squeezes candidates within the search range. 8. Store the motion compensation prediction error power when using the obtained motion vector in σ _i ² [W _i , W _i−1 , p _i , p _i−1 ]. The value of σ _i ² [W _i , W _i-1 , p _i , p _i-1 ] + S _i-1 (W _i-1 , p _i-1 ) is calculated.
(However, when i = 1, the value of σ ₁ ² [W ₁ , W ₀ , p ₁ , p ₀ ] is calculated.)
Ten. σ _i ² [W _i , W _i−1 , p _i , p _i−1 ] + S _i−1 (W _i−1 , p _i−1 ) (W _i−1 = Ψ ₀ ,..., Ψ _N-1 , p _i-1 = 0,..., P-1) are stored in S _i (W _i , p _i ).
(However, when i = 1, σ ₁ ² [W ₁ , W ₀ , p ₁ , p ₀ ] (W ₀ = Ψ ₀ ,..., Ψ _N−1 , p ₀ = 0,. ..., P- ₁ ) is stored in S ₁ (W ₁ , p ₁ ).
11． Store W _i-1 and p _i-1 giving S _i (W _i , p _i ) in ^ W _i-1 (W _i , p _i ) and ^ p _i-1 (W _i , p _i ).
12． Minimize S _{J / M-1} (W _{J / M-1} , p _{J / M-1} ) W _{J / M-1} , p _{J / M-1} to W ^* _{J / M-1} , p ^* _J Store in _{/ M-1}
13. i = J / M-2,...
14. W ^* _i-1 = ^ W _i-1 (W _i , p _i ) is read
15． p ^* _i-1 = ^ p _i-1 (W _i , p _i ) is read
16． As the i-th frame after _{downsampling, ^ f (x, iMδ t} , W * i, p * i) (x = 0, ...., X-1) to store.

In this way, the down-sampling execution unit 122 first uses the filter coefficients and the position correction parameter values prepared in advance, respectively, in the vicinity of the frame position defined by equal-sampling down-sampling. A frame is generated by filtering the frames to be processed, and as shown in FIG. 5, each of the generated frames is associated with selection of each filter coefficient and each position correction parameter value at the previous frame position. Similarly, the motion compensation prediction error power is calculated between each of the generated frames, and the calculated motion compensation prediction error power is added to the sum of the motion compensation prediction error powers of the reference frame that is the calculation target. Optimum at the previous frame position by finding the value and identifying the minimum value among them A process of obtaining the filter coefficients and the position correction parameter value, repeatedly running until the final frame position,
Subsequently, by detecting the frame having the minimum value among the motion compensated prediction error powers calculated at the final frame position, the optimum filter coefficient and position correction parameter value at the final frame position are obtained. By detecting and tracing the filter coefficient and position correction parameter value specified above starting from the detected filter coefficient and position correction parameter value toward the beginning frame position, it is defined by equal sampling downsampling. Extract the optimal filter coefficient and position correction parameter value for each frame position,
Finally, the frame rate conversion processing from the high frame rate video signal to the low frame rate video signal is performed by performing filtering at each frame position using the extracted optimum filter coefficient and position correction parameter value. To do.

  Here, the down-sampling execution unit 122 is not in front of the previous frame position (including the correction position) in accordance with the encoding process of the low frame rate video signal, but before the specified number of previous times. The motion compensation prediction error power may be calculated between the frame position (including the correction position).

  In this case, there are a plurality of paths for specifying the filter coefficient, and the execution of repetition is started from the beginning frame position and the following specified number of frame positions as the starting point, and the path is specified more than the final frame position. The repeated execution is performed for the frame position up to the previous frame position and the subsequent frame position.

  Then, a frame having the minimum value of the sum is generated by detecting a frame having the minimum value of the sum of the motion compensation prediction error power from a plurality of frames generated at the frame position where the execution of the repetition is completed. For each of the frame positions defined by equally spaced down-sampling, the filter coefficient used in the above is detected, and the filter coefficient specified above is traced to the head frame position starting from the detected filter coefficient. The optimum filter coefficient is extracted.

  Next, the present invention will be described in detail according to examples.

[1] First Embodiment FIGS. 10 to 12 show examples of flowcharts of a frame rate downsampling process executed by the downsampling execution unit 122 configured as shown in FIG.

  Next, according to these flowcharts, the frame rate downsampling process to be executed by the downsampling execution unit 122 configured as shown in FIG. 8 will be described in detail.

[1-1] Overall Processing When there is a request for execution of the frame rate down-sampling process, the down-sampling execution unit 122 first selects the frame rate conversion process target in step S101 as shown in the flowchart of FIG. comprising a high frame rate video signal, and the frame number J, and the frame interval [delta] _t, with read and downsampling ratio M, the set of filter coefficients stored in the filter coefficient storage unit 121 Ψ _{n (n} = 0~ N-1) is read.

Subsequently, in step S102, the frame designated by i = 1 and W ₁ (W ₁ : filter coefficient used at the frame position of i = 1), and i = 0 and W ₀ (W ₀ : frame position of i = 0). the W ₀ between the frame specified by the filter coefficient), calculates the S ₁ (W ₁₎ above, to give stores it in the working memory 123, the S ₁ a (W ₁₎ to be used in ^ W ₀ (W ₁ ) is stored in the work memory 123.

  Here, details of the processing in step S102 will be described later with reference to the flowchart of FIG.

Subsequently, in step S103, for 2 ≦ i ≦ J / M−1, a frame designated by i and W _i (W _i : a filter coefficient used at frame position i), and i−1 and W _i−1 (W _i-1 : The above-described S _i (W _i ) is calculated with respect to the frame specified by the filter coefficient used at the frame position i-1, and stored in the work memory 123, and the S _i ( W _{i -1} giving W _i ) is stored in work memory 123 as ^ W _i-1 (W _i ).

  Here, details of the processing in step S103 will be described later with reference to the flowchart of FIG.

Subsequently, at step S104, and stores the _{S J / M-1 (W} J / M-1) working memory 123 W _{J / M-1} to minimize the W ^* _{J / M-1} to. That is, the filter coefficient W _{J / M-1} that minimizes S _{J / M-1} (W _{J / M-1} ) is obtained at the final frame position J _{/ M−1,} and is obtained as the final frame position J _{/ M−1.} / M-1 is stored in the working memory 123 as the optimum filter coefficient W ^* _{J / M-1} .

  Subsequently, in step S105, J / M-2 is set to the variable i.

Subsequently, in step S106, W ^* and _{J / M-1} as a starting point, and stores the _{^{W * i-1 = ^ W}} i-1 (W i) working memory 123 to determine the W ^* _i-1 in accordance with .

That is, as shown in FIG. 3, ^ W since _i-1 (W i) is the W _i-1 which gives the minimum value for that W _i, with _{^{it, W * i-1 = ^}} W i- _In accordance with ₁ (W _i ), the optimum filter coefficient W ^* _i−1 used at the frame position i ₋₁ is determined and stored in the working memory 123.

  Subsequently, in step S107, it is determined whether or not the value of the variable i has reached 0, and when it is determined that the value has not reached 0, the process proceeds to step S108, and the value of the variable i is decremented by one. After incrementing, the process returns to the process of step S106, and when it is determined that 0 has been reached, the process proceeds to the process of step S109.

Subsequently, in step S109, 0 ≦ i ≦ J / M−1 is set as the processing target, and as the i-th frame after down-sampling, ^ f (x, iMδ _t , W ^* _i ) (x = 0,... , X-1) and stored in the low frame rate video file 11, a low frame rate video signal after downsampling is generated, and the process is terminated.

[1-2] Details of Processing in Step S102 Next, details of processing executed in step S102 of the flowchart of FIG. 10 will be described according to the flowchart of FIG.

  When entering the process of step S102, the downsampling execution unit 122 first sets 1 to the variable i indicating the frame position i in step S201 as shown in the flowchart of FIG. In the flowchart of FIG. 11, the value of i is fixed to 1.

Subsequently, in step S202, sets 0 to the variable u indicating the number of the filter coefficients W _i used in the frame position i, in the subsequent step S203, sets the [psi _u as the filter coefficient W _i used in the frame position i.

Subsequently, in step S204, 0 is set to the variable v indicating the number of the filter coefficient W _i-1 used at the frame position i-1, and in step S205, the filter coefficient W _i-1 used at the frame position _i-1. Set Ψ _v as

Subsequently, in step S206, i and W _i frames to specify the (frame position i, frame generated by filtering using a filter coefficient W _i), designated by the i-1 and W _i-1 frames between the (at frame position i-1, frame generated by filtering using a filter coefficient W _i-1), the minimum value sigma _i ² [W _i of the motion compensated prediction error power, W _{i- 1} ] is calculated and stored in the work memory 123.

That is, since i = 1, σ ₁ ² [W ₁ , W ₀ ] is calculated and stored in the work memory 123.

  Subsequently, in step S207, it is determined whether or not the value of the variable v has reached N-1, and when it is determined that the value has not reached N-1, the process proceeds to step S208, where the value of the variable v is reached. Is incremented by 1, and the process returns to step S205.

In this way, in the case of i = 1, by executing the process of step S206 while incrementing the value of the variable v from 0 to N−1 one by one, i = 1 and W ₁ (value of u The minimum value σ of the motion compensation prediction error power between the frame specified by the filter coefficient to point to and the frame specified by i = 0 and W ₀ (W ₀ = Ψ ₀ ,..., Ψ _N−1 ). ₁ ² [W ₁ , W ₀ ] is calculated and stored in the work memory 123.

On the other hand, when it is determined in step S207 that the value of the variable v has reached N-1, the process proceeds to step S209, where σ _i ² [W _i , W _i-1 ] (W _i−1 = Ψ ₀ ,..., Ψ _N−1 ), the minimum value S _i (W _i ) is identified and stored in the working memory 123, and its S _i (W _i W _i-1 to be given to the work memory 123 as ^ W _i-1 (W _i ).

That is, when i = 1, σ ₁ ² [W ₁ , W ₀ ] (W ₀ = Ψ ₀ ,...,..., Obtained by repeatedly executing the process of step S206 for a certain filter coefficient W ₁ . The minimum value S ₁ (W ₁ ) in Ψ _N-1 ) is specified and stored in the work memory 123, and W ₀ giving the S ₁ (W ₁ ) is set to ^ W ₀ (W ₁ ) In the work memory 123.

  Subsequently, in step S210, it is determined whether or not the value of the variable u has reached N-1, and when it is determined that the value has not reached N-1, the process proceeds to step S211 and the value of the variable u is reached. Is incremented by 1, and the process returns to step S203.

Thus, in the case of i = 1, the value of the variable u by executing the processing in step S206 / Step S209 while incrementing by one to N-1 to 0, the value of W ₁ [psi _0,. , Ψ _N−1 , σ ₁ ² [W ₁ , W ₀ ] (W ₀ = Ψ ₀ ,..., Ψ _N-1 ) is determined, and the minimum value S ₁ among them is determined. (W ₁ ) is specified and stored in the work memory 123, and W ₀ giving S ₁ (W ₁ ) is stored in the work memory 123 as ^ W ₀ (W ₁ ).

  When it is determined in step S210 that the value of the variable u has reached N-1, the process of step S102 in the flowchart of FIG. 10 ends.

In this way, the down-sampling execution unit 122 starts the processing of step S102 in the flowchart of FIG. 10, by executing the flowchart of FIG. 11, frame and i = 0 and the specification of i = 1 and W ₁ S ₁ (W ₁ ) is calculated with respect to the frame designated by W ₀ , stored in the work memory 123, and W ₀ giving the S ₁ (W ₁ ) is represented by ^ W ₀ (W ₁ ) To be stored in the work memory 123.

[1-3] Details of Processing in Step S103 Next, details of processing executed in step S103 of the flowchart of FIG. 10 will be described according to the flowchart of FIG.

  In step S103, the downsampling execution unit 122 first sets 2 to a variable i indicating the frame position i in step S301, as shown in the flowchart of FIG.

Subsequently, in step S302, sets 0 to the variable u indicating the number of the filter coefficients W _i used in the frame position i, in the subsequent step S303, sets the [psi _u as the filter coefficient W _i used in the frame position i.

Subsequently, in step S304, 0 is set to the variable v indicating the number of the filter coefficient W _i-1 used at the frame position i-1, and in the subsequent step S305, the filter coefficient W _i-1 used at the frame position _i-1. Set Ψ _v as

Subsequently, in step S306, a frame specified by i and W _i (a frame generated by filtering using the filter coefficient W _i at frame position i), and i−1 and W _i−1 are specified. frames between the (at frame position i-1, frame generated by filtering using a filter coefficient W _i-1), the minimum value sigma _i ² [W _i of the motion compensated prediction error power, W _{i- 1} ] is calculated.

In step S307, σ _i ² [W _i , W _i-1 ] + S _i-1 (W _i-1 ) is calculated and stored in the work memory 123. Since S _i-1 (W _i-1 ) used at this time is obtained in the process of the flowchart of FIG. 11 or the process of step S310 described later and stored in the work memory 123, it is used.

  Subsequently, in step S308, it is determined whether or not the value of the variable v has reached N-1, and when it is determined that the value has not reached N-1, the process proceeds to step S309, where the value of the variable v is reached. Is incremented by 1, and the process returns to step S305.

In this way, by executing the processing of step S306 / step S307 while incrementing the value of the variable v one by one from 0 to N-1, i and the variable W _i (filter coefficient indicated by the value of u) Minimum value σ of motion compensation prediction error power between a frame to be specified and a frame to be specified by i−1 and variables W _i−1 (W _i−1 = Ψ ₀ ,..., Ψ _N−1 ). _i ² [W _i , W _i-1 ] is calculated, the calculated σ _i ² [W _i , W _i-1 ], its i-1 and the variable W _i-1 (W _i-1 = Ψ). ₀ , ...., Ψ _N-1 ), and the sum of the motion compensated prediction error power S _i-1 (W _i-1 ) from the head frame position of each frame specified by These are stored in the working memory 123.

On the other hand, when it is determined in step S308 that the value of the variable v has reached N−1, the process proceeds to step S310, and σ _i ² [W _i , W _i-1 ] + S _i stored in the work memory 123 is reached. ₋₁ (W _i−1 ) (W _i−1 = Ψ ₀ ,..., Ψ _N−1 ), the minimum value S _i (W _i ) is identified and stored in the work memory 123. At the same time, W _i−1 giving S _i (W _i ) is stored in work memory 123 as W _i−1 (W _i ).

That is the value of variable W _i, step S306 / step processing of S307 was determined by repeatedly executing sigma _i ² [W _i, W _{i-1] + S i-1 (W i} -1) (W i ₋₁ = Ψ ₀ ,..., Ψ _N−1 ), the minimum value S _i (W _i ) is specified and stored in the working memory 123, and the S _i (W _i ) than is stored in the work memory 123 as W _i-1 a _{^ W i-1 (W i} ) to give.

  Subsequently, in step S311, it is determined whether or not the value of the variable u has reached N-1, and when it is determined that the value has not reached N-1, the process proceeds to step S312 and the value of the variable u is determined. Is incremented by 1, and the process returns to step S303.

In this way, by executing the processing in steps S306 / Step S307 / Step S310 while incrementing the value of the variable u by one from 0 to N-1, the value [psi ₀ of the variable W _i, .... , Ψ _N−1 , σ _i ² [W _i , W _i-1 ] + S _i-1 (W _i-1 ) (W _i-1 = Ψ ₀ ,..., Ψ _N-1 ) And the minimum value S _i (W _i ) is specified and stored in the working memory 123, and W _i-1 giving the S _i (W _i ) is set to ^ W _i-1 (W _i ).

  On the other hand, when it is determined in step S311, that the value of the variable u has reached N-1, the process proceeds to step S313, where it is determined whether the value of the variable i has reached J / M-1. When it is determined that J / M-1 has not been reached, the process proceeds to step S314, the value of the variable i is incremented by one, and then the process returns to step S302 to perform processing for the next frame position. .

  When it is determined in step S313 that the value of the variable i has reached J / M-1, the process of step S103 in the flowchart of FIG. 10 ends.

In this way, when the downsampling execution unit 122 enters the process of step S103 of the flowchart of FIG. 10, the downsampling execution unit 122 executes the flowchart of FIG. 12 to perform i and W _i for 2 ≦ i ≦ J / M−1. S _i between the designated frame and i-1 and W _i-1 of the specified frame of the (W _i) is calculated, and stores it in the working memory 123, the S _i (W _i) Is processed so that W _i-1 that gives is stored in the work memory 123 as W _i-1 (W _i ).

[1-4] Outline of Processing As described above, the down-sampling execution unit 122 configured as shown in FIG. 8 first specifies by using each of the filter coefficients prepared in advance by equally-sampling down-sampling. A frame is generated by filtering a frame located in the vicinity of the frame position to be processed, and as shown in FIG. 3, for each of the generated frames, each filter coefficient is selected at the previous frame position. In addition, the motion compensation prediction error power is calculated between each of the similarly generated frames, and the calculated motion compensation prediction error power and the sum of the motion compensation prediction error power of the reference frame that is the calculation target are calculated. Obtain the optimum filter coefficient at the previous frame position by obtaining the added value and specifying the minimum value among them. This process is repeated until the final frame position.

  Subsequently, the optimum filter coefficient at the final frame position is detected by detecting the frame having the minimum value among the motion compensation prediction error powers calculated at the final frame position, and the detection is performed. The optimum filter coefficient is extracted for each frame position defined by equally spaced down-sampling by tracing the previously specified filter coefficient toward the top frame position starting from the filter coefficient.

  Finally, the frame rate conversion process from the high frame rate video signal to the low frame rate video signal is executed by performing filtering at each frame position using the extracted optimum filter coefficients.

[2] Second Embodiment FIGS. 13 to 15 show examples of flowcharts of frame rate downsampling processing executed by the downsampling execution unit 122 configured as shown in FIG.

  Next, according to these flowcharts, the frame rate down-sampling processing to be executed by the down-sampling execution unit 122 configured as shown in FIG. 9 will be described in detail.

[2-1] Overall Processing When there is a request for execution of frame rate down-sampling processing, the down-sampling execution unit 122 first determines whether the frame rate conversion processing target is selected in step S401 as shown in the flowchart of FIG. comprising a high frame rate video signal, and the frame number J, and the frame interval [delta] _t, with read and downsampling ratio M, the set of filter coefficients stored in the filter coefficient storage unit 121 Ψ _{n (n} = 0~ N-1) is read.

Subsequently, in step S402, a frame designated by i = 1, W ₁ and p ₁ (W ₁ : a filter coefficient used at a frame position of i = 1, p ₁ : a parameter for correcting the frame position of i = 1) and , I = 0, W ₀ and p ₀ (W ₀ : filter coefficient used at the frame position of i = 0, p ₀ : parameter for correcting the frame position of i = 0). S ₁ (W ₁ , p ₁ ) is calculated and stored in the working memory 123, and W ₀ and p ₀ giving the S ₁ (W ₁ , p ₁ ) are respectively represented by ^ W ₀ (W ₁ , p ₁ ), ^ p ₀ (W ₁ , p ₁ ) are stored in the work memory 123.

  Here, details of the processing in step S402 will be described later with reference to the flowchart of FIG.

Subsequently, in step S403, designation of i, W _i and p _i (W _i : filter coefficient used at frame position i, p _i : parameter for correcting frame position i) for 2 ≦ i ≦ J / M−1. And i-1, W _i-1 and p _i-1 (W _i-1 : filter coefficient used at frame position i-1, p _i-1 : parameter for correcting frame position i-1) The above-described S _i (W _i , p _i ) is calculated with respect to the frame to be stored and stored in the work memory 123, and W _i−1 giving the S _i (W _i , p _i ) , P _i-1 are stored in the work memory 123 as ^ W _i-1 (W _i , p _i ) and ^ p _i-1 (W _i , p _i ), respectively.

  Details of the process in step S403 will be described later with reference to the flowchart of FIG.

Subsequently, in step _{S404, S J / M-1} (W J / M-1, p J / M-1) to minimize the W _{J / M-1,} a p J _{/ M-1,} respectively W ^* _{J / M-1} and p ^* _{J / M-1} are stored in the work memory 123. That is, the filter coefficient W _{J / M-1} and position correction for minimizing S _{J / M-1} (W _{J / M-1} , p _{J / M-1} ) at the final frame position J / M-1. Determine the parameter p _{J / M-1} and use it as the optimum filter coefficient W ^* _{J / M-1} and position correction parameter p ^* _{J / M-1} to be used at the final frame position J / M-1. It is stored in the memory 123.

  In step S405, J / M-2 is set to the variable i.

Subsequently, in step S406, starting from W ^* _{J / M-1} and p ^* _{J / M-1} , W ^* _i-1 = ^ W _i-1 (W _i , p _i ) and W ^* _i-1 stores in the work memory 123 to determine and store in _{^{p * i-1 = ^ p}} i-1 (W i, p i) working memory 123 to determine p ^* _i-1 in accordance with.

That is, as shown in FIG. _{5, ^ W i-1 (} W i, p i) so is a W _i-1 which gives the minimum value for that W _i, with it, W ^* _i-1 = ^ In accordance with W _i-1 (W _i , p _i ), the optimum filter coefficient W ^* _i-1 used at the frame position _i-1 is determined and stored in the working memory 123. Further, ^ p _i-1 (W _i , p _i ) is p _i-1 giving the minimum value for that p _i , so that it is used to determine the frame position i according to p ^* _i-1 = ^ p _i-1 (W _i , p _i ). The optimal position correction parameter p ^* _i−1 used in −1 is determined and stored in the work memory 123.

  Subsequently, in step S407, it is determined whether or not the value of the variable i has reached 0. If it is determined that the value has not reached 0, the process proceeds to step S408, and the value of the variable i is incremented by one. After incrementing, the process returns to the process of step S406, and when it is determined that 0 has been reached, the process proceeds to step S409.

Subsequently, in step S409, the as processing target 0 ≦ i ≦ J / M- 1, as the i-th frame after _{downsampling, ^ f (x, iMδ t} , W * i, p * i) (x = 0 ,..., X-1) are generated and stored in the low frame rate video file 11 to generate a low frame rate video signal after downsampling, and the process ends.

[2-2] Details of Processing in Step S402 Next, details of processing executed in step S402 of the flowchart of FIG. 13 will be described according to the flowchart of FIG.

  In step S402, the downsampling execution unit 122 first sets 1 to a variable i indicating the frame position i in step S501 as shown in the flowchart of FIG. In the flowchart of FIG. 14, the value of i is fixed to 1.

Subsequently, in step S502, 0 is set to the variable u indicating the number of the filter coefficient W _i used at the frame position i, and Ψ _u is set as the filter coefficient W _i used at the frame position i in the subsequent step S503. In step S504, 0 is set as the value of the position correction parameter p _i used at the frame position i.

Subsequently, in step S505, a variable v indicating the number of the filter coefficient W _i-1 used at the frame position i-1 is set to 0, and in step S506, the filter coefficient W _i-1 used at the frame position i-1 is set. set the [psi _v as, in subsequent step S507, the sets 0 as the value of the position correcting parameter p _i-1 used in the frame position i-1.

Subsequently, in step S508, a frame specified by i, W _i and p _i (a frame generated by filtering using the filter coefficient W _i at the frame position i corrected by p _i ), i− 1, a frame designated by W _i-1 and p _i-1 (a frame generated by filtering using a filter coefficient W _i-1 at a frame position i-1 corrected by p _i-1 ); The minimum value σ _i ² [W _i , W _i−1 , p _i , p _i−1 ] of the motion compensation prediction error power is calculated and stored in the work memory 123.

That is, since i = 1, σ ₁ ² [W ₁ , W ₀ , p ₁ , p ₀ ] is calculated and stored in the work memory 123.

Subsequently, in step S509, it is determined whether or not the value of p _i-1 has reached P-1, and when it is determined that it has not reached P-1, the process proceeds to step S510, where p _{i After} incrementing the value of ₋₁ by one, the processing returns to step S508.

On the other hand, when it is determined in step S509 that the value of p _i-1 has reached P-1, the process proceeds to step S511, where it is determined whether the value of variable v has reached N-1. When it is determined that N-1 has not been reached, the process proceeds to step S512, the value of the variable v is incremented by 1, and the process returns to step S506.

In this way, in the case of i = 1, the value of p _i-1 is incremented by 1 from 0 to P-1, and the value of variable v is incremented by 1 from 0 to N-1. By executing the processing of S508, i = 1, a frame designated by W ₁ (a certain filter coefficient) and p ₁ (a certain correction parameter), i = 0, W ₀ (W ₀ = Ψ ₀ ,... , Ψ _N-1 ) and p ₀ (p ₀ = 0,..., N-1) to specify the minimum motion compensation prediction error power σ ₁ ² [W ₁ , W ₀ , p ₁ , p ₀ ] are calculated and stored in the working memory 123.

On the other hand, when it is determined in step S511 that the value of the variable v has reached N−1, the process proceeds to step S513, and σ _i ² [W _i , W _i−1 , p _i stored in the work memory 123 is reached. , p _{i-1] (W i-1 = Ψ 0} , ...., Ψ N-1, p i-1 = 0, ...., N-1) the minimum value S _i (W in _i , p _i ) is specified and stored in the working memory 123, and W _i-1 and p _i-1 giving its S _i (W _i , p _i ) are respectively represented by ^ W _i-1 (W _i , p _i ), ^ p _i-1 (W _i , p _i ) are stored in the work memory 123.

That is, when i = 1, σ ₁ ² [W ₁ , W ₀ , p ₁ , p ₀ ] (W obtained by repeatedly executing the processing of step S508 for a certain filter coefficient W ₁ and a certain p _{1. 0} = Ψ ₀ ,..., Ψ _N−1 , p ₀ = 0,..., N−1), and specify the minimum value S ₁ (W ₁ , p ₁ ) W ₀ and p ₀ which are stored in the work memory 123 and give S ₁ (W ₁ , p ₁ ) are represented by ^ W ₀ (W ₁ , p ₁ ) and ^ p ₀ (W ₁ , p ₁ ), respectively. Is stored in the working memory 123.

Subsequently, in step S514, the value of p _i is determined whether or not reached the P-1, when it is determined that not reached P-1, the process proceeds to step S515, the value of p _i Is incremented by 1, and the process returns to step S505.

On the other hand, in the determination processing in step S514, when the value of p _i to determine the arrival at the P-1, the process proceeds to step S516, the value of the variable u is determined whether or not reaching the N-1 When it is determined that N-1 has not been reached, the process proceeds to step S517, the value of the variable u is incremented by 1, and the process returns to step S503.

In this way, when i = 1, the value of p _i is incremented by 1 from 0 to P−1 and the value of variable u is incremented by 1 from 0 to N−1. by executing the processing in step S513, W ₁ value _{(W 1 = Ψ 0, ....} , Ψ N-1) and the value of _{p i (p 1 = 0,} ...., N-1 ) Σ ₁ ² [W ₁ , W ₀ , p ₁ , p ₀ ] (W ₀ = Ψ ₀ ,..., Ψ _N−1 , p ₀ = 0,..., N− 1) is obtained, the minimum value S ₁ (W ₁ , p ₁ ) is specified and stored in the working memory 123, and W ₀ , p ₀ giving the S ₁ (W ₁ , p ₁ ) is obtained. Are stored in the work memory 123 as ^ W ₀ (W ₁ , p ₁ ) and ^ p ₀ (W ₁ , p ₁ ), respectively.

  If it is determined in step S516 that the value of the variable u has reached N-1, the process of step S402 is terminated.

In this way, when the downsampling execution unit 122 enters the process of step S402 of the flowchart of FIG. 13, by executing the flowchart of FIG. 14, the frame designated by i = 1, W ₁ and p ₁ , S ₁ (W ₁ , p ₁ ) is calculated between i = 0, W ₀ and the frame designated by p ₀ , stored in the work memory 123, and S ₁ (W ₁ , W ₀ and p ₀ giving p ₁ ) are processed to be stored in the work memory 123 as ^ W ₀ (W ₁ , p ₁ ) and ^ p ₀ (p ₁ , p ₁ ), respectively.

[2-3] Details of Processing in Step S403 Next, details of processing executed in step S403 of the flowchart of FIG. 13 will be described according to the flowchart of FIG.

  In step S403, the downsampling execution unit 122 first sets 2 to a variable i indicating the frame position i in step S601 as shown in the flowchart of FIG.

Subsequently, in step S602, 0 is set to the variable u indicating the number of the filter coefficient W _i used at the frame position i, and Ψ _u is set as the filter coefficient W _i used at the frame position i in the subsequent step S603. In step S604, 0 is set as the value of the position correction parameter p _i used at the frame position i.

Subsequently, in step S605, 0 is set to the variable v indicating the number of the filter coefficient W _i-1 used at the frame position i-1, and in step S606, the filter coefficient W _i-1 used at the frame position _i-1. set the [psi _v as, in subsequent step S607, the sets 0 as the value of the position correcting parameter p _i-1 used in the frame position i-1.

Subsequently, in step S608, a frame designated by i, W _i and p _i (a frame generated by filtering using the filter coefficient W _i at the frame position i corrected by p _i ), i− 1, a frame designated by W _i-1 and p _i-1 (a frame generated by filtering using a filter coefficient W _i-1 at a frame position i-1 corrected by p _i-1 ); Between the motion compensation prediction error powers σ _i ² [W _i , W _i−1 , p _i , p _i−1 ].

Subsequently, in step S609, σ _i ² [W _i , W _i−1 , p _i , p _i−1 ] + S _i−1 (W _i−1 , p _i−1 ) is calculated and the working memory 123 is calculated. To store. Since S _i-1 (W _i-1 , p _i-1 ) used at this time is obtained in the process of the flowchart of FIG. 14 and the process of step S614 described later, it is stored in the work memory 123. Use it.

Subsequently, in step S610, it is determined whether or not the value of p _i-1 has reached P-1, and when it is determined that the value has not reached P-1, the process proceeds to step S611, where p _{i After} incrementing the value of ₋₁ by one, the processing returns to step S608.

On the other hand, when it is determined in step S610 that the value of p _i-1 has reached P-1, the process proceeds to step S612 to determine whether the value of variable v has reached N-1. If it is determined that N-1 has not been reached, the process proceeds to step S613, the value of the variable v is incremented by 1, and the process returns to step S606.

In this manner, the increments the value of p _i-1 from 0 one by one until the P-1, the processing of steps S608 / Step S609 while incrementing one by one the value of the variable v from 0 to N-1 As a result of execution, frames designated by i, W _i (a certain filter coefficient) and p _i (a certain correction parameter), and i−1, W _i−1 (W _i−1 = Ψ ₀ ,..., Ψ _N−1 ) and p _i−1 (p _i−1 = 0,..., N−1) and the minimum value σ ₁ ² [W _i , W _i-1 , p _i , p _i-1 ] is calculated, and the calculated σ ₁ ² [W _i , W _i-1 , p _i , p _i-1 ] and its i-1, W _i are calculated. ₋₁ (W _i−1 = Ψ ₀ ,..., Ψ _N−1 ) and p _i−1 (p _i−1 = 0,..., N−1) Sum S _i-1 (W of motion compensation prediction error power from the leading frame position _i-1 , p _i-1 ) are calculated and stored in the work memory 123.

On the other hand, when it is determined in step S612 that the value of the variable v has reached N−1, the process proceeds to step S614, and σ _i ² [W _i , W _i−1 , p _i stored in the work memory 123 is reached. , p _i-1 ] + S _i-1 (W _i-1 , p _i-1 ) (W _i-1 = Ψ ₀ , ...., Ψ _N-1 , p _i-1 = 0, ... , N−1), the minimum value S _i (W _i , p _i ) is identified and stored in the working memory 123, and W _i giving the S _i (W _i , p _i ) is given. _-1 and p _i-1 are stored in the work memory 123 as ^ W _i-1 (W _i , p _i ) and ^ p _i-1 (W _i , p _i ), respectively.

That is, for a given filter coefficient W _i and some p _i, step S608 / Step sigma _i ² treatment was determined by repeated execution of S609 _{[W i, W i-1,} p i, p i-1 ] + S _{i −1} (W _i−1 , p _i−1 ) (W _i−1 = Ψ ₀ ,..., Ψ _N−1 , p _i−1 = 0,..., N−1) The minimum value S _i (W _i , p _i ) is specified and stored in the work memory 123, and W _i-1 and p _i-1 giving the S _i (W _i , p _i ) are determined. Are stored in the work memory 123 as ^ W _i-1 (W _i , p _i ) and ^ p _i-1 (W _i , p _i ), respectively.

Subsequently, in step S615, the value of p _i is determined whether or not reached the P-1, when it is determined that not reached P-1, the process proceeds to step S616, the value of p _i Is incremented by 1, and the process returns to step S605.

On the other hand, in the determination processing in step S615, when the value of p _i to determine the arrival at the P-1, the process proceeds to step S617, the value of the variable u is determined whether or not reaching the N-1 , When it is determined that N−1 has not been reached, the process proceeds to step S618, the value of the variable u is incremented by 1, and the process returns to step S603.

In this manner, the value of p _i is incremented by one from 0 to P-1, the process of step S608 / Step S609 / Step S614 while incrementing the value of the variable u by one from 0 to N-1 by the execution, the value of _{W i (W i = Ψ 0} , ...., Ψ N-1) and the value for _{p i (p i = 0,} ...., N-1) for each of the , Σ _i ² [W _i , W _i−1 , p _i , p _i−1 ] + S _i−1 (W _i−1 , p _i−1 ) (W _i−1 = Ψ ₀ ,..., Ψ _N−1 , p _i−1 = 0,..., N−1), the minimum value S _i (W _i , p _i ) is specified and stored in the work memory 123. And W _i-1 and p _i-1 that give S _i (W _i , p _i ) are respectively represented as ^ W _i-1 (W _i , p _i ) and ^ p _i-1 (W _i , p _i). ) In the work memory 123.

  On the other hand, when it is determined in step S617 that the value of the variable u has reached N-1, the process proceeds to step S618 to determine whether the value of the variable i has reached J / M-1. When it is determined that J / M-1 has not been reached, the process proceeds to step S619, the value of the variable i is incremented by one, and then the process returns to the process of step S602 to perform the process for the next frame position. .

  If it is determined in step S618 that the value of the variable i has reached J / M-1, the process of step S403 in the flowchart of FIG. 13 ends.

In this way, the down-sampling execution unit 122 starts the processing of step S403 in the flowchart of FIG. 13, by executing the flowchart of FIG. 15, i, a frame designated by W _i and p _i, i- 1, S _i (W _i , p _i ) is calculated between the frames specified by W _i−1 and p _i−1 , stored in the work memory 123, and the S _i (W _i, a _{W i-1, p i-} 1 to give p _i), respectively _{^ W i-1 (W i} , p i), ^ p i-1 (W i, the work memory 123 as a p _i) It is processed to store.

[2-4] Outline of Processing As described above, the downsampling execution unit 122 configured as shown in FIG. 9 first uses each filter coefficient and each parameter value for position correction, and uses an equal interval. A frame is generated by filtering a frame located in the vicinity of a frame position defined by downsampling, and each filter coefficient is generated at the previous frame position for each of the generated frames as shown in FIG. And motion compensation prediction error power between each of the similarly generated frames in association with the selection of each position correction parameter value, and the calculated motion compensation prediction error power and the reference frame that is the calculation target By adding the sum of motion compensation prediction error power and the minimum value among them, the previous value is determined. The process of obtaining the optimum filter coefficient and position correction parameter value at the frame position is repeated until the final frame position.

  Subsequently, by detecting the frame having the minimum value among the motion compensated prediction error powers calculated at the final frame position, the optimum filter coefficient and position correction parameter value at the final frame position are obtained. By detecting and tracing the filter coefficient and position correction parameter value specified above starting from the detected filter coefficient and position correction parameter value toward the beginning frame position, it is defined by equal sampling downsampling. The optimum filter coefficient and position correction parameter value are extracted for each frame position.

  Finally, a frame rate conversion process from a high frame rate video signal to a low frame rate video signal is performed by performing filtering at each frame position using the extracted optimum filter coefficient and position correction parameter value. Execute.

[3] Experiments performed to verify the effectiveness of the present invention Next, experiments performed to verify the effectiveness of the present invention will be described.

  In this experiment, an RGB color image captured by a high-speed camera was used as a high frame rate image serving as a reference for frame thinning. The frame rate of the high frame rate video is 1000 [frames / second], the total number of frames is 480 [frames], and the resolution is 640 × 480 [pixels]. The video material is a mini-cart running scene.

  The block size at the time of motion compensation is 16 × 16 [pixel]. Further, the inter-frame prediction is unidirectional prediction, and the reference frame is the immediately preceding frame. As a motion estimation algorithm, a full search based on SSD minimization was performed.

  In this experiment, the PSNR of the predicted image is compared with a conventional average value filter when the value of the downsampling ratio M is 32 and the value of Δ that defines the tap length (2Δ + 1) of the filter coefficient is 1. I went there.

  FIG. 16 illustrates the comparison result. Here, the values in parentheses are the mean square error values. The average value filter is a filter having filter coefficients (1/3, 1/3, 1/3). In the present invention, two values of 16, 64 are set as the value of N (the number of filter coefficients belonging to the filter coefficient set).

  As can be seen from the comparison results shown in FIG. 16, it can be confirmed that the present invention achieves a PSNR improvement of 0.16 [db] with respect to the average value filter. From this, the effectiveness of the present invention could be verified.

  Although the present invention has been described with reference to the illustrated embodiments, the present invention is not limited thereto. For example, in the embodiment, the motion compensation prediction error power is calculated with respect to the previously selected frame, but the motion compensation prediction error power is calculated with a plurality of selected frames. It may be.

  The present invention can be applied when performing frame rate conversion from a high frame rate video signal to a low frame rate video signal. By applying the present invention, the present invention can be obtained by downsampling a high frame rate video signal. When a video signal having a desired frame rate is encoded, the amount of code can be kept lower than the video signal obtained by the conventional frame thinning process, and the image quality caused by time-direction aliasing can be reduced. Deterioration can be prevented.

DESCRIPTION OF SYMBOLS 1 Frame rate conversion apparatus 10 High frame rate video file 11 Low frame rate video file 12 Frame rate downsampling part 120 Motion compensation prediction error calculation part 121 Filter coefficient memory | storage part 122 Downsampling execution part 123 Working memory 1200 Motion estimation part 1220 Input Unit 1221 Processing frame position selection unit 1222 Processing filter coefficient selection unit 1223 Reference filter coefficient determination unit 1224 All frame position end determination unit 1225 Cumulative prediction error minimized final filter coefficient detection unit 1226 Determination reference filter coefficient tracking unit 1227 Filtering execution unit

Claims (9)

A frame rate conversion method for performing downsampling by selecting a filter coefficient from a set of filter coefficients prepared in advance at equally spaced frame positions and filtering a plurality of frames of a high frame rate video signal using the filter coefficient Because
A low frame generated by down-sampling between each of a plurality of frames generated when using each filter coefficient at an equally spaced frame position and each of a plurality of frames generated at a frame position a predetermined number before. A value indicating the coding efficiency of the rate video signal is calculated, and based on the value, a filter coefficient used for generating a frame having a connection that minimizes the sum of the coding efficiency display values from the head frame position is specified. Repeatedly executing from the first frame position toward the last frame position,
By detecting a frame having the minimum sum of the encoding efficiency display values from a plurality of frames generated at the frame position where the execution of the repetition has been completed, a frame having the minimum sum is generated. Detecting the filter coefficients used;
Selecting a filter coefficient to be used at each frame position by tracing the specified filter coefficient from the detected filter coefficient as a starting point,
A featured frame rate conversion method. The frame rate conversion method according to claim 1,
Comprising a step of setting a parameter for position correction for correcting frame positions at equal intervals,
In the process of executing, the processing is executed in such a manner that a plurality of frames generated at the frame position corrected by the position correction parameter are included in the frames generated at equally spaced frame positions.
A featured frame rate conversion method. The frame rate conversion method according to claim 1 or 2,
In the execution process, an addition value of the calculated encoding efficiency display value and the sum of the encoding efficiency display values of the frames at the frame position before the specified number as the calculation target is obtained, and among them, By specifying the minimum value of, the filter coefficient at the frame position before the specified number used to generate a frame having a connection that minimizes the sum of the encoding efficiency display values from the head frame position is specified, and Setting the added value as the sum of the encoding efficiency display values of the original frame,
A featured frame rate conversion method. The frame rate conversion method according to any one of claims 1 to 3,
In the process of executing, calculating the motion compensation prediction error power as the coding efficiency display value,
A featured frame rate conversion method. The frame rate conversion method according to any one of claims 1 to 3,
In the process of executing, calculating the encoding cost obtained when encoding using a predetermined encoder as the encoding efficiency display value,
A featured frame rate conversion method. A frame rate conversion device that performs downsampling by selecting a filter coefficient from a set of filter coefficients prepared in advance at equally spaced frame positions and filtering a plurality of frames of a high frame rate video signal using the filter coefficient Because
A low frame generated by down-sampling between each of a plurality of frames generated when using each filter coefficient at an equally spaced frame position and each of a plurality of frames generated at a frame position a predetermined number before. A value indicating the coding efficiency of the rate video signal is calculated, and based on the value, a filter coefficient used for generating a frame having a connection that minimizes the sum of the coding efficiency display values from the head frame position is specified. Means for repeatedly executing from the first frame position toward the last frame position,
By detecting a frame having the minimum sum of the encoding efficiency display values from a plurality of frames generated at the frame position where the execution of the repetition has been completed, a frame having the minimum sum is generated. Means for detecting the filter coefficients used;
Means for selecting a filter coefficient to be used at each frame position by tracing the specified filter coefficient starting from the detected filter coefficient,
A frame rate conversion device. The frame rate conversion device according to claim 6, wherein
Means for setting position correction parameters for correcting equally spaced frame positions;
The executing means executes processing in such a manner that a plurality of frames generated at a frame position corrected by the position correction parameter are included in frames generated at equally spaced frame positions.
A frame rate conversion device. The frame rate conversion device according to claim 6 or 7,
The means for executing obtains an added value of the calculated encoding efficiency display value and the sum of the encoding efficiency display values of the frames at the frame position before the specified number as the calculation target, and among them, By specifying the minimum value of, the filter coefficient at the frame position before the specified number used to generate a frame having a connection that minimizes the sum of the encoding efficiency display values from the head frame position is specified, and Setting the added value as the sum of the encoding efficiency display values of the original frame,
A frame rate conversion device.   6. A frame rate conversion program for causing a computer to execute the frame rate conversion method according to claim 1.

Images