JP5161163B2 - Frame rate conversion method, frame rate conversion device, and frame rate conversion program - Google Patents

Description

  The present invention relates to a frame rate conversion method and apparatus for converting a frame of a high frame rate video signal into a low frame rate video signal by selecting by downsampling, and a frame rate conversion program used for realizing the frame rate conversion method. And about.

  In recent years, there are growing expectations for super-high-quality images such as large-screen sports images and digital cinema that are full of realism. In response to this, research on high-quality video has been vigorously conducted.

  The following four elements are necessary to realize super high-quality video. That is, spatial resolution, pixel value depth, color reproducibility, and temporal resolution. In response, the former three elements are being studied in applications such as digital cinema and natural vision projects. In addition, improvement of time resolution, that is, an increase in video frame rate, which is indispensable for expressing a natural movement of a subject, has been studied.

  High-speed cameras capable of capturing high frame rate images exceeding 1000 [frame / sec] are already on the market. However, video captured by such a high-speed camera is used for slow playback applications. The upper limit of the frame rate of the video input / output system is asymmetric. Since the upper limit of the current display is about 120 [fps], it is necessary to thin out the frame rate of a video source shot with a high-speed camera in a display format intended for real-time playback.

  Normally, as shown in FIG. 7, downsampling is performed so that the frame time intervals after downsampling are equal (see, for example, Patent Document 1). This is based on the assumption that jerkiness (image quality degradation) occurs in time sampling at non-equal intervals.

  This assumption is correct when the downsampling target is a low frame rate video. However, this is not the case when the object of downsampling is a high frame rate video. When the target of downsampling is a high frame rate video, even if the frames are not arranged at regular intervals, if the deviation from the regular intervals is within a certain threshold, visually large jerkiness (image quality degradation) will occur. Does not occur.

  Therefore, when the target of downsampling is a high frame rate video, it is possible to relax the constraint condition of equal length with respect to the frame interval after downsampling. By relaxing this constraint, the degree of freedom regarding frame selection after downsampling is increased. That is, there is room for studying an optimal frame rate downsampling method from the viewpoint of coding efficiency.

  Against this background, the present inventor has studied a method of selecting a frame after downsampling so as to minimize motion compensation prediction error power (see Non-Patent Document 1).

JP 2004-201165 A Yukihiro Bando, Noriyuki Takamura, Hitoshi Uekura, Yoshiyuki Yashima: “A Study on Downsampling Method for High Frame Rate Video Signals”, Image Coding Symposium, p5-02, 2008

  As described above, when the target of downsampling is a high frame rate video, it is possible to relax the constraint condition of equal length with respect to the frame interval after downsampling. Can be high.

  However, the prior art has not studied this point at all, and from now on, when encoding a low frame rate video obtained by downsampling a high frame rate video, the inter-frame prediction error power is reduced. There was room for improvement in reduction.

  Against this background, the present inventor examined a method of selecting a frame after downsampling so as to minimize motion compensation prediction error power in Non-Patent Document 1, but this In the method, selection frames are set sequentially, and minimization of motion compensation prediction error power in the entire sequence is not guaranteed, and there is room for improvement from that point. Moreover, this method does not consider suppression of jerkiness (image quality degradation), and there is still room for improvement in this respect.

  The present invention has been made in view of such circumstances, and suppresses the code amount in a video encoding process in which a low frame rate video signal obtained by downsampling a high frame rate video signal is input. It is an object of the present invention to establish a downsampling technique with a new frame rate that can reduce the deterioration in subjective image quality.

[1] Basic concept of the present invention For a high frame rate video signal subject to frame rate conversion, the frame interval is δ _t , and the time t = jδ _t (j = 0, 1,...) The pixel value at the position x in the frame is represented as f (x, t) (x = 0, 1,..., X−1).

Consider a case where the number of frames is converted to 1 / M by down-sampling the pixel signal f (x, t). A frame rate ratio M before and after downsampling is called a downsampling ratio. That is, this conversion assumes that the frame rate is converted from “1 / δ _t ” to “1 / Mδ _t ”.

  In the following, a one-dimensional signal will be described as an example for the sake of simplicity, but the same discussion can be easily extended to a two-dimensional signal.

When performing the above conversion by frame thinning, conventionally,
f (x, iMδ _t ) i = 0, 1,...
Thus, a method of multiplying the frame interval by M has been used.

This is based on an empirical rule that the smoothness of motion is lost when the frame interval after downsampling is unequal. This rule of thumb is valid in the case of a conventional frame rate video signal such as δ _t = 1/30 [seconds].

However, this is not the case in the case of a video signal with a high frame rate such as δ _t = 1/1000 [second]. For example, when converting a video signal with a high frame rate with a frame interval of δ _t = 1/1000 [seconds] into a video signal with a low frame rate with a frame interval of 1 / 62.5 [seconds] as M = 16, Even if the frame interval after conversion expands or contracts by about 1/1000 [second], it cannot be visually detected.

  For this reason, image quality deterioration due to non-equal frame intervals is not a problem. Therefore, it is possible to relax the constraint condition of equal length with respect to the frame interval after downsampling. By relaxing this constraint, the degree of freedom regarding frame selection after downsampling is increased.

Therefore, in the present invention, frame selection candidates are
f (x, (iM + L _i ) δ _t ) i = 1, 2,...
L _i = −Δ,..., Δ
Where L ₀ = 0,..., Δ
In this way, the frame is expanded to 2Δ + 1 frame (Δ + 1 frame when i = 0).

Here, L _i is a parameter that takes an integer value in the range of −Δ,..., Δ (hereinafter, L _i is referred to as an expansion / contraction parameter), and is used to specify a frame after downsampling. And 2Δ + 1 ≦ M. Further, Δ represents an existing section of a frame candidate to be down-sampled, and is a parameter given from the outside, for example.

The reason why the expansion / contraction parameter L _i is limited to Δ is that if the value of L _i is too large, the frame interval greatly deviates from the equal length, which causes deterioration of image quality.

  As described above, in the present invention, the frame selection candidates are expanded to 2Δ + 1 frames. However, which frame is selected is determined according to the following criteria.

[Evaluation scale for frame selection in downsampling]
As reference for frame selection, motion compensated prediction error power and sequence jerkiness after downsampling are used.

  H. In typical coding schemes such as H.264 / AVC and MPEG-2, motion compensation interframe prediction is adopted, and reduction of motion compensation prediction error power is important. From the viewpoint of subjective image quality, it is also important to suppress the jerkiness of the sequence after downsampling. Henceforth, the motion compensation prediction error power and the jerkiness of the sequence after downsampling are used as the frame selection criteria.

The signal f (x, (iM + L _i ) δ _t ) is divided into sections B [k] (k = 0, 1,..., K−1) of size S, and each section B [k] Consider motion compensation (estimated displacement d [k]: displacement is synonymous with motion vector) for (k = 0, 1,..., K−1).

The interval B [k] in the frame f (x, (iM + L _i ) δ _t ) refers to the frame f (x, ((i−1) M + L _i−1 ) δ _t ), and the estimated displacement d [k ], The prediction error power after motion compensation in that section can be expressed as the following equation (1).

When L _i and L _i-1 are given, the estimated displacement d [k] (k = 0, k = 0, so as to minimize σ _i ² [L _i , L _i-1 ] according to the equation (1). 1, ...., K-1) set, d _{[k, L i, L i-} 1 ] (k = 0, 1, put ...., K-1) and. At this time, the accuracy of the estimated displacement d [k] is given from the outside (eg, integer pixel accuracy, 1/2 pixel accuracy, 1/4 pixel accuracy, etc.).

  Furthermore, the value of the following formula (2) is used as an evaluation scale for jerkiness.

The jerkiness evaluation measure ε ² [L _i ] shown in the equation (2) represents a difference value (interframe difference value) between a frame down-sampled with an equal length and a frame down-sampled with a variable length. .

  In addition, the value of the following formula (3) can be used as an evaluation scale for jerkiness.

Here, L _i is the distance between a frame that is down-sampled with an equal length and a frame that is down-sampled with a variable length.

The jerkiness evaluation measure ε ² [L _i ] shown in Equation (3) weights the inter-frame difference value between an equal-length down-sampled frame and a variable-length down-sampled frame by the distance between both frames. The value can be said.

  Moreover, it is also possible to use the value of the following formula (4) as an evaluation scale for jerkiness.

Here, | M + L _i −L _i−1 | is a distance between the reference frame and the predicted frame.

It can be said that the jerkiness evaluation measure ε ² [L _i ] shown in the equation (4) is a value obtained by weighting the inter-frame difference value between the reference frame and the predicted frame by the distance between the frames.

  Moreover, it is also possible to use the value of the following formula (5) as an evaluation scale for jerkiness.

Here, α is a parameter given from the outside, and is a threshold for | L _i −L _i−1 |. | M + L _i −L _i−1 | is a distance between the reference frame and the predicted frame, and considering that M is a constant, whether or not this distance is equal to or greater than a certain threshold value is determined by | L It is possible to evaluate using _i −L _i−1 |.

The jerkiness evaluation measure ε ² [L _i ] shown in the equation (5) is a value obtained by weighting the inter-frame difference value between the reference frame and the predicted frame by the distance between the frames, and | L _i − This is set based on the hypothesis that jerkiness cannot be detected when L _i-1 | is equal to or smaller than a predetermined threshold value α.

In the present invention, a cost function Ψ [L _i , L _i-1 ] of the following equation (6) is used as an evaluation measure for down-sampling a frame in order to consider coding efficiency and suppression of jerkiness.

Here, λ is a weighting factor and is given from the outside. Also, ε ² [L _i ] is calculated according to the formulas (2), (3), (4), and (5).

In the present invention, a frame that optimizes the trade-off between the motion compensation prediction error and the jerkiness after down-sampling is selected through the minimization of the cost function Ψ [L _i , L _i-1 ].

Here, it should be noted, is a L _i method setting (selection method). If L _i is set so as to minimize Ψ [L _i , L _i-1 ], the value of the cost function in the entire sequence ΣΨ [L _i , L _i-1 ]
However, Σ is the sum of i = 1 to (J / M−1), and J does not mean that the number of frames is minimized. This is because selection of L _i is because also affects Ψ [L _i, L _i-1].

  Therefore, the parameters to be obtained are J / M parameters satisfying the following expression (7).

However, the possible combinations of _{J / M} parameters (L ₀ ,..., L _{J / M−1} ) are (Δ + 1) × (2Δ + 1) ^{J / M−1.} It is computationally difficult to search for combinations (L ^* ₀ ,..., L ^* _{J / M-1} ) brute force.

Therefore, in the present invention, paying attention to the fact that Ψ [L _i , L _i-1 ] depends only on the result of the immediately preceding frame, the optimal solution is calculated as follows.

First, for all possible values as L _i, the optimal L _i-1, ...., ΣΨ when using the L ₀ [L _i, L _i-1] (where, sigma is i = 1 to _i ) is defined as S _i (L _i ).

Here, paying attention to the fact that Ψ [L _i , L _i-1 ] depends only on the result of the previous frame, S _i (L _i ) uses S _i-1 (L _i-1 ), and FIG. As shown in the following equation (8). Note that i = 1,..., (J / M−1).

S _i (L _i ) (L _i = −Δ,..., Δ) calculated in accordance with this equation (8) is stored in the memory and used in the calculation of S _{i + 1} (L _{i + 1} ). Shall. Further, L _i-1 giving the minimum value of the equation (8) is set as ^ L _i-1 (L _i ) as shown in FIG. This ^ L _i-1 (L _i ) (L _i = −Δ,..., Δ) is all stored in the memory. Here, the symbol ^ in “^ X” (X is a letter) indicates a symbol attached to “X”.

  The minimization problem of Equation (7) can be expressed as Equation (9) below.

If L _{J / M-1} that minimizes the equation (9) is L ^* _{J / M-1} , this L ^* _{J / M-1} can be expressed by the following equation (10).

Since the optimal solution for L ^* _{J / M-2} is stored in the memory as ^ L _{J / M-2} (L _{J / M-1} ), L ^* _{J / M-2} = Let L _{J / M-2} (L _{J / M-1} ). Hereafter, the same reference process is
_{^{L * J / M-3 =}} ^ L J / M-3 (L J / M-2), ·····, L * 0 = ^ L 0 (L 1)
Repeat as.

In the above description, the motion-compensated prediction error power is used as a value indicating the coding efficiency in the cost function Ψ [L _i , L _i-1 ] of the equation (6), but a specific encoder (for example, An encoding cost obtained when encoding by an H.264-compliant encoder) may be used.

[2] Configuration of the Present Invention Next, the configuration of the present invention will be described.

  The frame rate conversion apparatus according to the present invention is defined by (1) downsampling at equal intervals in order to realize conversion to a low frame rate video signal by selecting a frame of a high frame rate video signal by downsampling. Low frame rate video generated when down-sampling each of a plurality of frames in a predetermined range of frame positions to be performed with each of a plurality of frames in a predetermined range of frame positions before that An evaluation measure indicating the coding efficiency and subjective image quality of a signal, and an evaluation measure indicating a higher evaluation as the value decreases, and based on the calculated evaluation measure, a predetermined range of the previous frame position Minimize the sum of evaluation scales from the first frame position among multiple frames in The execution means for repeatedly executing from the head frame position toward the final frame position, and (2) the execution means calculated for each of the plurality of frames at the final frame position. Detecting means for detecting a frame having the minimum value among the sums of the evaluation scales; and (3) tracing the frames specified by the executing means starting from the frame detected by the detecting means, thereby reducing the equal length interval. Selection means is provided for extracting frames for each frame position defined by sampling and selecting them as frames after downsampling.

  When adopting this configuration, the execution means obtains an added value of the calculated evaluation scale and the sum of the evaluation scales of the frames at the previous frame position to be calculated, and calculates the minimum value among them. By specifying, among the multiple frames within the predetermined range of the previous frame position, the frame having the connection that minimizes the sum of the evaluation scale from the first frame position is specified, and the added value is specified. Process to set as the evaluation scale of the original frame.

  Further, the execution means, as an evaluation measure indicating the encoding efficiency and subjective image quality of the low frame rate video signal generated when down-sampling, the encoding efficiency value indicating higher evaluation as the value decreases, and the value is In some cases, a weighted sum with a subjective image quality value indicating a higher evaluation as the value decreases.

  At this time, the executing means may calculate the motion compensation prediction error power as the encoding efficiency value or the encoding cost obtained when encoding using a predetermined encoder.

  In addition, the execution means calculates a difference value between a frame sampled at an equal length and a frame sampled at a variable length as a subjective image quality value, or a frame sampled at an equal length and a frame sampled at a variable length. A value obtained by weighting the difference value between the frames with the distance between the two frames or a value obtained by weighting the difference value between the reference frame and the predicted frame with the distance between the two frames may be calculated.

  The frame rate conversion method of the present invention realized by the operation of each of the above processing means can also be realized by a computer program. This computer program can be provided by being recorded on an appropriate computer-readable recording medium. The present invention is realized by being provided via a network, installed when the present invention is implemented, and operating on a control means such as a CPU.

In the frame rate conversion apparatus of the present invention configured as described above, first, for each of a plurality of frames within a predetermined range of frame positions defined by equal-length interval downsampling, for example, as shown in FIG. In addition, an evaluation scale (refer to FIG. 5) showing coding efficiency and subjective image quality for a low frame rate video signal generated when down-sampling with each of a plurality of frames within a predetermined range of the previous frame position. 1 calculates Ψ [L _i , L _i-1 ]), and the sum of the calculated evaluation scale and the evaluation scale of the frame at the previous frame position that is the calculation target (S _{i in} FIG. 1). _-1 (L _i-1 )) is obtained, and the minimum value among them is specified, so that the first frame is selected from a plurality of frames within a predetermined range of the previous frame position. The process of identifying the frame that has the connection that minimizes the sum of the evaluation scales from the current position and repeatedly setting the added value as the sum of the evaluation scales of the source frame is repeated until the final frame position. To do.

  Subsequently, for the sum of the evaluation scales calculated for each of the plurality of frames at the final frame position, a frame having the minimum value among them is detected, and the previously identified frame from the detected frame as a starting point is detected. By tracing toward the first frame position, frames are extracted for each of the frame positions defined by equal-length interval down-sampling, and these are selected as frames after down-sampling.

  In this way, the frame rate conversion device of the present invention performs the frame rate conversion from the high frame rate video signal to the low frame rate video signal with respect to the entire sequence of the low frame rate video signal after the downsampling. The frame rate conversion process is executed in consideration of the coding efficiency and the suppression of jerkiness.

  In the present invention, when performing frame rate conversion from a high frame rate video signal to a low frame rate video signal, the coding efficiency and jerkiness suppression for the entire sequence of the low frame rate video signal after downsampling are taken into account. Then, the frame rate conversion process is executed.

  Thus, according to the present invention, when encoding a video signal having a desired frame rate obtained by downsampling a high frame rate video signal, the video signal obtained by thinning out the uniform frame interval is used. The amount of codes can be kept low.

  According to the present invention, the amount of codes can be suppressed to be lower than that of a video signal obtained by thinning out non-equal-length frame intervals performed without considering encoding efficiency and jerkiness suppression for all sequences. At the same time, jerkiness can be suppressed.

It is explanatory drawing of the calculation process of S _i (L _i ) calculated in the present invention. It is an apparatus block diagram of the frame rate conversion apparatus of this invention. It is an apparatus block diagram of the frame rate conversion apparatus of this invention. It is a flowchart which a downsampling execution part performs. It is a flowchart which a downsampling execution part performs. It is a flowchart which a downsampling execution part performs. It is explanatory drawing of the frame rate downsampling process by a prior art.

  Hereinafter, the present invention will be described in detail according to embodiments.

  FIG. 2 shows an example of the device configuration of the frame rate conversion device 1 of the present invention.

  As shown in this figure, the frame rate conversion apparatus 1 of the present invention includes a high frame rate video file 10 storing a high frame rate video signal to be subjected to frame rate conversion processing, and a low frame rate subjected to frame rate conversion processing. A low frame rate video file 11 that stores a video signal, and a frame rate downsampling unit 12 that executes a frame rate conversion process from a high frame rate video signal to a low frame rate video signal are provided.

The frame rate downsampling unit 12 includes a cost function calculation unit 120 that calculates a function value of the cost function Ψ [L _i , L _i-1 ] represented by Equation (6), and a cost calculated by the cost function calculation unit 120. A downsampling execution unit 121 that executes a frame rate downsampling process by selecting a frame from the frames of the high frame rate video signal using the function value, and a work for storing work data of the downsampling execution unit 121 Memory 122.

The cost function calculation unit 120 includes a motion estimation unit 1201, a motion compensation prediction error calculation unit 1200 that calculates the motion compensation prediction error power represented by Equation (1), Equation (2), Equation (3), By providing a jerkiness evaluation scale calculation unit 1202 that calculates the jerkiness evaluation scale ε ² [L _i ] shown in the expressions (4) and (5), the cost function Ψ [L _i , L _i-1 ] is calculated.

[1] Process of Motion Estimation Unit 1201 Before entering the description of the frame selection process executed by the downsampling execution unit 121, the process of the motion estimation unit 1201 included in the motion compensation prediction error calculation unit 1200 will be described.

  The motion estimator 1201 receives the prediction target frame and the reference frame as input, and executes the following processing to thereby execute a motion vector d [k] for each block in the prediction target frame, that is, d in Expression (1). [K] (k = 0, 1,..., K−1) is estimated. Here, k is an index for identifying a block.

Processing: • Finds a motion vector d [k] that minimizes the term of Σ {x∈B [k]} (prediction error sum of k-th block) in Equation (1). Selection is made from values in the range −D ≦ d [k] ≦ D−1. • The selection method calculates the sum of prediction errors when using the candidate vectors for all candidate vectors in the search range. Then, the motion compensation prediction error calculation unit 1200, which is performed by setting the vector that minimizes the prediction error sum to d [k], uses the vector d [k] estimated by the motion estimation unit 1201 and uses Equation (1). Based on this, the motion compensation prediction error power between the prediction target frame and the reference frame is calculated.

[2] Processing of Downsampling Execution Unit 121 Next, frame rate downsampling processing executed by the downsampling execution unit 121 will be described.

  FIG. 3 illustrates an example of a device configuration of the downsampling execution unit 121.

  As shown in this figure, the downsampling execution unit 121 includes an input unit 1210, a processing frame position selection unit 1211, a processing frame selection unit 1212, a reference frame determination unit 1213, an all frame position end determination unit 1214, The accumulated cost function value minimized final frame detection unit 1215 and a decision reference frame tracking unit 1216 are provided.

The input unit 1210 reads the number of frames J of the high frame rate video signal, the frame interval δ _t of the high frame rate video signal, the downsampling ratio M, and the expansion / contraction parameter Δ.

Processing frame position selecting unit 1211, down-sampling frame position defined by the down-sampling, such as interval length based on the downsampling ratios M and frame interval [delta] _t (hereinafter, abbreviated as frame position) as selection, the head of the Starting from the frame position immediately after the frame position, frame positions up to the final frame position are sequentially selected as processing frame positions.

  The processing frame selection unit 1212 sequentially selects the processing frame position selected by the processing frame position selection unit 1211 and 2Δ + 1 frames located in the vicinity thereof as processing frames.

  The reference frame determination unit 1213 calculates the cost function expressed by Expression (6) between the processing frame selected by the processing frame selection unit 1212 and each of the 2Δ + 1 frames positioned in the vicinity of the previous frame position. Is calculated, and an added value of the calculated cost function value and the sum of the cost function values of the frame at the previous frame position to be calculated is obtained (accumulated cost function value calculation function). ) By specifying the minimum value among them (minimum cumulative cost function value specifying function), the sum of cost function values from the first frame position can be calculated from 2Δ + 1 frames at the previous frame position. A reference frame is determined by specifying a frame having a connection to be minimized.

  The all frame position end determination unit 1214 determines whether or not processing has been performed for all frame positions up to the final frame position, and determines that processing has not been performed for all frame positions. When instructing the unit 1211 to select the next frame position and determining that all the frame positions have been processed, the cumulative cost function value minimized final frame detecting unit 1215 is instructed to execute the process. To do.

  The accumulated cost function value minimized final frame detection unit 1215 calculates the sum of the cost function values calculated by the final frame position and 2Δ + 1 frames located in the vicinity thereof (calculated by the reference frame determination unit 1213). By detecting the frame having the minimum value among them, the final frame that minimizes the sum of the cost function values from the head frame position is detected.

  The determined reference frame tracking unit 1216 traces the reference frame determined by the reference frame determining unit 1213 from the last frame detected by the accumulated cost function value minimized final frame detecting unit 1215 as a starting point, thereby downsampling at equal intervals. Extract reference frames for each of the defined frame positions and select them as frames after downsampling.

  The flow of the frame rate downsampling process executed by the downsampling execution unit 121 configured as described above is shown below.

[I] When using the value of formula (2) as an evaluation scale for jerkiness. Captured image signal (high frame rate video signal subject frame rate conversion process), reads the number of frames J, the frame rate (used in order to calculate the frame interval [delta] _t) 2. 2. Read downsampling ratio M. 3. Read the weighting factor λ used for calculating the cost function. 4. Perform the following processing for i = 1,..., J / M-1. 5. Perform the following processing for L _i = −Δ,. The following processing is performed for L _i-1 = −Δ,..., Δ
(However, in the case of _{L 0, L 0 = 0,} ...., Δ)
7). When the reference frame is f (x, ((i−1) M + L _i−1 ) δ _t ), the motion compensation prediction error power (equation (1)) with respect to f (x, (iM + L _i ) δ _t ) is minimized. The motion vector to be converted is obtained. The method for obtaining the motion vector is given from the outside. For example, a full search method is used that squeezes candidates within the search range. 7. Store the motion compensation prediction error power when using the obtained motion vector in σ _i ² [L _i , L _i-1 ]. As an evaluation measure of jerkiness, a difference value ε ² [L _i ] between a frame down-sampled with an equal length and a frame down-sampled with a variable length is calculated. The specific calculation method follows formula (2). The cost function Ψ [L _i , L _i-1 ] is obtained by the following equation as a weighted sum of the motion compensation prediction error power and the jerkiness evaluation scale.
Ψ [L _i , L _i-1 ] = σ _i ² [L _i , L _i-1 ] + λε ² [L _i ]
Ten. Calculate the value of Ψ [L _i , L _i-1 ] + S _i-1 (L _i-1 )
(However, in the case of L _0, to calculate the value of Ψ [L _1, L _0])
11． The minimum value of Ψ [L _i , L _i-1 ] + S _i-1 (L _i-1 ) (L _i-1 = −Δ,..., Δ) is set to S _i (L _i ). Store
(However, in the case of L _0, [psi [L _1, L _0] (L ₀ = 0, ...., and stores the minimum value among the delta) in S ₁ (L ₁₎₎
12． Store L _i-1 giving S _i (L _i ) in ^ L _i-1 (L _i ).
13. Storing S J _{/ M-1} and (L J _{/ M-1)} to L _{J / M-1} to minimize the L ^* _{J / M-1}
14. i = J / M-2,...
15． L ^* _i-1 = ^ L _i-1 (L _i ) is calculated
16． As the i-th frame after downsampling, f (x, (iM + L _i ^* ) δ _t )
(X = 0,..., X−1) are stored.

[Ii] When using the value of equation (3) as an evaluation scale for jerkiness. Captured image signal (high frame rate video signal subject frame rate conversion process), reads the number of frames J, the frame rate (used in order to calculate the frame interval [delta] _t) 2. 2. Read downsampling ratio M. 3. Read the weighting factor λ used for calculating the cost function. 4. Perform the following processing for i = 1,..., J / M-1. 5. Perform the following processing for L _i = −Δ,. The following processing is performed for L _i-1 = −Δ,..., Δ
(However, in the case of _{L 0, L 0 = 0,} ...., Δ)
7). When the reference frame is f (x, ((i−1) M + L _i−1 ) δ _t ), the motion compensation prediction error power (equation (1)) with respect to f (x, (iM + L _i ) δ _t ) is minimized. The motion vector to be converted is obtained. The method for obtaining the motion vector is given from the outside. For example, a full search method is used that squeezes candidates within the search range. 7. Store the motion compensation prediction error power when using the obtained motion vector in σ _i ² [L _i , L _i-1 ]. 8. Calculate the difference value tildeε ² [L _i ] between the frame down-sampled with equal length and the frame down-sampled with variable length. Calculate the absolute value | L _i | of the interframe distance between a frame down-sampled with equal length and a frame down-sampled with variable length
Ten. A value obtained by multiplying the difference value calculated in step 8 by the inter-frame distance calculated in step 9 is calculated and used as an evaluation measure ε ² [L _i ] of jerkiness. The specific calculation method of 8. to 10. follows formula (3).
11． As a weighted sum of the motion compensation prediction error power and the jerkiness evaluation scale, the cost function Ψ [L _i , L _i-1 ] is obtained by
Ψ [L _i , L _i-1 ] = σ _i ² [L _i , L _i-1 ] + λε ² [L _i ]
12． Calculate the value of Ψ [L _i , L _i-1 ] + S _i-1 (L _i-1 )
(However, in the case of L _0, to calculate the value of Ψ [L _1, L _0])
13. The minimum value of Ψ [L _i , L _i-1 ] + S _i-1 (L _i-1 ) (L _i-1 = −Δ,..., Δ) is set to S _i (L _i ). Store
(However, in the case of L _0, [psi [L _1, L _0] (L ₀ = 0, ...., and stores the minimum value among the delta) in S ₁ (L ₁₎₎
14. Store L _i-1 giving S _i (L _i ) in ^ L _i-1 (L _i ).
15． Storing S J _{/ M-1} and (L J _{/ M-1)} to L _{J / M-1} to minimize the L ^* _{J / M-1}
16． i = J / M-2,...
17． L ^* _i-1 = ^ L _i-1 (L _i ) is calculated
18． As the i-th frame after downsampling, f (x, (iM + L _i ^* ) δ _t )
(X = 0,..., X−1) are stored.

In this way, first, the downsampling execution unit 121, as shown in FIG. 1, 1 for each frame position defined by equal-length interval downsampling and 2Δ + 1 frames located in the vicinity thereof. A function value of a cost function, which is an evaluation measure for down-sampling the frame, is calculated between the previous frame position and each of 2Δ + 1 frames located in the vicinity thereof, and the calculated cost function value and By calculating the sum of the cost function values of the frame at the previous frame position to be calculated and specifying the minimum value among them, it is possible to locate the previous frame position and its vicinity. Among the 2Δ + 1 frames that are positioned, there is a connection that minimizes the sum of cost function values from the first frame position. A process of specifying the frame, repeatedly executed until the final frame position,
Subsequently, for the sum of the cost function values calculated for the final frame position and each of the 2Δ + 1 frames located in the vicinity thereof, the frame having the minimum value is detected, and the detected frame is used as the starting point. By tracing the previously specified frame toward the first frame position, extracting frames for each frame position defined by equal-length interval down-sampling, and selecting them as frames after down-sampling The frame rate conversion process from the high frame rate video signal to the low frame rate video signal is executed.

  Next, the present invention will be described in detail according to examples.

  4 to 6 show an example of a flowchart of the frame rate downsampling process executed by the downsampling execution unit 121. FIG.

  Next, according to this flowchart, the frame rate downsampling process executed by the downsampling execution unit 121 will be described in detail.

Here, in the following description, it is assumed that the cost function Ψ [L _i , L _i-1 ] shown in Expression (6) is used as an evaluation measure for downsampling the frame, and this cost function Ψ Use _{one of} Equation (2), Equation (3), Equation (4), and Equation (5) as the jerkiness evaluation measure ε ² [L _i ] described in [L _i , L _i-1 ] Is assumed.

When there is a request for execution of the frame rate downsampling process, the downsampling execution unit 121 first, in step S10, the high frame rate video signal to be subjected to the frame rate conversion process, as shown in the flowchart of FIG. reads its frame number J, and the frame interval [delta] _t, and down-sampling ratio M, and a stretch parameter delta.

Subsequently, in step S20, S ₁ (L ₁ ) is calculated between the frame designated by i = 1 and L ₁ and the frame designated by i = 0 and L ₀ , and this is calculated as work memory 122. And L ₀ giving S ₁ (L ₁ ) is stored in work memory 122 as L ₀ (L ₁ ). Details of the processing in step S20 will be described later with reference to the flowchart of FIG.

Subsequently, in step S30, for 2 ≦ i ≦ J / M- 1, S i (L i) between a frame to specify the i and L _i, a i-1 and L _i-1 of the specified frame Is stored in the work memory 122, and L _i-1 giving S _i (L _i ) is stored in the work memory 122 as ^ L _i-1 (L _i ). Details of the processing in step S30 will be described later with reference to the flowchart of FIG.

Subsequently, in step S40, and stores the _{S J / M-1 (L} J / M-1) working memory 122 L _{J / M-1} to minimize the L ^* _{J / M-1} to.

  Subsequently, in step S50, J / M-2 is set to the variable i.

Subsequently, in step S60, L ^* _i-1 is calculated and stored in the work memory 122 in accordance with L ^* _i-1 = ^ _Li-1 ( _Li ).

  Subsequently, in step S70, the value of the variable i is decremented by one.

  Subsequently, in step S80, it is determined whether or not the value of the variable i has reached 0, and when it is determined that the value has not reached 0, the process returns to step S60 to determine that 0 has been reached. If so, the process proceeds to step S90.

Subsequently, in step S90, the i-th frame after downsampling, f stores ^{(x, (iM + L *} i) δ t) (x = 0, ...., X-1).

  Next, details of the process executed in step S20 will be described with reference to the flowchart of FIG.

Downsampling execution unit 121 starts the processing of step S20, as shown in the flowchart of FIG. 5, first, at step S201, sets a variable i, at subsequent step S202, the variable L _i - Δ is set, and in subsequent step S203, 0 is set to the variable L _i−1 . That is, 1 is set to the variable i, −Δ is set to L ₁ , and 0 is set to L ₀ .

Subsequently, in step S204, in the case of i = 1, the frame designated by i and L _i, the minimum value of the motion compensated prediction error power between i-1 and L _i-1 of the specified frame sigma _i ² [L _i , L _i-1 ] is calculated, and the function value of the cost function Ψ [L _i , L _i-1 ] taking into account the jerkiness evaluation measure ε ² [L _i ] is calculated. It is stored in the work memory 122. That is, the function value of the cost function Ψ [L ₁ , L ₀ ] is calculated and stored in the work memory 122.

Subsequently, in step S205, in the case of i = 1, the value of the variable L _i-1 is incremented by 1, and in the subsequent step S206, it is determined whether or not the value of the variable L _i-1 is larger than Δ. If it is determined that it is not greater than Δ, the process returns to step S204. That is, when the value of L ₀ is incremented by 1 and it is determined whether or not the value is greater than Δ, and when it is determined that the value is not greater than Δ, the process returns to step S204. is there.

In this way, in the case of i = 1, by executing the process of step S204 while incrementing the value of the variable L ₀ by 1 from 0 to Δ, the frame designated by i = 1 and the variable L ₁ A function value of the cost function Ψ [L ₁ , L ₀ ] is calculated between each of the Δ + 1 frames at the previous frame position and stored in the work memory 122.

On the other hand, when it is determined in step S206 that the value of the variable L _i-1 is larger than Δ, the process proceeds to step S207, and the cost function value Ψ [L _i , L _i− stored in the work memory 122 is reached. to identify the minimum value S _i (L _i) in _1], and stores it in the working memory 122, the S _i (L _i) a L _i-1 to give ^ L _i-1 (L _i ) is stored in the working memory 122.

That is, when i = 1, the cost function value Ψ [L ₁ , L ₀ ] (L ₀ = 0,..., Δ) obtained by repeatedly executing Step S204 for a certain value of the variable L _1. ) identifies the minimum value S ₁ (L ₁₎ in the, working stores it in the work memory 122, as its S ₁ to L ₀ to give _{(L 1) ^ L 0 (} L 1) It is stored in the memory 122.

Subsequently, in step S208, in the case of i = 1, it is incremented by one the value of the variable L _i, in the following step S209, it is determined whether or not the value of the variable L _i is greater than delta, When it is determined that it is not greater than Δ, the process returns to step S203. That is, when the value of the variable L ₁ is incremented by 1 and it is determined whether or not the value is greater than Δ, it is determined that the value is not greater than Δ, the process returns to step S203. It is.

In this way, when i = 1, the value of the variable L ₁ is incremented by one from −Δ to Δ, and the process of step S204 / step S207 is executed, so that the value of the variable L ₁ is −Δ. ˜Δ, the function value of the cost function Ψ [L ₁ , L ₀ ] (L ₀ = 0,..., Δ) is obtained, and the minimum value S ₁ (L ₁ ) is specified. Are stored in the work memory 122, and L ₀ giving S ₁ (L ₁ ) is stored in the work memory 122 as L ₀ (L ₁ ).

Then, in step S209, in the case of i = 1, when it is determined that the value of the variable L _i is larger than Δ, the process ends in step S20. That is, when it is determined that the value of L ₁ is greater than Δ, the process of step S20 is terminated.

In this way, the down-sampling execution unit 121 starts the processing of step S20, by executing the flowchart of FIG. 5, a frame designated by i = 1 and L _1, designated i = 0 and L ₀ S ₁ (L ₁ ) is calculated with respect to the frame to be stored and stored in the work memory 122, and L ₀ giving that S ₁ (L ₁ ) is set as ^ L ₀ (L ₁ ). It is processed so as to be stored in the memory 122.

  Next, details of the process executed in step S30 will be described with reference to the flowchart of FIG.

Downsampling execution unit 121 starts the processing of step S30, as shown in the flowchart of FIG. 6, first of all, at step S301, sets 2 to the variable i, the following step S302, the variable L _i - Δ is set, and in step S303, -Δ is set to the variable Li _-1 .

Subsequently, in step S304, i and L and the frame designated by _i, i-1 and L a minimum value of motion compensated prediction error power between the _i-1 of the specified frame sigma _i ² [L _i, L _{i −1} ] is calculated, and the function value of the cost function Ψ [L _i , L _i-1 ] taking into account the jerkiness evaluation measure ε ² [L _i ] is calculated.

In step S 305, Ψ [L _i , L _i-1 ] + S _i-1 (L _i-1 ) is calculated and stored in the work memory 122.

That is, in the case of i = 2, S ₁ (L ₁ ) has already been calculated according to the process of the flowchart of FIG. 5, so that S ₁ (L ₁ ) is used to satisfy Ψ [L ₂ , L ₁ ] + S ₁ (L ₁ ) is calculated and stored in the working memory 122. If i> 2, since S _i-1 (L _i-1 ) has already been calculated in accordance with the processing of step S308 described later, the S _i-1 (L _i-1 ) is used. , Ψ [L _i , L _i-1 ] + S _i-1 (L _i-1 ) is calculated and stored in the work memory 122.

Subsequently, in step S306, the value of the variable L _i-1 is incremented by one, and in the subsequent step S307, it is determined whether or not the value of the variable L _i-1 is greater than Δ, When it is determined that the value has not increased, the process returns to step S304.

In this way, by executing the processing of step S304 / step S305 while incrementing the value of the variable L _i-1 one by one from −Δ to Δ, the frame designated by the variable i and the variable L _i , 1 The minimum value σ _i ² [L _i , L _i-1 ] of motion compensation prediction error power is calculated between each of 2Δ + 1 frames at the previous frame position, and is used to evaluate the jerkiness evaluation measure ε ² Calculate the function value of the cost function Ψ [L _i , L _i-1 ] taking into account [L _i ], and calculate from the calculated cost function value and the top frame position of each of the 2Δ + 1 frames. The sum of the cost function values and the total sum S _i-1 (L _i-1 ) is calculated and stored in the work memory 122.

On the other hand, when it is determined in step S307 that the value of the variable L _i-1 is larger than Δ, the process proceeds to step S308, and Ψ [L _i , L _i-1 ] + S stored in the work memory 122. The minimum value S _i (L _i ) in _{i −1} (L _i−1 ) (L _i−1 = −Δ,..., Δ) is identified and stored in the work memory 122. At the same time, L _i-1 giving the S _i (L _i ) is stored in the work memory 122 as ^ L _i-1 (L _i ).

That is, Ψ [L _i , L _i-1 obtained by repeatedly executing step S304 / step S305 for a certain value of variable i and variable L _i (i = 2, L _i = −Δ at this time). ] The minimum value S _i (L _i ) in + S _i-1 (L _i-1 ) (L _i-1 = −Δ,..., Δ) is specified and stored in the work memory 122. stores, is to stored in the working memory 122 as a L _i which gives the _{S i (L i) ^ L} i-1 (L i).

Subsequently, in step S309, the incremented by one the value of the variable L _i, in the subsequent step S310, the it is determined whether or not the value of the variable L _i is larger than delta, not greater than delta When this is determined, the process returns to step S303.

In this manner, the value of the variable L _i by executing the processing in steps S304 / Step S305 / Step S308 while incrementing one by one from -Δ to delta, each -Δ~Δ value of the variable L _i , The minimum value S _i (L _i ) in Ψ [L _i , L _i-1 ] + S _i-1 (L _i-1 ) (L _i-1 = −Δ,..., Δ) Specifically, it is stored in the work memory 122, and L _i-1 giving its S _i (L _i ) is stored in the work memory 122 as ^ L _i-1 (L _i ).

On the other hand, in step S310, the value of the variable L _i is when it is determined that is greater than Δ, the process proceeds to step S311, increments by one the value of the variable i.

Subsequently, in step S312, it is determined whether or not the value of the variable i is greater than J / M-1, and when it is determined that the value is not greater than J / M-1, the process proceeds to step S302. By returning to the processing, S _i (L _i ) is specified and stored in the work memory 122 up to the final frame position, and L _i giving the S _i (L _i ) is set to ^ L _i-1 (L _i ), the process of storing in the work memory 122 is executed.

  In step S312, when it is determined that the value of the variable i is larger than J / M-1, the process of step S30 is terminated.

In this way, the down-sampling execution unit 121 starts the processing of step S30, by executing the flowchart of FIG. 6, for 2 ≦ i ≦ J / M- 1, a frame designated by i and L _i , I−1 and L _i−1 , calculate S _i (L _i ), store it in the work memory 122, and give L _i (S _i (L _i )) _-1 is processed so as to be stored in the work memory 122 as ^ L _i-1 (L _i ).

As described above, the down-sampling execution unit 121 first executes the flowcharts of FIGS. 4 to 6 so that, first, as shown in FIG. And for each of 2Δ + 1 frames located in the vicinity thereof, a function value of a cost function considering a jerkiness evaluation measure between the previous frame position and each of 2Δ + 1 frames located in the vicinity thereof is obtained. By calculating, adding the calculated cost function value and the sum of the cost function values of the frame at the previous frame position that is the target of calculation, and specifying the minimum value among them Of the 2Δ + 1 frames located in the previous frame position and its vicinity, the cost function value from the first frame position A process of identifying a frame having a connection that minimizes the sum repeatedly executed until the final frame position,
Subsequently, for the sum of the cost function values calculated for the final frame position and each of the 2Δ + 1 frames located in the vicinity thereof, the frame having the minimum value is detected, and the detected frame is used as the starting point. By tracing the previously specified frame toward the first frame position, extracting frames for each frame position defined by equal-length interval down-sampling, and selecting them as frames after down-sampling The frame rate conversion process from the high frame rate video signal to the low frame rate video signal is executed.

  Although the present invention has been described with reference to the illustrated embodiments, the present invention is not limited thereto. For example, in the embodiment, the function value of the cost function is calculated with respect to the previous frame position according to Equation (6), but in accordance with the encoding process of the low frame rate video signal, 1 The function value of the cost function may be calculated not with the previous frame position but with the previous frame position.

  The present invention can be applied when performing frame rate conversion from a high frame rate video signal to a low frame rate video signal. By applying the present invention, the present invention can be obtained by downsampling a high frame rate video signal. When a video signal having a desired frame rate is encoded, the code amount can be suppressed to be lower than that of a video signal obtained by thinning out uniform frame intervals.

  And, by applying the present invention, the code amount can be kept lower than the video signal obtained by thinning out non-equal length frame intervals without considering the coding efficiency and jerkiness suppression for all sequences. Will be able to suppress jerkiness.

DESCRIPTION OF SYMBOLS 1 Frame rate conversion apparatus 10 High frame rate video file 11 Low frame rate video file 12 Frame rate downsampling part 120 Cost function calculation part 121 Downsampling execution part 122 Work memory 1200 Motion compensation prediction error calculation part 1201 Motion estimation part 1202 Jerkyness Evaluation scale calculation unit 1210 Input unit 1211 Processing frame position selection unit 1212 Processing frame selection unit 1213 Reference frame determination unit 1214 All frame position end determination unit 1215 Cumulative cost function value minimized final frame detection unit 1216 Determination reference frame tracking unit

Claims (10)

A frame rate conversion method of converting a frame of a high frame rate video signal into a low frame rate video signal by selecting by downsampling,
When downsampling each of a plurality of frames within a predetermined range of frame positions defined by equal-length interval downsampling with each of a plurality of frames within a predetermined range of frame positions before that Is an evaluation scale indicating the coding efficiency and subjective image quality for the low frame rate video signal generated in the above, and calculating an evaluation scale indicating a higher evaluation as the value decreases, and based on the calculated evaluation scale, From the first frame position to the last frame position, a frame having a connection that minimizes the sum of the evaluation scales from the first frame position is identified from a plurality of frames within the predetermined range of the previous frame position. The process of repeated execution towards
Detecting a frame having a minimum value among the sums of the evaluation measures calculated for each of the plurality of frames at the final frame position;
Tracing the specified frame from the detected frame as a starting point to extract frames for each of the frame positions and selecting them as frames after downsampling.
A featured frame rate conversion method. The frame rate conversion method according to claim 1,
In the executing process, an addition value of the calculated evaluation scale and a sum of evaluation scales of the frame at the previous frame position that is the calculation target is obtained, and a minimum value among them is specified. Then, from among the plurality of frames in the predetermined range of the previous frame position, the frame having a connection that minimizes the sum of the evaluation scales from the head frame position is specified, and the addition value is determined from the identification source. To set as an evaluation scale of the frame,
A featured frame rate conversion method. The frame rate conversion method according to claim 1 or 2,
In the process of executing, using the weighted sum of the coding efficiency value indicating a higher evaluation as the value becomes smaller and the subjective image quality value indicating a higher evaluation as the value becomes smaller as the evaluation scale,
A featured frame rate conversion method. In the frame rate conversion method according to claim 3,
In the process of executing, calculating the motion compensated prediction error power as the encoding efficiency value,
A featured frame rate conversion method. In the frame rate conversion method according to claim 3,
In the process of executing, calculating the encoding cost obtained when encoding using a predetermined encoder as the encoding efficiency value,
A featured frame rate conversion method. In the frame rate conversion method according to claim 3,
In the executing step, as the subjective image quality value, calculating a difference value between a frame sampled at an equal length and a frame sampled at a variable length,
A featured frame rate conversion method. In the frame rate conversion method according to claim 3,
In the executing step, as the subjective image quality value, calculating a value obtained by weighting a difference value between a frame sampled at an equal length and a frame sampled at a variable length by a distance between both frames,
A featured frame rate conversion method. In the frame rate conversion method according to claim 3,
In the process of executing, as the subjective image quality value, calculating a value obtained by weighting a difference value between a reference frame and a predicted frame by a distance between both frames,
A featured frame rate conversion method. A frame rate conversion device that converts a frame of a high frame rate video signal into a low frame rate video signal by selecting by downsampling,
When downsampling each of a plurality of frames within a predetermined range of frame positions defined by equal-length interval downsampling with each of a plurality of frames within a predetermined range of frame positions before that Is an evaluation scale indicating the coding efficiency and subjective image quality for the low frame rate video signal generated in the above, and calculating an evaluation scale indicating a higher evaluation as the value decreases, and based on the calculated evaluation scale, From the first frame position to the last frame position, a frame having a connection that minimizes the sum of the evaluation scales from the first frame position is identified from a plurality of frames within the predetermined range of the previous frame position. Means to repeatedly execute for
Means for detecting a frame having a minimum value among the sums of evaluation scales calculated for each of the plurality of frames at the final frame position;
Means for extracting the frame for each of the frame positions by tracing the identified frame from the detected frame as a starting point, and selecting them as frames after downsampling;
A frame rate conversion device.   A frame rate conversion program for causing a computer to execute the frame rate conversion method according to any one of claims 1 to 8.

Images