JP6538619B2 - Video filtering method, video filtering apparatus and video filtering program - Google Patents

Description

  The present invention relates to a video filtering method, a video filtering apparatus and a video filtering program.

  In response to recent advances in semiconductor technology, the image acquisition speed of image pickup devices has been greatly improved. At present, applications of high frame rate video acquired by a high speed camera are classified into high image quality at the time of video reproduction and high accuracy of video analysis. The former aims to approach the upper limit of the frame rate that can be detected (displayed on display) by the visual system, and aims to express smooth motion by increasing the frame rate. For this reason, real-time reproduction on the display is assumed.

  The latter is intended to improve the accuracy of video analysis by using a high frame rate video beyond the detection limit of vision. Analysis of high-speed moving objects (sports video, FA, inspection, cars, etc.) by slow reproduction is a typical application example. This is because the upper limit of the frame rate of the video input / output system is asymmetric. At present, high-speed cameras capable of acquiring high frame rate video exceeding 10000 [frame / sec (fps)] are available as imaging systems.

  On the other hand, the upper limit of the current display is 120-240 [frame / sec]. For this reason, an image captured by such a high speed camera is used in slow playback applications.

  As a prior art, there is known a technology for converting a moving image captured at 30 frames per second or 60 fields per second into a moving image at 48 frames per second without causing the motion to be unnatural. This is to generate an image of 48 frames per second without causing the motion to be unnatural by repeating a part of frames of a progressive image of 30 frames per second twice. Alternatively, an interlaced image of 60 fields per second is converted to a progressive image of 60 frames per second, and a part of the frames is omitted to generate an image of 48 frames per second without causing the motion to be unnatural (for example, , Patent Document 1).

JP, 2004-201165, A

  By the way, there is a possibility that a video signal for real-time reproduction with high affinity to video coding can be generated by using a high frame rate video that exceeds the detection limit of vision. Such high frame rate video includes a frame group sampled at high density in the time direction, and information in the time axis direction of the imaging target is acquired with high time resolution. Therefore, if an image (for example, 30 Hz) for real-time reproduction is generated using a high-density time-sampled frame group (for example, 1000 Hz), the generation of the image can be controlled with high time resolution.

  However, the conventional method related to the pre-processing of video coding for the purpose of reducing the amount of generated code has been premised on acquiring video at a reproduction frame rate. For this reason, sampling frames with a temporal resolution higher than the playback frame rate has not been addressed.

  In simple frame decimation processing, image quality degradation caused by aliasing in the time direction becomes a problem. In order to avoid such problems, band limitation filtering in the time axis direction is required. On the other hand, in the case of an encoder using motion compensated inter-frame prediction, the reduction of aliasing in the time direction is not directly linked to the reduction of prediction error.

  At the same time, the frame sampled at high density was not fully utilized, and the degree of freedom as a time filter was restricted. In the case of a conventional 30 fps, 60 fps frame rate video, it is difficult to approximate the filter characteristics with high accuracy because sufficient samples (i.e., frames) for filtering can not be secured. For example, in the case where a video signal of 60 fps is filtered to generate a video signal of 30 fps, the frames to be filtered are limited to two frames under the condition that duplication of the frames to be filtered is not permitted. On the other hand, in the case of high frame rate video, the degree of freedom in filter design is extended.

  For example, in the case where a video signal of 1000 fps is filtered to generate a video signal of 62.5 fps, 16 frames can be subjected to the filtering even under conditions that do not allow duplication of the frames to be filtered. That is, when high frame rate video is input and low frame rate video is obtained by filtering, the degree of freedom in filtering design is increased. Therefore, there is a possibility that improvement in coding efficiency can be realized by using the high degree of freedom, and there is room for optimization of the time direction filter from the viewpoint of coding efficiency.

  The present invention has been made in view of such circumstances, and provides a video filtering method, a video filtering apparatus, and a video filtering program that can reduce the amount of computation while maintaining the suitability of temporal filtering. The purpose is

  One embodiment of the present invention is a video filtering method performed by a video filtering apparatus that performs frame rate downsampling processing for generating a low frame rate video signal with respect to an input video signal, which is low after filtering A code amount calculating step of calculating a generated code amount of a frame to be coded in a frame rate video signal; a frame to be coded; and a frame in the video signal before filtering corresponding to a time sampling position of the frame to be coded Candidates of filter coefficients for minimizing the weighted sum of the deviations from the set of filter coefficients based on the deviations calculation step of calculating the weighted sum of the deviations and the weighted sum of the generated code amount and the deviations Of the candidates of the filter coefficients, and accumulation of the weighted sum of the deviations for all frames. Which is the image filtering method and a filter coefficient selection step of selecting a filter coefficient to minimize.

  One embodiment of the present invention is the video filtering method, wherein, in the divergence calculation step, when obtaining a weighted sum of the divergence, a coding target frame and a prediction frame in a low frame rate video signal after filtering are obtained. A weighting factor to be multiplied by the degree of divergence is determined based on the difference value and the weighted sum of difference values between the encoding target frame and the frame in the video signal before filtering corresponding to the time sampling position of the encoding target frame.

One aspect of the present invention is the video filtering method, wherein the deviation degree is [[w _i , p _i ], w _i is a filter coefficient of the ith frame, p _i is a correction parameter of filter position, M is time The parameter that determines the frame rate of the image generated by the filter, δ _t is the frame interval, x is the pixel position, f (), ^ f () (^ is attached to f) is a function to obtain the pixel value At this time, the degree of deviation is calculated by equation (6) described later.

  One embodiment of the present invention is a video filtering apparatus that performs frame rate downsampling processing for generating a low frame rate video signal with respect to an input video signal, and in the low frame rate video signal after filtering. Weighted sum of the degree of deviation between a code amount calculation unit for calculating the generated code amount of the encoding target frame, the encoding target frame, and a frame in the video signal before filtering corresponding to the time sampling position of the encoding target frame From the set of filter coefficients, a candidate for a filter coefficient for minimizing the weighted sum of the divergence is selected from the set of filter coefficients based on the divergence calculation unit that calculates the and the generated code amount and the weighted sum of the divergence, From among the filter coefficient candidates, select a filter coefficient that minimizes the accumulated value of the weighted sum of the deviations for all frames. An image filtering apparatus and a data coefficient selector.

  One aspect of the present invention is a video filtering program for causing a computer to execute the video filtering method.

  According to the present invention, when performing filtering to generate a low frame rate video signal using a temporal filter in video encoding processing, it is possible to reduce the amount of computation while maintaining the suitability for temporal filtering. Effect is obtained.

FIG. 1 is a block diagram showing a configuration of a video filtering device according to an embodiment of the present invention. It is a flowchart which shows operation | movement of the imaging | video filtering apparatus shown in FIG. It is a flowchart which shows the detailed operation | movement of step S3 shown in FIG. It is a flowchart which shows the detailed operation | movement of step S20 shown in FIG. It is a flowchart which shows the detailed operation | movement of step S21 shown in FIG.

  Hereinafter, a video filtering apparatus according to an embodiment of the present invention will be described with reference to the drawings. First, on the premise that a video can be acquired with high temporal resolution, time filter setting processing for generating a video suitable for encoding using the video will be described. In the following, each frame is represented as a one-dimensional signal for simplification of the notation.

ｉはダウサンプリング後のフレームを指定するインデックスであり、非負の整数値をとる。時間フィルタの入力信号のフレーム間隔をｔとして、各フレームは、ｔ＝ｊδ_ｔ（ｊ＝０，１，・・・） においてサンプリングされる。 The ith frame generated by the (2Δ + 1) tap time filter is expressed by the following equation.

i is an index specifying a frame after Dow sampling, and takes a non-negative integer value. Each frame is sampled at t = jδ _t (j = 0, 1,...), Where _t is the frame interval of the input signal of the time filter.

f (x, t) (x = 0, ..., X-1) is a pixel value at the position x of the t-th frame. w _i [j] is the filter coefficient for the reference frame, and satisfies the relationship of the following equation.

Further, w _i is a vector w _i having a filter coefficient as an element w _i = (w _i [−Δ],..., W _i [Δ]) and is called a coefficient vector. p _i and p _i-1 are parameters for correcting the filter position. M is a parameter for determining the frame rate of the video generated by the time filter. In the case of Equation (1), the frame rate of the video output from the time filter is 1 / Mδt. In the present embodiment, it is assumed that 2Δ + 1 ≦ M. As a special form of equation (1), a filter in which the filter coefficient is a fixed value w [i] = 1 / (2Δ + 1) is referred to as an average filter.

係数ベクトルとして選択する候補ベクトル（以後、係数候補ベクトルと呼ぶ）として、Ｎ種類の係数候補ベクトルγ_ｎ＝（γ_ｎ［−Δ］，・・・，γ_ｎ［Δ］），（ｎ＝０，・・・，Ｎ−１）とする。さらに、これらのフィルタ係数を施す位置として、Ｐ通りの位置を選択可能とする。各フレームでは、これらＮ×Ｐ通りのＮ種類の係数候補ベクトルの中から、符号化フレームを生成するために最適な係数候補ベクトルを選択する。以下では、係数候補ベクトルの集合を辞書と呼び、表記を簡略する為に、Ｎ種類の係数候補ベクトルからなる辞書をΓ_Ｎ＝（γ_０，・・・，γ_Ｎ−１）として表すこととする。 The i-th frame output by the averaging filter is expressed by the following equation.

As candidate vectors to be selected as coefficient vectors (hereinafter referred to as coefficient candidate vectors), N types of coefficient candidate vectors γ _n = (γ _n [−Δ],..., Γ _n [Δ]), (n = 0 , ..., N-1). Further, P positions can be selected as positions to which these filter coefficients are to be applied. In each frame, an optimal coefficient candidate vector for generating a coded frame is selected from the N × P N types of coefficient candidate vectors. In the following, a set of coefficient candidate vectors is referred to as a dictionary, and a dictionary consisting of N types of coefficient candidate vectors is represented as Γ _N = (γ ₀ ,..., Γ _N-1 ) in order to simplify the notation. Do.

[Optimization criteria for filter coefficients]
As an optimization criterion in the time filter design, the generated code amount for the frame generated by the time filter is used. The same code amount is assumed to be obtained by the lossless encoder with motion compensation prediction (MC prediction). A case will be considered in which a frame composed of X pixels is divided into K, and motion compensated interframe prediction is performed for each divided section.

A frame ^ f (x, _i M δ _t , w _i ) (^ is placed on the following character, and so on) is divided into sections B [k] (k = 0, 1, ..., K of size X / K) Divided into -1), and frame f f (x, (i-1) M δ _t , w _i-1 ) is a reference frame, and each section B [k] (k = 0, 1, ..., K- When motion compensation (displacement amount d _i = (d _i [0],..., D _i [K−1]) is performed for 1), motion compensated inter-frame prediction error in that frame (hereinafter referred to as The prediction error (abbreviated simply) can be expressed as follows.

ここで、Ｒ_ｅ（ｅ_ｉ（０，ｗ_ｉ，ｗ_ｉ−１，ｐ_ｉ，ｐ_ｉ−１），・・・，ｅ_ｉ（Ｘ−１，ｗ_ｉ，ｗ_ｉ−１，ｐ_ｉ，ｐ_ｉ−１））は動き補償フレーム間予測誤差に対する符号量、Ｒ_ｄ（ｄ_ｉ［０］，・・・， ｄ_ｉ［Ｋ − １］）は推定変位量ｄｉ［０］，・・・，ｄｉ［Ｋ−１］に対する符号量、Ｒｈは符号化器が生成するヘッダー情報の符号量である。前述の通り、可逆符号化器の発生符号量を用いるため、動き補償フレーム間予測誤差は符号化対象フレームおよび参照フレームのみに依存する。従って、式（５）における発生符号量Ψ（）は、変位量およびヘッダ情報が定まれば、第ｉフレームに対するフィルタ係数ベクトルｗ_ｉフィルタ位置の補正パラメータｐ_ｉおよび第ｉ−１フレームに対するフィルタ係数ベクトルｗ_ｉ−１、フィルタ位置の補正パラメータｐ_ｉ−１により定まる。 The amount of generated code of the encoder that encodes this motion compensated inter-frame prediction error is expressed as follows.

Here, R _e (e _i (0, w _i , w _i−1 , p _i , p _i−1 ),..., E _i (X−1, w _i , w _i−1 , p _i , p _{i -1} )) is a code amount for motion compensation inter-frame prediction error, R _d (d _i [0], ..., d _i [K-1]) is an estimated displacement amount di [0], ... , Di [K−1], and Rh is the code amount of header information generated by the encoder. As described above, in order to use the generated code amount of the lossless encoder, the motion compensated inter-frame prediction error depends only on the encoding target frame and the reference frame. Therefore, if the displacement amount and header information are determined, the generated code amount 係数 () in equation (5) is the filter coefficient vector w _i for the i th frame, the correction parameter p _i of the filter position and the filter coefficient for the i th −1 frame It is determined by the vector w _i-1 and the correction parameter p _i-1 of the filter position.

Next, the value of the following equation is evaluated as the degree of deviation between the image generated by the filtering and the sampled image.

上記の尺度を最小化するようにフィルタを設計する。Ψ［ｗ_ｉ，ｗ_ｉ−１，ｐ_ｉ，ｐ_ｉ−１］を考慮することで、フィルタリング後の発生符号量を低減する効果を期待できる。また、Φ［ｗ_ｉ，ｐ_ｉ］の項を考慮することで、フィルタリング後の画像がサンプリングされた現画像と乖離することを抑止する効果が期待できる。 As a weighted sum of the above two types of values, the value of the following expression is used as an evaluation measure of the filter design.

Design the filter to minimize the above scale. By considering Ψ [w _i , w _i-1 , p _i , p _i-1 ], it is possible to expect an effect of reducing the generated code amount after filtering. Further, by considering the term Φ [w _i , p _i ], it is possible to expect an effect of suppressing the separation of the image after filtering from the current image sampled.

から、λを以下として定める。

(7) is set as follows.

From this, we define λ as

および

である。ここで、αは近似を補正する為の制御係数であり、外部から与えられる。 At this time,

and

It is. Here, α is a control coefficient for correcting the approximation, and is given from the outside.

Substituting the above two equations into equation (9) yields:

Ｎ×Ｐ種類のフィルタ係数ベクトルを候補ベクトルとする場合、フィルタ係数ベクトルの取り得る組合せはＮＰ^Ｊ／Ｍ通りとなり、最適なフィルタ係数ベクトル選択は、指数オーダの計算量が必要になる。このため、最適な組み合わせ（ｗ^＊ _０，・・・，ｗ^＊ _{Ｊ／Ｍ−１}，ｐ^＊ _０，・・・，ｐ^＊ _{Ｊ／Ｍ−１}）を総当りで探索するのは、計算量の観点から現実的ではない。 [Filter coefficient optimization]
In order to generate a frame that minimizes the amount of generated code, it is necessary to solve the minimization problem using the generated code amount of equation (5) as a cost function and to obtain J / M sets of filter coefficient vectors that satisfy the following equation .

When N × P types of filter coefficient vectors are used as candidate vectors, possible combinations of filter coefficient vectors are NP ^{J / M} , and optimal filter coefficient vector selection requires computational complexity of exponential order. Therefore, searching for the optimum combination (w ^* ₀ , ..., w ^* _{J / M-1} , p ^* ₀ , ..., p ^* _{J / M-1} ) in a round-robin manner is a computational effort. It is not realistic from the point of view of

Focusing on that 単 純 [w _i , w _i−1 , p _i , p _i−1 ] depends only on w _i , p _i and w _i−1 , p _i−1 , equation (13) is simple It can be formulated as an optimization problem in the Markov process. The optimization problem is based on dynamic programming, and it is possible to find the optimal solution with polynomial order complexity. The following shows the solution method using dynamic programming. First, S _i (w _i , p _i ) of the following equation is defined for w _i and p _i (i = 1,..., J / M−1).

S i _(w i, _{p i)} in the i-th stage, the cost of the total in the case of using the optimum filter coefficient vector in the path leading to the state that generated the frame by p _i shifted filter coefficient w _i was. Here, when w _i and p _i are fixed, focusing attention on the fact that Ψ [w _i , w _i−1 , p _i , p _i−1 ] depends only on w _i−1 and p _i−1 ,
S _i (w _i , p _i ) can be expressed as a recurrence equation as follows.

Note that S _i-1 (w _{i -1} , p _{i -1} ) has already been calculated using the same recurrence equation, and can be referred to as a referenceable value when calculating S _i (w _i , p _i ) It has been stored in. In this case, from a recurrence relationship of equation _{(15), S i (w} i, p i) For the calculation _{of, Ψ [w i, w i} -1, p i, p i-1] + S i-1 ( It is sufficient to select the candidate vector of the dictionary Γ _N and the shift amount p _i which minimize w _{i -1} and p _{i -1} ). When the index _{n i} of the candidate vectors for w _i, and stores for each _{n i,} wherein the index of the candidate vector giving the minimum value of _{(15) ^ n i-1} (n i, p i) as similar The shift amount is stored as ^ p _i-1 (n _i , p _i ), and can be referred to in the subsequent processing.

このように、式（１５）の漸化式を用いる方法であれば、式（１３）の最適解（ｗ^＊ _０，・・・，ｗ^＊ _{Ｊ／Ｍ−１}，ｐ^＊ _０，・・・，ｐ^＊ _{Ｊ／Ｍ−１}）は、（ＮＰ）^２Ｊ／Ｍ通りの中から最適解を探索する問題に帰着でき、多項式オーダの計算量で算出することが可能である。 By recursively using the recurrence equation of equation (15), the minimization problem of equation (13) can be expressed as the following equation.

Thus, if it is a method using the recurrence formula of Formula (15), the optimal solution (w ^* ₀ , ..., w ^* _{J / M-1} , p ^* ₀ , ...) of Formula (13) , P ^* _{J / M-1} ) can be reduced to the problem of searching for an optimal solution among the (NP) ² J / M ways, and can be calculated with the amount of computation in polynomial order.

_{^{Σ J / M-1 i =}} 1 Ψ [w i, w i-1, p i, p i-1] After obtaining the minimum value of, optimal solution ^{_{(w * 0, ···, w}} * J / _M-1 , p ^* ₀ , ..., p ^* _{J / M-1} ) are obtained by the following backtracking process. Following equation _{w J / M-1, p} J / M-1 that minimizes the equation ^(17), put the ^{_{w * J / M-1,}} p * J / M-1.

The index of the candidate vector representing w ^* _{J / M-1} is nJ _{/ M-1} . The index of the candidate vector optimum for the J / M-2 frame when the index of the candidate vector of the J / M-1 frame is n _{J / M-1} and the shift amount is p _{J / M-1} , and the index The shift amount for the frame is ^ n _{J / M-2} (n _{J / M-1} , p _{J / M-1} ), ^ p _{J / M-2} (n _{J / M-1} , p _{J / M-1} ) As stored.

Therefore, the filter coefficient vector of the J / M-2 frame is represented by w ^* _{J / M-2} = γ ^ n _{J / M-2} (nJ _{/ M-1} , pJ _{/ M-1} ), p ^* _{J / M-2} = ^ pJ _{/ M-2} (nJ _{/ M-1} , pJ _{/ M-1} ). Hereinafter, the same reference processing is performed as w ^* _{J / M-3} = γ ^ n _{J / M-3} (nJ _{/ M-2} , pJ _{/ M-2} ), p ^* _{J / M-3} = ^ p _{J / M-3} (n _{J / M-2} , p _{J / M-2} ), ..., w ^* ₀ = γ ^ n ₀ (n ₁ , p ₁ ), p ^* ₀ = ^ p ₀ (n ₁ , P ₁ ) and repeat.

Filter Dictionary Dictionary Update
The filter coefficient dictionary is updated until convergence conditions are satisfied by the following iterative processing, and candidate vectors in the dictionary are constructed. According to the [filter coefficient optimization] algorithm, an optimal solution of filter coefficients is determined using a given dictionary. A histogram h [n] (n = 0,..., N-1) of candidate vectors in the dictionary with respect to the determined optimal solution W ^* _i (i = 0,..., J / M-1) Calculate). Here, h [n] (n = 0, ..., N-1) stores the frequency with which the candidate vector γ _n in the dictionary is selected as the optimum solution.

Based on the frequencies of candidate vectors in the above dictionary, the dictionary is updated according to the following policy. If a low frequency candidate vector selected as the optimal solution exists in the dictionary, it is deleted from the dictionary. Specifically, candidate vectors satisfying h [n] ≦ θ ₁ are deleted from the dictionary. Here, θ ₁ is a threshold given from the outside.

If a high frequency candidate vector selected as the optimal solution exists in the dictionary, a filter coefficient vector obtained by modifying the candidate vector is added to the dictionary. Specifically, for a candidate vector that satisfies h [n] ≧ θ _u , θ _u is a threshold given from the outside. The correction of candidate vectors is as follows.

[Method 1 for correcting candidate vectors]
The maximum value in the element γ n [j] (j = −Δ,..., Δ−1) of the candidate vector γ n is identified.

To correct the γ _n [^ j] to αγ _n [^ j]. Here, it is set as a parameter of α <1. Furthermore, in the case of j j =-DELTA +1, ..., DELTA-2, γ n [j j +1] is corrected to n n [j j +1] + / 2/2 n n [j j], and γ n [j j -1] is γ n [ It is corrected to ^ j -1] + α / 2 γ n [^ j]. In the case of ^ j = Δ -1, it is corrected to γ n [^ j -1] + α γ n [^ j]. In the case of ^ j =-Δ, it is corrected to γ n [^ j + 1] + α γ n [^ j].

  The candidate vectors in the dictionary are updated by performing deletion, correction, and addition on the above-mentioned candidate vectors. Using the updated dictionary, the optimum filter coefficient and the generated code amount when using the same filter coefficient are determined based on the [filter coefficient optimization] algorithm.

  Before and after the update of the dictionary, the update is ended when the difference between the generated code amounts when the optimum filter coefficient is used becomes sufficiently small (less than a given threshold). If the number of candidate vectors in the dictionary exceeds a predetermined upper limit, the addition of candidate vectors is discontinued, and the updating process is ended.

Next, the operation of the frame rate downsampling process will be described. The leftmost numeral in the following description is a step number for identifying the process step. "For" means performing iterative processing while satisfying the following conditions.
1. 1. Read the captured video signal, its frame number J, and its frame rate Read downsampling ratio M3. for i = 1, ..., J / M-1
4. for W _i = Ψ ₀ , ..., Ψ _N-1
5. for p _i = 0, ..., P-1
6. for W _i-1 = Ψ ₀ , ..., Ψ _N-1
7. for p _i-1 = 0, ..., P-1
8. If the reference frame is f (x-d _i [k], (i-1) M δt, W _i-1 , p _i-1 ) f (x, i M δt, Wi, pi) ( Find a motion vector that minimizes motion compensated prediction errors for subsequent characters, and so on. The way of obtaining the motion vector is given from the outside. For example, a full search method can be applied which categorizes candidates within the search range. A motion compensation prediction error when using the same motion vector is stored in σ ² _i [W _i , W _i−1 , p _i , p _i−1 ].
9. The value of σ ² _i [W _i , W _i-1 , p _i , p _i-1 ] + S _i-1 (W _i-1 , p _i-1 ) is calculated. (If i = 1, the value of σ ² _i [W ₁ , W ₀ , p ₁ , p ₀ ] is calculated.)
10. σ ² _i [W _i , W _i-1 , p _i , p _i-1 ] + S _i-1 (W _i-1 , p _i-1 ) (W _i-1 = Ψ ₀ ,..., Ψ _N The minimum value among _-1 , p _i-1 = 0, ..., P-1) is stored in S _i (W _i , p _i ). (If i = 1, then σ ² ₁ [W ₁ , W ₀ , p ₁ , p ₀ ] (W ₀ = _{0 0} ,..., Ψ _N−1 , p ₀ = 0..., P− The minimum value in ₁ ) is stored in S ₁ (W ₁ , p ₁ ).)
11. S _i (W _i , p _i ) is given W _i-1 and p _i-1 are stored in ^ W _i-1 (W _i , p _i ) and ^ p _i-1 (W _i , p _i ).
12. _{S J / M-1 (W} J / M-1, p J / M-1) W J / M-1 to _{minimize, p J / M-1} to ^{_{W * J / M-1,}} p * J _{/ M-1} is stored.
13. for i = J / M-2, ..., 1
14. W ^* _i-1 = ^ _Wi-1 ( _Wi , _pi ) is read.
15. Read p ^* _i-1 = ^ _pi-1 ( _Wi , _pi ).
16. As the i-th frame after the down-sampling, storing ^{_{¯f (x, iMδ t, W}} * i, p i) (x = 0, ···, X-1) a.

  FIG. 1 is a block diagram showing the configuration of the video filtering device of the embodiment. In the figure, reference numeral 1 denotes an original image storage unit which stores an original image. Reference numeral 2 is a dictionary storage unit that stores a dictionary. The code | symbol 3 is a coding image generation part which produces | generates a coding image. The code | symbol 4 is a produced | generated image storage part which memorize | stores the produced | generated image. Reference numeral 5 is a reference image selection unit that selects a reference image. The code | symbol 6 is an encoding process part which encodes an image. Reference numeral 7 denotes a subjective distortion amount calculation unit that calculates the subjective distortion amount.

  Reference numeral 8 is a weighting factor calculation unit that calculates weighting factors. The code | symbol 9 is an optimal cumulative cost calculation part which calculates an optimal cumulative cost. Reference numeral 10 is a cumulative cost storage unit that stores the calculated cumulative cost. Reference numeral 11 denotes a filter coefficient index storage unit that stores a filter coefficient index. Reference numeral 12 denotes a final level determination unit that performs determination of the final level. Reference numeral 13 denotes a filter coefficient updating unit that updates filter coefficients. Reference numeral 14 denotes a filter coefficient update determining unit that determines whether to update the filter coefficient. The code | symbol 15 is a convergence determination part which determines convergence. The code | symbol 16 is a filter process part which performs a filter process.

  Next, the operation of the video filtering apparatus shown in FIG. 1 will be described. FIG. 2 is a flowchart showing the operation of the video filtering device shown in FIG. First, the video filtering apparatus externally reads video data to be filtered and a parameter M for determining a frame rate after filtering (step S1).

  Next, the filter coefficient update unit 12 reads a dictionary of filter coefficients used for filtering (step S2). Subsequently, the video filtering apparatus reads video data to be filtered, a parameter M for determining a frame rate after filtering, and a dictionary of filter coefficients as input, and calculates an optimal filter coefficient for each coding frame (step S3). ). The specific process of step S3 will be described later.

  Next, the filter coefficient updating unit 12 reads as input the video data to be filtered, the parameter M for determining the frame rate after filtering, and the filter coefficient dictionary, and updates the candidate vectors in the dictionary (step S4).

  Next, the video filtering device determines whether there is no updated filter coefficient (step S5). As a result of the process of step S4, when there is no updated candidate vector in the dictionary, the process proceeds to the process of step S7, and in other cases, the process proceeds to the process of step S6.

となる場合、ステップＳ７の処理に移り、それ以外の場合、ステップＳ８の処理に移る。 Next, the filter coefficient update determination unit 13 determines whether the reduction rate of the generated code amount using the dictionary before and after the update is less than a threshold (step S6).

When it becomes, it moves to the process of step S7, and in other cases, it moves to the process of step S8.

  Next, the filter processing unit 15 reads video data to be filtered, a parameter M for determining a frame rate after filtering, and a dictionary of filter coefficients as an input, and designates with parameter M using filter coefficients in the dictionary. The video data is filtered in the time direction so that the frame rate is reached, and the filtered video data is output (step S7).

  On the other hand, when the reduction rate of the generated code amount is not less than the threshold, the dictionary storage unit 2 reads the candidate vector including the updated filter coefficient, adds it to the dictionary, and outputs the updated dictionary (step S8), step S2 Return to and repeat the process.

  Next, the detailed operation of step S3 shown in FIG. 2 will be described with reference to FIG. FIG. 3 is a flowchart showing the detailed operation of step S3 shown in FIG. First, the video filtering apparatus reads a dictionary of video data to be filtered, a parameter M for determining a frame rate after filtering, and a filter coefficient (step S11).

Next, the video filtering apparatus repeats the processing of steps S12 to S28 for the index i = 0,..., J / M-1 indicating the encoding target frame (step S12). Subsequently, the video filtering device repeats the process of steps S13 to S27 with respect to the filter coefficient w _i ∈Γ _N of the encoding target frame (step S13).

Next, the video filtering apparatus reads input video data, performs temporal filtering processing using the filter coefficient w _i , and generates a frame to be encoded (step S14). Then, the video filtering device repeats the processing of steps S15 to S26 with respect to the filter coefficient w _i-1 ∈Γ _N of the reference frame (step S15).

Next, the accumulated cost S _i (bswi _-1 ) in the i-th-1 stage when the filter coefficient wi _-1 is used is read (step S16). Subsequently, the reference image selection 5 reads the encoding target frame at the (i-1) -th stage generated using the filter coefficient w _i-1 as a reference frame at the i-th stage (step S17).

  Next, the video filtering apparatus reads an original frame f (x, (iM + k) δt) (k = 0,..., M−1) of the ith stage (step S18).

Next, the encoding processing unit 6 reads the encoding target frame generated in step S14 and the reference frame read in step S17 as inputs, performs encoding processing, and generates the generated code amount Ψ [w _i , w _{i -1.} ] Is calculated (step S19).

  Next, the subjective distortion amount calculation unit 7 calculates the deviation amount of the encoding target frame with respect to the original frame in the i-th stage (step S20). Details will be described later. Subsequently, the weighting factor calculation unit 8 calculates a weighting factor for the amount of divergence calculated in step S20 (step S21). Details will be described later.

Next, the optimum cumulative cost calculation unit 9 sets the filter coefficient of the reference frame to the number w _i−1, and sets the filter coefficient of the encoding target frame to the number w _i as 累積 [w _i , W _i-1 ] + S _i-1 (w _i-1 ) (step S22).

Next, the optimal accumulated cost calculation unit 9 compares the calculated accumulated cost with the provisional value of S _i (w _i ), and if the former is smaller than the latter, it proceeds to the process of step S24; otherwise, it proceeds to step The process proceeds to the process of S26 (step S23).

Next, the optimal accumulated cost calculation unit 9 updates the provisional value of S _i (w _i ) with the previously calculated accumulated cost. The processes of steps S23 and S24 are processes for realizing equation (15) (step S24). Then, when the filter coefficient is w _i , the filter coefficient index storage unit 11 stores the index of the candidate vector of the filter coefficient of the reference frame giving the temporary minimum value of the accumulated cost of the i-th stage (step S25).

Next, the weighting factor calculation unit 8 calculates w ^* _{J / M-1} shown in Expression (17) as a filter factor that minimizes the accumulated cost in the J / M-1 stage (step S29).

  Next, the video filtering apparatus repeats the process of step S31 with respect to the index i = J / M-2,..., 1 indicating the encoding target frame (step S30).

Next, when the index of the candidate vector representing the w ^* _i and n _i, the index of the optimal candidate vector for the i-1 frame when the index of the candidate vectors of the i-th frame and the n _i is ^ n _i- It is stored as ₁ (n _i ). Therefore, when the filter coefficient is a candidate vector representing w ^* _i , the optimum accumulated cost calculation unit 8 calculates the filter coefficient of the reference frame that gives the minimum accumulated cost of the i-th stage (filter coefficient of the (i-1) th frame) the identified as _{^{w * i-1 = γ ^}} n i-1 (n i) ( step S31).

Next, with reference to FIG. 4, the detailed operation of step S20 shown in FIG. 3 will be described. FIG. 4 is a flowchart showing the detailed operation of step S20 shown in FIG. First, the subjective distortion amount calculation unit 7 reads out the encoding target frame f (x, iMδ _t , w _i , p _i ) of the i-th stage (step S 41). Then, the subjective distortion amount calculation unit 7 repeats the processing of steps S42 to S46 with respect to the index k = 0,..., M-1 specifying the original frame (step S42).

Next, the subjective distortion amount calculation unit 7 reads the k-th original frame f (x, (iM + k) δ _t ) in the i-th stage (step S43). Subsequently, the subjective distortion amount calculation unit 7 calculates the following square error as the deviation amount between the encoding target frame and the original frame (step S44).

Next, the subjective distortion amount calculation unit 7 calculates a cumulative sum of deviation amounts between the encoding target frame and the original frame in the stage (step S45).
S = S + sk
In the final step of the iterative process of step S42, the value of equation (6) is obtained.

Next, with reference to FIG. 5, the detailed operation of step S21 shown in FIG. 3 will be described. FIG. 5 is a flowchart showing the detailed operation of step S21 shown in FIG. First, the subjective distortion amount calculation unit 7 reads out the encoding target frame f (x, iMδ _t , w _i , p _i ) of the i-th stage (step S 51). Subsequently, the subjective distortion amount calculation unit 7 calculates a prediction frame f (x−d _i [k], (i−1) M δ _t , w _i−1 by the reference frame of the i-th stage (read by FE2-07). , P _i−1 ) (step S52).

Next, the subjective distortion amount calculation unit 7 calculates the sum of differences between the encoding target frame and the prediction frame (step S53).

Next, the subjective distortion amount calculation unit 7 repeats the processing of steps S54 to S58 with respect to the index k = 0,..., M-1 specifying the original frame (step S54). Then, the subjective distortion amount calculation unit 7 reads the k-th original frame f (x, (iM + k) δt) in the i-th stage (step S55). The subjective distortion amount calculation unit 7 calculates the following as the sum of differences between the encoding target frame and the original frame (step S56).

Next, the subjective distortion amount calculation unit 7 calculates an accumulated value of the sum of differences between the encoding target frame and the original frame in the stage (step S57).
S = S + sk
In the final step of the iterative process of step S54, the value of the following equation is obtained.

  Next, the subjective distortion amount calculation unit 7 reads the control coefficient α (step S59). Then, the subjective distortion amount calculation unit 7 reads the control coefficient Rα, the value obtained in step S57, and the value obtained in step S53, multiplies the control coefficient Rα by the value obtained in step S57, and further obtains it in step S53. The result divided by the value is output (step S60). This output is the value of equation (12).

  As described above, with respect to the high frame rate video signal sampled at high density, all the frames of the low frame rate video signal in the video encoding process using the low frame rate video signal obtained by the time filter as an input. The low frame rate video signal can be generated in consideration of the coding efficiency of According to this configuration, in time filtering involving dynamic updating of the coefficient vector dictionary, it is possible to appropriately suppress the position shift amount of the filter coefficient, and to convert the appropriate filter coefficient into an image from the viewpoint of improvement of coding efficiency. Accordingly, it becomes possible to set, and it becomes possible to maintain the subjective image quality of the image after filtering and to reduce the amount of generated code.

  All or part of the video filtering device in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer readable recording medium, and the program recorded in the recording medium may be read and executed by a computer system. Here, the “computer system” includes an OS and hardware such as peripheral devices. The term "computer-readable recording medium" refers to a storage medium such as a flexible disk, a magneto-optical disk, a ROM, a portable medium such as a ROM or a CD-ROM, or a hard disk built in a computer system. Furthermore, “computer-readable recording medium” dynamically holds a program for a short time, like a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include one that holds a program for a certain period of time, such as volatile memory in a computer system that becomes a server or a client in that case. Further, the program may be for realizing a part of the functions described above, or may be realized in combination with the program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  Although the embodiments of the present invention have been described above with reference to the drawings, it is apparent that the above embodiments are merely examples of the present invention, and the present invention is not limited to the above embodiments. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical spirit and scope of the present invention.

  The present invention can also be applied to applications where it is essential to perform filtering to generate a low frame rate video signal using a temporal filter in video coding processing.

  Reference Signs List 1: Original image storage unit 2: Dictionary storage unit 3: Coded image generation unit 4: Generated image storage unit 5. Reference image selection unit 6: Sign A processing unit 7, a subjective distortion amount calculation unit, a weighting coefficient calculation unit, an optimum cumulative cost calculation unit, a cumulative cost storage unit, and a filter coefficient index storage Unit 12, final level determination unit 13, filter coefficient update unit 14, filter coefficient update determination unit 15, convergence determination unit 16, filter processing unit

Claims (5)

A video filtering method performed by a video filtering apparatus that performs frame rate downsampling processing to generate a low frame rate video signal with respect to an input video signal.
A code amount calculating step of calculating a generated code amount of a coding target frame in the low frame rate video signal after filtering;
A divergence degree calculating step of calculating a weighted sum of the degree of divergence between the encoding target frame and the frame in the video signal before filtering corresponding to the time sampling position of the encoding target frame;
Based on the generated code amount and the weighted sum of the degrees of divergence, candidate filter coefficients for minimizing the weighted sum of the degrees of divergence are selected from the set of filter coefficients, and all candidates for the filter coefficients are selected. A filter coefficient selection step of selecting a filter coefficient that minimizes an accumulated value of the weighted sum of the degree of deviation with respect to a frame. In the deviation degree calculating step, when obtaining a weighted sum of the deviation degrees,
The difference value between the coding target frame and the prediction frame in the low frame rate video signal after filtering, and the difference value between the coding target frame and the frame in the video signal before filtering corresponding to the time sampling position of the coding target frame The video filtering method according to claim 1, wherein a weighting factor to be multiplied by the degree of divergence is determined based on a weighted sum. The deviation degree is [[w _i , p _i ], w _i is the filter coefficient of the ith frame, p _i is a correction parameter of the filter position, M is a parameter for determining the frame rate of the video generated by the time filter, δ Let _t be a frame interval, x be a pixel position, f (), ^ f () (^ be on f) be a function to obtain pixel values,

The video filtering method according to claim 1, wherein the divergence degree is calculated by A video filtering device that performs frame rate downsampling processing for generating a low frame rate video signal on an input video signal, comprising:
A code amount calculation unit that calculates a generated code amount of a coding target frame in a low frame rate video signal after filtering;
A divergence degree calculation unit that calculates a weighted sum of the degree of divergence between the encoding target frame and the frame in the video signal before filtering that corresponds to the time sampling position of the encoding target frame;
Based on the generated code amount and the weighted sum of the degrees of divergence, candidate filter coefficients for minimizing the weighted sum of the degrees of divergence are selected from the set of filter coefficients, and all candidates for the filter coefficients are selected. And a filter coefficient selection unit that selects a filter coefficient that minimizes an accumulated value of the weighted sum of the degree of deviation with respect to a frame.   A video filtering program for causing a computer to execute the video filtering method according to any one of claims 1 to 3.

Images