JP3609679B2 - Moving image processing apparatus and method - Google Patents

Description

ここで、μ_ｂ （ｘ，ｙ） 、μ_ｎ （ｘ＋ｕ，ｙ＋ｖ）はそれぞれ局所マスクＷ_ｂ （ｘ，ｙ） 、Ｗ_ｎ （ｘ＋ｕ，ｙ＋ｖ） 内の輝度の平均値、σ_ｂ （ｘ，ｙ） 、σ_ｎ （ｘ＋ｕ，ｙ＋ｖ）はそれぞれ局所マスクＷ_ｂ （ｘ，ｙ） 、Ｗ_ｎ （ｘ＋ｕ，ｙ＋ｖ） 内の輝度の分散である。
【００４５】
その他にも、これを簡略化した手法として、相互相関
【数２】

や、相互相関を局所マスクの平均値で正規化したものを用いる方法、あるいは、マスクどうしの対応している画素の輝度の差分の自乗和（ＳＳＤ：Ｓｕｍ ｏｆ Ｓｑｕａｒｅｄ Ｄｉｆｆｅｒｅｎｃｅ）や、差分の絶対値の和（ＳＡＤ：Ｓｕｍ ｏｆ Ａｂｓｏｌｕｔｅ Ｄｉｆｆｅｒｅｎｃｅ） を用いる方法、また差分の自乗や絶対値の和をマスク内の輝度の平均値及び分散値で正規化したものを用いる方法などが考えられる。
【００４６】
ただし、相関係数や相互相関を用いる方法では、輝度パターンの類似度が高いものほど相関値が大きくなるが、輝度の差分を用いる方法では輝度パターンの類似度が高いものほど相関値が小さくなる。
【００４７】
以下の説明では、輝度パターンの類似度が高いものほど相関値が大きいとする。
【００４８】
高速な動画像を用いる場合、連続するフレーム間のフローは非常に小さくなる。このことを利用して、基準フレームと後続フレームの局所マスク間の相関値を計算するとき、相関値を計算する画素を通常の場合よりも削減することができ、計算量を減らすことができる。
【００４９】
［対応判定部１２２ ］
次に対応判定部１２２ において、各フレームの各画素について前記相関値がしきい値以上、かつ最大となる画素の組を求める。この結果、基準フレームの各画素から後続のフレーム内の対応画素へのベクトルが定義され、このベクトルが画像間フローを表す。
【００５０】
つまり、（１）式で表されるような相関値のうち、基準フレームからｎフレーム後の画素（ｘ，ｙ） の最大、かつ、しきい値以上の相関値をｓ（ｕ _ｍ ，ｖ_ｍ ；ｘ，ｙ，ｎ） とすると、画像間フローは（ｕ_ｍ ，ｖ_ｍ ） となる。
【００５１】
以下では、この最大の相関値をｓ（ｘ，ｙ，ｎ）と表記する。
【００５２】
（ｂ） フロー検証部２
フロー検証部２は、フロー蓄積部２０、信頼性計算部２１からなる（図５参照）。
【００５３】
（ｂ−１） フロー蓄積部２０
フロー蓄積部２０では、後に続く信頼性計算部２１や速度決定部３で必要となる、各画素の基準フレームから現在のフレームまでの画像間フローに関するデータを蓄積する。
【００５４】
例えば、各画素（ｘ，ｙ） について、基準フレームとそれに続くｎ番目のフレームとの間で計算された画像間フローｖ（ｘ，ｙ，ｎ）とその相関値ｓ（ｘ，ｙ，ｎ）を蓄積することが考えられる。
【００５５】
実際には、これらすべてを蓄積しておくのは膨大な記憶領域を必要とするため、信頼性計算部２１や速度決定部３での処理内容に必要な情報を、画像間フローｖ（ｘ，ｙ，ｎ）とその相関値ｓ（ｘ，ｙ，ｎ）から得て蓄積する。蓄積する必要のあるデータの種類は、後続の処理に依存するため、データの詳細については後述（速度決定部３参照）する。
【００５６】
蓄積されたデータは、基準フレームを更新するとき同時にクリアし、新たに蓄積を開始する。
【００５７】
（ｂ−２） 信頼性計算部２１
信頼性計算部２１では、画像間フローの大きさが指標距離Ｄに達するまでの画像間フローの方向によって各画素の信頼性を評価する。
【００５８】
指標距離Ｄより小さい画像間フローが観測されたフレームまでの集合をＮ_１ （ｘ，ｙ） とする。ただし、大きさ０の画像間フローは方向が定義できないため除く。
【００５９】
Ｎ_１ （ｘ，ｙ） は次式のように表される。
【００６０】
【数３】

ここで、｜ｖ ｜はベクトルｖ の大きさを表し、α_０ 、α_Ｄ は正の定数で、それぞれ画像間フローの大きさを０，Ｄとみなす範囲を表す。
【００６１】
図７は画素（ｘ，ｙ） についての模式図である。
【００６２】
画像間フローの終点が画素（ｘ，ｙ） を中心とする半径α_０ の円内（図７の領域１）にあるとき、その画像間フローは大きさ０と見なし、半径Ｄ−α_０ の円外かつ半径Ｄ＋α_０ の円内（図７の領域３）に終点があるとき、その画像間フローの大きさはＤとみなす。よって、フレーム集合Ｎ_１ は図７の領域２ の内部に、（ｘ，ｙ） の対応点が見つかったフレームの集合となる。
【００６３】
画素（ｘ，ｙ） に関するフレーム集合Ｎ_１ （ｘ，ｙ） で表されるフレームの画像間フローのうち、最も信頼性の高い（相関値の大きい）画像間フローをｖ（ｘ，ｙ，ｎ _ｒ （ｘ，ｙ））と表す（図７参照）。ここで、 ｎ_ｒ （ｘ，ｙ） は、最も信頼性の高い（相関値が大きい）画像間フローが観測されたフレームで、次式で表される。
【００６４】
【数４】

フレームｎの、画素（ｘ，ｙ） の画像間フローの方向ｄ（ｘ，ｙ，ｎ）を、画像の水平方向と画像間フローのなす角と定義すると、最も信頼性の高い（相関値が大きい）画像間フローの方向 ｄ_ｒ （ｘ，ｙ） は、次式で表される。
【００６５】
【数５】

画像間フローの大きさが指標距離Ｄとほぼ等しくなるフレームの集合は次式で表される。
【００６６】
【数６】

つまり、終点が図７の領域３の内部にある画像間フローが観測されたフレームがＮ_２ （ｘ，ｙ） の各要素となる。Ｎ_２ （ｘ，ｙ） に属するフレームｎ_２で観測された画像間フロー ｖ（ｘ，ｙ，ｎ_２ ） の信頼性は、大きさがＤ付近に達する以前で最も信頼性の高かった画像間フロー ｖ（ｘ，ｙ，ｎ_ｒ ） のなす角が小さいほど高いと考えられるので、信頼性の指標 ｒ（ｘ，ｙ，ｎ_２ ） を次式のように計算する。
【００６７】
【数７】

ただし、Ｎ_２ （ｘ，ｙ） で表されるフレーム以外ではｒ（ｘ，ｙ，ｎ）＝∞とする。
【００６８】
このように計算した信頼性の指標 ｒ（ｘ，ｙ，ｎ_２ ） があるしきい値を上回ったらその画素は信頼性がないとし、その画像間フローを用いない。
【００６９】
（ｃ） 速度決定部３
図６に速度決定部３の構成図を示す。
【００７０】
速度決定部３は、移動時間決定部３０と速度情報出力部３１からなる。以下でこれらの構成例を示す。
【００７１】
（ｃ−１） 移動時間決定部３０
移動時間決定部３０では、フロー蓄積部２０で記憶されている画像間フローに関する時系列データを用いて、信頼性計算部２１で信頼性があると判断された画素の画像間フローの大きさが、指標距離設定部１１で設定した指標距離に達するまでのフレーム数を計測する。フロー検出部１ で述べた、小領域の輝度パターンの対応付けに基づく方式では、画像間フローは画素単位の精度であるため、１秒間に１０００フレームといった高速の動画像では、画像間フローの大きさが数フレームに渡って指標距離Ｄと等しくなる。これらのフレームから適切なフレームを選択しないと、各画素の速度を精度よく求めることはできない。
【００７２】
画像間フローが指標距離Ｄと等しくなるフレームの集合Ｎ_２ （ｘ，ｙ） の中から次式を満たすフレームを、画像間フローが指標距離Ｄに達したフレームｋ（ｘ，ｙ）として選択する（図８参照）。
【００７３】
【数８】

すなわち、画素（ｘ，ｙ） について、画像間フローの大きさが指標距離Ｄにほぼ等しくなるフレーム集合Ｎ（ｘ，ｙ） のうち、画像中の小領域の輝度パターンの相関値が最も大きくなるフレームを、指標距離に達したフレームとして選択する。
【００７４】
基準フレームからフレームｋ（ｘ，ｙ）までのフレーム数が指標距離に達したフレーム数である。
【００７５】
［画像間フローに関する情報の記憶内容］
このような信頼性計算部２１と移動距離決定部３０の構成にした場合、フロー蓄積部２０で時系列の画像間フローとその相関値を全て蓄積しておく必要はなく、各画素について以下の５つの画像間フローに関する情報を記憶しておけばよい。
【００７６】
１ ．第１の情報
画像間フローの大きさが、指標距離Ｄより小さく０でない（フレーム集合Ｎ_１（ｘ，ｙ）で表される）フレームでの画像間フローの相関値のうち、最大の相関値。
【００７７】
２ ．第２の情報
第１の情報における最大の相関値をとる画像間フローの方向。
【００７８】
３ ．第３の情報
画像間フローの大きさが指標距離Ｄとほぼ等しくなる（フレーム集合Ｎ_２（ｘ，ｙ）で表される）フレームでの画像間フローの相関値のうち、最大の相関値。
【００７９】
４ ．第４の情報
第３の情報における最大の相関値をとる画像間フローの方向。
【００８０】
５ ．第５の情報
基準フレームから、第３の情報における最大の相関値をとるフレームまでのフレーム数。
【００８１】
以上で述べた手法では、移動時間はフレーム単位の解像度しか得られない。
【００８２】
そこで、相関値が最大となったフレームｋ（ｘ，ｙ）の前後のｍフレームの相関値｛ｓ（ｘ，ｙ，ｋ−ｍ） 、…、ｓ（ｘ，ｙ，ｋ）、…、ｓ（ｘ，ｙ，ｋ＋ｍ）｝ に二次曲線をフィッティングし、二次曲線が最小となるフレームｋ’（ｘ，ｙ） を求めることによって、より精度の高い移動時間を求めることも可能である（図８参照）。
【００８３】
この場合、フロー蓄積部２０で蓄積しておく情報は、前記の第３、５の情報を次のように変更すればよい。
【００８４】
・変更後の第３の情報
画像間フローの大きさが指標距離Ｄとほぼ等しくなる（フレーム数集合Ｎ_２（ｘ，ｙ）で表される）フレームでの画像間フローの相関値のうち、最大の相関値と、その前後の数フレームの相関値。
【００８５】
・変更後の第５の情報
基準フレームから、変更後の第３の情報で蓄積した相関値をとるフレームまでのフレーム数。
【００８６】
また、移動方向についても画素単位の解像度しか得られないので、指標距離に達したフレームｋ（ｘ，ｙ）において、相関値計算部１２１ で計算した各画像間フローの相関値ｓ（ｕ，ｖ；ｘ，ｙ，ｋ）のうち、例えば図９の左図の網掛け部のように最大の相関値ｓ（１，−２；ｘ，ｙ，ｋ） をとる画素とその両隣の画素の相関値ｓ（０，−２；ｘ，ｙ，ｋ）， ｓ（２，−２；ｘ，ｙ，ｋ）に二次曲線を当てはめて、図９の右図のように二次曲線が最大値をとる角度 ｄ’（ｘ，ｙ，ｋ） を移動方向として求めることによって、精度を高める事も可能である。この場合、フロー蓄積部２０で蓄積しておく情報は、前記の第４の情報を次のように変更すればよい。
【００８７】
・変更後の第４の情報
第３の情報における最大の相関値をとる画像間フローの方向と、その両隣の画素の画像間フローの方向。
【００８８】
［速度情報出力部３１］
速度情報出力部３１は、移動時間決定部３０で得られた移動時間と移動方向を入力として、速度情報を出力する。
【００８９】
例えば、指標距離Ｄを移動時間で割って、各画素の速度ベクトルの大きさを求め、各画素について速度ベクトルの大きさと方向の２つのデータを出力する方法、あるいは、各画素の移動ベクトルの要素を出力する方法、あるいは、移動時間と移動方向をそのまま出力する方法などが考えられる。
【００９０】
〈変更例１〉
本発明は、前記実施例で記載した内容に限定されるものではない。
【００９１】
フロー検出部１において、前記実施例では画像間フローの検出に画像中の小領域の輝度パターンの対応付けに基づく方法を用いたが、これの代わりに既に提案されている様々な方法を用いることができる。
【００９２】
〈変更例２〉
信頼性計算部２１において、画像間フローの大きさが指標距離Ｄより小さいフレームの信頼できる画像間フローの方向（（４）式参照）として、例えばこれらのフレームの画像間フローの角度の平均値などを用いることもできる。
【００９３】
この場合、フロー蓄積部２０で蓄積しておく情報は、画像間フローの角度の和とフレーム集合Ｎ_１ （ｘ，ｙ） の要素の個数で、信頼性計算部２１において前記フローの角度の和を前記要素の数で割ることによって、画像間フローの角度の平均値を求めることができる。この方法では、相関値のようなフロー検出部１から出力される画像間フローの計算の信頼性の情報を必要としない。
【００９４】
〈変更例３〉
（６）式の各画素の画像間フローの信頼性として、得られたフローベクトルの角度の分散を用いることも考えられる。
【００９５】
分散が大きいほど得られたフローの方向には一貫性がなく、信頼性が低いと考えられる。この場合、フロー蓄積部２０で記憶しておく情報は、画像間フローの角度Ｓｄ（ｘ，ｙ）の和と、角度の自乗和Ｓｄ（ｘ，ｙ）^２と、足し算の回数ｍで、信頼性計算部２１において次式によって、角度の分散σ^２ｄ（ｘ，ｙ）を計算する。
【００９６】
【数９】

〈変更例４〉
これまで実施例の信頼性計算部２１の説明や、変更例で述べてきたように、画像間フローの信頼性に画像間フローの角度に関する指標を用いる以外にも、画像間フローの大きさに関する指標を用いる方法も考えられる。
【００９７】
画像間フローの大きさは、画像間フローの大きさが指標距離Ｄに達するまでのフレーム数に反比例するので、例えば、指標距離Ｄ＝２のとき、（６）式の代わりに次式を用いることも考えられる。
【００９８】
【数１０】

これは、例えばＤ＝２のときには、 ｎ_ｒ （ｘ，ｙ） が大きさ１の画像間フローが観測された時のフレーム数となるので、大きさ２の画像間フローが観測されるのは、基準フレームからのフレーム数が ２× ｎ_ｒ （ｘ，ｙ） である可能性が高いということを利用している。
【００９９】
また、 ｎ_ｒ （ｘ，ｙ） の代わりにＮ_１ （ｘ，ｙ） の要素の平均値や中間値を用いることも考えられる。
【０１００】
〈変更例５〉
移動時間決定部３０では、画像間フローが指標距離Ｄに達するフレームとして、画像間フローの大きさが、指標距離Ｄにほぼ等しくなるまでのフレームの集合Ｎ_２ （ｘ，ｙ） の要素の平均値、中間値、あるいは、最小フレーム数と最大フレーム数の中央の値を用いることなどが考えられる。
【０１０１】
〈変更例６〉
速度出力部３１で速度を様々な形態で出力することを述べたが、さらにその信頼性も付加して出力することも考えられる。
【０１０２】
例えば、信頼計算部２１でしきい値処理を行わずに、各画素の画像間フローの信頼性をそのまま出力し、速度決定部３で計算される速度とともに出力することが考えられる。
【０１０３】
また、速度決定部３で計算した速度の信頼性は画像間フローの計算の信頼性にも依存するので、フロー検出部１ で計算された画像間フローの相関値を速度の信頼性としてともに出力することなども考えられる。
【０１０４】
〈変更例７〉
上記実施形態では、１秒間に１０００フレームよりなる高速画像について対象としたため、各画素の移動軌跡は、略直線運動として推定できるが、例えば、各画素の移動がそれ以上に早く、かつ、直線運動以外の規則的な運動（例えば、渦巻き状の運動）をする場合には、その動きを考慮して信頼性計算部２１で各画素の信頼性を計算すればよい。なお、高速画像でなく、これより低速で画像を取得する場合も同様にして各画素の動きを考慮すればよい。
【０１０５】
【発明の効果】
本発明によれば、例えば、高速な時系列画像（動画像）を入力し、各画素の移動量がある一定の移動量量に達するまでの移動時間と移動方向を求めることによって各画素の移動速度を求める場合において、各画素のフローベクトルの信頼性を求めて、その信頼性のある画素のみを処理することにより、移動時間と移動方向の検出精度を向上させ、各画素の移動速度を正確に求めることが可能となり、画素の移動速度を用いる画像処理の精度向上に貢献する。
【図面の簡単な説明】
【図１】本発明の一実施形態の構成を示すブロック図である。
【図２】画像間フローの概念の説明図である。
【図３】図１のフロー検出部の一実施形態を示すブロック図である。
【図４】図３のフロー計算部の一実施形態を示すブロック図である。
【図５】図１のフロー検証部の一実施形態を示すブロック図である。
【図６】図１の画像間速度決定部の一実施形態を示すブロック図である。
【図７】画像間フローの信頼性計算手法の一実施形態を示す説明図である。
【図８】移動時間を決定する手法の説明図である。
【図９】移動方向をの精度を向上する手法の説明図である。
【符号の説明】
０…画像入力部
１…フロー検出部
１０…基準フレーム設定部
１１…指標距離設定部
１２…フロー計算部
１２０…局所マスク設定部
１２１…相関値計算部
１２２…対応判定部
２…フロー検証部
２０…フロー蓄積部
２１…信頼性計算部
３０…移動時間決定部
３１…速度情報出力部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a moving image processing apparatus capable of calculating with high accuracy the moving speed of each pixel on an image from a high-speed time-series image input to a sensor capable of processing at the same time as capturing the image at high speed, The present invention relates to a method and a recording medium on which the method is recorded.
[0002]
[Prior art]
Conventionally, as a means for inputting time-series images to a computer, a CCD camera is mostly used, and data transfer from the camera to the computer is performed at a rate of 30 frames per second in accordance with the NTSC standard. It was done. When a moving vector of each pixel of an image, that is, an optical flow (hereinafter simply referred to as a flow) is obtained using such a moving image, a large movement is required for an object that moves at high speed, such as a car or a quick body movement. You need to find a vector. If the movement vector becomes large in this way, there will be a problem that the change in image brightness due to the change in the appearance of the object and the background hidden behind the object will appear, and accurate motion vector detection will occur. Becomes difficult.
[0003]
On the other hand, in the gesture input type interactive game (Image Lab March 1998), a high-speed motion detection of 1000 pieces per second is performed using a 32 pixel × 32 pixel artificial retina chip. If such a high-speed moving image is used, it is considered that the above problem can be fundamentally solved. However, the motion information obtained with this apparatus is insufficient for accuracy and spatial resolution for complex image processing.
[0004]
In addition, in the massively parallel and ultra-high-speed visual information processing system (Journal of Applied Physics, Vol. 67, No. 1 by Ishikawa et al.), A digital processing circuit is installed in the sensor, so that the rate is 1000 frames per second. Various image processing algorithms can be executed. Ishikawa et al. Uses a flow detection method conventionally used in this system to detect the flow. However, in a high-speed moving image such as 1000 frames per second, the flow vector of each pixel is very small, so the difference in the flow vector between each pixel is very small, and the conventional flow detection method provides sufficient accuracy. It may not be possible.
[0005]
As a method for stably detecting the movement vector of each pixel using a high-speed moving image with respect to these conventional image processing apparatuses, the movement amount of each pixel reaches a certain amount (hereinafter referred to as an index distance). A method has been proposed in which a moving speed and a moving direction are obtained and a moving speed is obtained by dividing an index distance by a moving time.
[0006]
In this method, the key is how to accurately determine the travel time and travel direction. In order to obtain the movement time and the movement direction, it is necessary to detect the flow between images. In this method, a conventionally proposed method is used as it is. In the conventional method, a point where the flow is not accurately obtained during the pre-processing or calculation of the flow calculation is detected, and the flow is not calculated at that point, thereby preventing erroneous detection of the flow.
[0007]
For example, A.I. D. Jeffon and D.C. J. et al. Fleet detects the singularity of the phase of the luminance signal of the time-series image and detects the pixel where the flow extraction becomes unstable in “Phase singles in scale-space”, Image and Vision Computing, 9: 338-343, 1991 doing.
[0008]
S. Yamamoto et al., “Realtime Multiple Object Tracking Based on Optical Flow”, Proc. of R & A, 2328-2333, 1995, a pixel having only an edge in a single direction in a small neighborhood in the image is obtained, and a pixel in which the extraction of the flow becomes unstable is detected.
[0009]
However, although these methods can evaluate the reliability of the flow of each frame alone, there is a problem that the reliability due to the time-series change of the flow is not considered.
[0010]
In addition, in order to accurately measure the movement time until the movement amount of each pixel reaches the index distance, the accuracy of the flow size is important, but even if the flow is accurately obtained to some extent, it takes 1 second. In high-speed moving images such as 1000 frames, the flow near the index distance is observed many times in time, and the movement time to reach the index distance is uniquely determined by the measurement error of the flow between frames and the influence of quantization. There is a problem that it becomes difficult. Furthermore, the conventional method has a problem that it does not take into account that the travel time is uniquely and appropriately determined.
[0011]
[Problems to be solved by the invention]
As described above, in the method of calculating the moving speed by determining the moving time and moving direction until the moving amount of each pixel reaches the index distance, how to accurately determine the moving time and moving direction is the key. However, the improvement in accuracy is not considered, and there remains a problem regarding the accuracy of the movement vector.
[0012]
The present invention has been made in view of the above problems, and using a high-speed time-series image input to a sensor capable of processing an image at a high speed, the movement amount of each pixel is set as an index distance. Provided are a moving image processing apparatus, a method thereof, and a recording medium for obtaining an accurate moving speed of a pixel in a process of obtaining a moving speed by obtaining a moving time and a moving direction until reaching the target distance and dividing an index distance by the moving time.
[0013]
[Means for Solving the Problems]
Claim 1 According to the present invention, there is provided an image input means for capturing an image in time series, a flow detection means for detecting an optical flow having information on movement of each pixel between images from the time series image captured by the image input means, Based on the flow verification means for verifying the reliability of the flow vector for each pixel in the optical flow detected by the flow detection means for each pixel, and the flow vector recognized as reliable by the flow verification means And determining a moving time at which the moving amount of the pixel becomes a constant moving amount, and obtaining the moving speed of the pixel based on the determined moving time, the fixed moving amount, and the direction component of the flow vector of the pixel. Speed determining means, and the flow verifying means includes a flow vector for each pixel detected by the flow detecting means. And flow accumulation means for time-sequentially storing the information, using the time-series information regarding the flow vector for each pixel stored in the flow storage unit, Obtain the time-series change of the flow vector for each pixel, and based on this change, A moving image processing apparatus comprising: a reliability calculating unit that calculates the reliability of a flow vector for each pixel.
[0015]
Claim 2 In the invention, the flow accumulating unit includes a reference image at an arbitrary time detected by the flow detecting unit and each of an image that is temporally earlier or later than the reference image. Store time-series data relating to the flow vector of a pixel as it is, or information for determining the reliability of the pixel based on the time-series data, or to determine a movement time when the movement amount of the pixel is a constant movement amount The information is processed and stored. 1 It is a moving image processing apparatus of description.
[0016]
Claim 3 The reliability calculation means uses the time-series information regarding the flow vector of each pixel accumulated in the flow accumulation means to use the time-series change in the movement direction of each pixel and the movement amount of each pixel. Or a time-series change in the amount and direction of movement of each pixel is changed based on a predetermined rule, and is detected at the pixel based on the evaluation. The reliability of the flow vector is calculated. 1 It is a moving image processing apparatus of description.
[0017]
Claim 4 According to the invention, the speed determining means uses the time series information regarding the flow vector of each pixel stored in the flow accumulating means to calculate a movement time for the movement amount of each pixel to reach the certain movement amount. And a speed information output means for obtaining and outputting the moving speed of the pixel from the determined moving time, the fixed moving amount, and the direction component of the flow vector of the pixel. Claim 1 It is a moving image processing apparatus of description.
[0018]
Claim 5 The invention includes an image input step for capturing an image in time series, a flow detection step for detecting an optical flow having information on movement of each pixel between images from the time series image captured in the image input step, Based on the flow verification step for verifying the reliability of the flow vector for each pixel in the optical flow detected in the flow detection step for each pixel, and the flow vector recognized as reliable in this flow verification step And determining a moving time at which the moving amount of the pixel becomes a constant moving amount, and obtaining the moving speed of the pixel based on the determined moving time, the fixed moving amount, and the direction component of the flow vector of the pixel. A speed determination step; The flow verification step includes: a flow accumulation step for accumulating information on a flow vector for each pixel detected in the flow detection step in time series; and a flow accumulation step for each pixel accumulated in the flow accumulation step. A time series information on the flow vector is used to obtain a time-series change of the flow vector for each pixel, and based on this change, a reliability calculation step of calculating the reliability of the flow vector for each pixel; A moving image processing method characterized by comprising:
[0019]
Claim 6 The invention includes an image input function for capturing images in time series, a flow detection function for detecting an optical flow having information on movement of each pixel between images from the time series images captured by the image input function, Based on the flow verification function for verifying the reliability of the flow vector for each pixel in the optical flow detected by the flow detection function for each pixel, and the flow vector recognized as reliable by this flow verification function And determining a moving time at which the moving amount of the pixel becomes a constant moving amount, and obtaining the moving speed of the pixel based on the determined moving time, the fixed moving amount, and the direction component of the flow vector of the pixel. Speed decision function, The flow verification function stores a flow accumulation function for accumulating information on the flow vector for each pixel detected by the flow detection function in a time series, and for each pixel accumulated by the flow accumulation function. A time series information on the flow vector is used to obtain a time-series change of the flow vector for each pixel, and based on this change, a reliability calculation function for calculating the reliability of the flow vector for each pixel; The recording medium of the moving image processing method characterized by recording the program which implement | achieves.
[0020]
According to the present invention, for example, a high-speed time-series image (moving image) is input, and the movement amount of each pixel is obtained by determining the movement time and movement direction until the movement amount of each pixel reaches a certain movement amount. In determining the speed, the reliability of the flow vector of each pixel is obtained, and only the reliable pixel is processed, thereby improving the detection accuracy of the moving time and the moving direction. Can be obtained accurately.
[0021]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0022]
FIG. 1 shows a basic configuration example of the present embodiment.
[0023]
A reference frame at an arbitrary time is set by the flow detection unit 1 from a moving image sequence obtained by the image input unit 0 which is a device capable of acquiring a high-speed image such as 1000 frames per second. A flow of each pixel (hereinafter referred to as an inter-image flow) between a later frame in time is calculated.
[0024]
In the case of FIG. 2, if a pixel (x, y) in a reference frame moves to a pixel (x + u, y + v) after n frames, the flow between the images of the pixel (x, y) is (u, v) and Become. The flow verification unit 2 verifies the reliability of the inter-image flow obtained at each pixel, and the speed determination unit 3 determines the time and moving direction until the size of the inter-image flow obtained at each pixel reaches the index distance. By obtaining, the speed information of each pixel is obtained and output.
[0025]
In FIG. 2, the number of frames (time) until the inter-image flow at the point (x, y) reaches a certain index distance D pixel from (x, y) is obtained.
[0026]
Basically, the magnitude of the flow vector of each pixel is obtained by dividing D by the number of frames, and the direction is obtained from the direction of the flow between images.
[0027]
(A) Flow detection unit 1
The flow detection unit 1 includes a reference frame setting unit 10, an index distance setting unit 11, and a flow calculation unit 12 (see FIG. 3).
[0028]
(A-1) Reference frame setting unit 10
In order to obtain the inter-image flow, first, it is necessary to set a reference frame. The reference frame setting unit 10 sets this.
[0029]
As a reference frame setting policy,
(I) A method in which the reference frame is the same for all pixels,
(Ii) Method for setting different reference frames for each pixel
There is.
[0030]
The method (i) includes a method of updating the reference frame to the current frame every time a predetermined time elapses, or a predetermined ratio of pixels in which an inter-image flow is detected. There is a method of updating the reference frame to the current frame when it is determined that the size of the inter-flow has exceeded the index distance.
[0031]
As a method (ii), when it is determined that the size of the inter-image flow exceeds the index distance for each pixel, the reference frame of the pixel is updated to the current frame.
[0032]
(A-2) Index distance setting unit 11
In this method, the speed of each pixel is obtained by obtaining the moving distance of each pixel, that is, the time until the magnitude of the flow between images reaches the index distance and the moving direction.
[0033]
The index distance setting unit 11 sets this index distance to “D”.
[0034]
Although the selection of D is arbitrary, as will be described later, since the reliability of the inter-image flow is verified using the time-series data regarding the inter-image flow of each pixel, the data necessary for this verification is obtained. It is necessary to set the index distance D so as to be obtained. In addition, it is necessary to set D in a range in which the flow can be accurately obtained by the flow detection method used in the flow calculation unit 12.
[0035]
The movement distance when a certain pixel (x, y) is moved to (x + u, y + v) is represented by (u ² + V ² ) ^1/2 In this case, the movement distance reaches D on the circumference of radius D centered at (x, y). If the moving distance is defined as max (| u |, | v |), the moving distance becomes D because the one side with (x, y) at the midpoint of the diagonal is on the outer periphery of the 2D square. .
[0036]
(A-3) Flow calculation unit 12
The flow calculation unit 12 calculates the inter-image flow between the reference frame set by the reference frame setting unit 10 and the subsequent frame.
[0037]
Various conventionally proposed methods can be used as the flow extraction method used here.
[0038]
As an example, FIG. 4 shows a detailed configuration example of a method based on association of luminance patterns of small areas in an image.
[0039]
In the case of this method, the flow calculation unit 12 includes a local mask setting unit 120, a correlation value calculation unit 121, and a correspondence determination unit 122.
[0040]
[Local mask setting unit 120]
In the local mask setting unit 120, a local mask (size is 5 × 5, 7 × 7 pixels, etc.) including pixels for which the flow between images is desired is set in each image.
[0041]
However, a pixel having no contrast in the local mask does not have a feature and it is difficult to obtain a correct flow, and therefore, no local mask is set.
[0042]
Further, even if there is contrast in the local mask, a pixel that has only edges in a single direction does not have a local mask because a feature is incomplete to obtain a correct flow.
[0043]
[Correlation value calculation unit 121]
Next, in the correlation value calculation unit 121, the luminance distribution of the reference frame is determined as f _b (X, y), and the luminance distribution of the frame after the nth frame f _n If (x, y), the local mask W centered on each pixel (x, y) of the reference frame _b (X, y) and the same coordinate in the subsequent frame and a local mask W centered on the surrounding pixels _n As the correlation value s (u, v; x, y, n) of the luminance signal distribution between (x + u, y + v), a correlation coefficient represented by the following equation can be used.
[0044]
[Expression 1]

Where μ _b (X, y), μ _n (X + u, y + v) are local masks W, respectively. _b (X, y), W _n Average value of luminance in (x + u, y + v), σ _b (X, y), σ _n (X + u, y + v) are local masks W, respectively. _b (X, y), W _n (X + u, y + v) is the luminance dispersion.
[0045]
As another simplified method, cross-correlation
[Expression 2]

Or a method using a cross-correlation normalized by an average value of a local mask, or a sum of squares of differences in luminance of pixels corresponding to each other (SSD: Sum of Squared Difference) or an absolute value of the difference A method using a sum of absolute differences (SAD), a method using a square of a difference or a sum of absolute values normalized by an average value and a variance value of luminance in a mask, and the like can be considered.
[0046]
However, in the method using the correlation coefficient or the cross-correlation, the correlation value increases as the luminance pattern similarity increases, but in the method using the luminance difference, the correlation value decreases as the luminance pattern similarity increases. .
[0047]
In the following description, it is assumed that the correlation value is higher as the similarity of luminance patterns is higher.
[0048]
When using a high-speed moving image, the flow between successive frames is very small. By utilizing this fact, when calculating the correlation value between the local mask of the reference frame and the subsequent frame, the pixels for calculating the correlation value can be reduced as compared with the normal case, and the calculation amount can be reduced.
[0049]
[Correspondence determination unit 122]
Next, the correspondence determination unit 122 obtains a set of pixels for which the correlation value is equal to or greater than a threshold value for each pixel in each frame. As a result, a vector from each pixel in the reference frame to the corresponding pixel in the subsequent frame is defined, and this vector represents the flow between images.
[0050]
That is, among the correlation values represented by the expression (1), the correlation value that is the maximum of the pixel (x, y) n frames after the reference frame and equal to or greater than the threshold value is represented by s (u _m , V _m X, y, n), the flow between images is (u _m , V _m )
[0051]
Hereinafter, this maximum correlation value is expressed as s (x, y, n).
[0052]
(B) Flow verification unit 2
The flow verification unit 2 includes a flow accumulation unit 20 and a reliability calculation unit 21 (see FIG. 5).
[0053]
(B-1) Flow accumulation unit 20
The flow accumulation unit 20 accumulates data relating to the inter-image flow from the reference frame of each pixel to the current frame, which is necessary for the reliability calculation unit 21 and the speed determination unit 3 that follow.
[0054]
For example, for each pixel (x, y), the inter-image flow v (x, y, n) calculated between the reference frame and the subsequent nth frame and its correlation value s (x, y, n) It is possible to accumulate
[0055]
Actually, since storing all of these requires a huge storage area, information necessary for processing contents in the reliability calculation unit 21 and the speed determination unit 3 is used as the inter-image flow v (x, y, n) and its correlation value s (x, y, n) are obtained and stored. Since the type of data that needs to be accumulated depends on the subsequent processing, details of the data will be described later (see the speed determination unit 3).
[0056]
The accumulated data is cleared at the same time when the reference frame is updated, and a new accumulation is started.
[0057]
(B-2) Reliability calculation unit 21
The reliability calculation unit 21 evaluates the reliability of each pixel based on the direction of the flow between images until the size of the flow between images reaches the index distance D.
[0058]
N is a set up to frames in which the flow between images smaller than the index distance D is observed. ₁ (X, y). However, the flow between images of size 0 is excluded because the direction cannot be defined.
[0059]
N ₁ (X, y) is expressed by the following equation.
[0060]
[Equation 3]

Where | v | represents the magnitude of the vector v and α ₀ , Α _D Is a positive constant and represents a range in which the magnitude of the flow between images is regarded as 0 and D, respectively.
[0061]
FIG. 7 is a schematic diagram of the pixel (x, y).
[0062]
Radius α with the end of the inter-image flow centered at pixel (x, y) ₀ Is within the circle (region 1 in FIG. 7), the inter-image flow is considered to have a magnitude of 0 and the radius D-α. ₀ Outside the circle and radius D + α ₀ When the end point is in the circle (region 3 in FIG. 7), the size of the flow between images is regarded as D. Therefore, the frame set N ₁ Is a set of frames in which the corresponding point of (x, y) is found in the region 2 in FIG.
[0063]
Frame set N for pixel (x, y) ₁ Among the inter-image flows of the frame represented by (x, y), the most reliable (high correlation value) inter-image flow is represented by v (x, y, n _r (X, y)) (see FIG. 7). Where n _r (X, y) is a frame in which an inter-image flow with the highest reliability (high correlation value) is observed, and is represented by the following equation.
[0064]
[Expression 4]

When the direction d (x, y, n) of the inter-image flow of the pixel (x, y) of the frame n is defined as the angle formed by the horizontal direction of the image and the inter-image flow, the most reliable (correlation value is Large) Inter-image flow direction d _r (X, y) is expressed by the following equation.
[0065]
[Equation 5]

A set of frames in which the size of the inter-image flow is approximately equal to the index distance D is expressed by the following equation.
[0066]
[Formula 6]

That is, the frame in which the flow between images whose end point is inside the region 3 in FIG. ₂ It becomes each element of (x, y). N ₂ Frame n belonging to (x, y) ₂ Flow between images v (x, y, n ₂ ) Is the most reliable inter-image flow v (x, y, n) before the size reaches around D. _r ) Is considered to be higher as the angle formed by is smaller, the reliability index r (x, y, n ₂ ) Is calculated as follows:
[0067]
[Expression 7]

However, N ₂ For other than the frame represented by (x, y), r (x, y, n) = ∞.
[0068]
Reliability index r (x, y, n) calculated in this way ₂ If the pixel exceeds a certain threshold, the pixel is not reliable, and the flow between images is not used.
[0069]
(C) Speed determination unit 3
FIG. 6 shows a configuration diagram of the speed determination unit 3.
[0070]
The speed determination unit 3 includes a travel time determination unit 30 and a speed information output unit 31. Examples of these configurations are shown below.
[0071]
(C-1) Travel time determination unit 30
The moving time determination unit 30 uses the time-series data regarding the inter-image flow stored in the flow accumulation unit 20 to determine the size of the inter-image flow of the pixels determined to be reliable by the reliability calculation unit 21. The number of frames until the index distance set by the index distance setting unit 11 is reached is measured. In the method based on the association of small area luminance patterns described in the flow detection unit 1, the flow between images has an accuracy in units of pixels. Therefore, in a high-speed moving image such as 1000 frames per second, the flow between images is large. Becomes equal to the index distance D over several frames. Unless an appropriate frame is selected from these frames, the speed of each pixel cannot be obtained accurately.
[0072]
A set N of frames in which the flow between images becomes equal to the index distance D ₂ A frame satisfying the following expression is selected from (x, y) as a frame k (x, y) in which the inter-image flow has reached the index distance D (see FIG. 8).
[0073]
[Equation 8]

That is, for the pixel (x, y), the correlation value of the luminance pattern of the small region in the image is the largest in the frame set N (x, y) in which the magnitude of the inter-image flow is approximately equal to the index distance D. The frame is selected as the frame that has reached the index distance.
[0074]
The number of frames from the reference frame to the frame k (x, y) is the number of frames that has reached the index distance.
[0075]
[Contents of information related to flow between images]
In the case of such a configuration of the reliability calculation unit 21 and the movement distance determination unit 30, it is not necessary to store all the time-series inter-image flows and their correlation values in the flow storage unit 20, and for each pixel the following Information relating to the flow between the five images may be stored.
[0076]
1. First information
The size of the flow between images is smaller than the index distance D and is not 0 (frame set N ₁ The maximum correlation value among the correlation values of the flow between images in a frame (represented by (x, y)).
[0077]
2. Second information
Direction of flow between images taking the maximum correlation value in the first information.
[0078]
3. Third information
The size of the inter-image flow is approximately equal to the index distance D (frame set N ₂ The maximum correlation value among the correlation values of the flow between images in a frame (represented by (x, y)).
[0079]
4. Fourth information
Direction of the flow between images taking the maximum correlation value in the third information.
[0080]
5. 5th information
The number of frames from the reference frame to the frame having the maximum correlation value in the third information.
[0081]
With the method described above, the moving time can be obtained only in the frame unit resolution.
[0082]
Therefore, the correlation values {s (x, y, km),..., S (x, y, k),... S of the m frames before and after the frame k (x, y) having the maximum correlation value. By fitting a quadratic curve to (x, y, k + m)} and obtaining a frame k ′ (x, y) that minimizes the quadratic curve, it is also possible to obtain a more accurate travel time ( (See FIG. 8).
[0083]
In this case, the information stored in the flow storage unit 20 may be changed as described above for the third and fifth information.
[0084]
・ Third information after change
The size of the inter-image flow is almost equal to the index distance D (frame number set N ₂ Among the correlation values of the flow between images in a frame (represented by (x, y)), the maximum correlation value and the correlation values of several frames before and after that.
[0085]
・ Fifth information after change
The number of frames from the reference frame to the frame that takes the correlation value accumulated in the changed third information.
[0086]
Further, since only the resolution in pixel units can be obtained in the moving direction, the correlation value s (u, v) of the flow between images calculated by the correlation value calculation unit 121 in the frame k (x, y) reaching the index distance. ; X, y, k), for example, the correlation between a pixel having the maximum correlation value s (1, -2; x, y, k) and its neighboring pixels as shown by the shaded portion in the left diagram of FIG. A quadratic curve is applied to the values s (0, -2; x, y, k) and s (2, -2; x, y, k), and the quadratic curve has the maximum value as shown in the right figure of FIG. It is also possible to improve the accuracy by obtaining the angle d ′ (x, y, k) taking the as the moving direction. In this case, the information stored in the flow storage unit 20 may be changed from the fourth information as follows.
[0087]
・ Fourth information after change
The direction of the flow between images which takes the maximum correlation value in the third information, and the direction of the flow between images of the pixels adjacent to both.
[0088]
[Speed information output unit 31]
The speed information output unit 31 receives the travel time and the travel direction obtained by the travel time determination unit 30 and outputs speed information.
[0089]
For example, the index distance D is divided by the moving time to obtain the magnitude of the velocity vector of each pixel, and two data of the magnitude and direction of the velocity vector are output for each pixel, or the elements of the moving vector of each pixel Or a method of outputting the movement time and the movement direction as they are.
[0090]
<Modification 1>
The present invention is not limited to the contents described in the above embodiments.
[0091]
In the flow detection unit 1, in the above-described embodiment, the method based on the association of the luminance pattern of the small area in the image is used for the detection of the flow between images. Instead, various methods that have already been proposed are used. Can do.
[0092]
<Modification 2>
In the reliability calculation unit 21, for example, an average value of the angles of the flow between images of these frames as the direction of the reliable flow between images of the frame whose size is smaller than the index distance D (see Expression (4)). Etc. can also be used.
[0093]
In this case, the information accumulated in the flow accumulation unit 20 includes the sum of the angles of the flow between images and the frame set N. ₁ An average value of the inter-image flow angles can be obtained by dividing the sum of the flow angles by the number of the elements in the reliability calculation unit 21 by the number of elements of (x, y). This method does not require information on the reliability of calculation of the flow between images output from the flow detector 1 such as a correlation value.
[0094]
<Modification 3>
As the reliability of the inter-image flow of each pixel in the equation (6), it is also conceivable to use the angle distribution of the obtained flow vector.
[0095]
The greater the variance, the less consistent the flow direction obtained and the less reliable. In this case, the information stored in the flow storage unit 20 includes the sum of the angles Sd (x, y) of the flow between images and the square sum Sd (x, y) of the angles. ² And the number of additions m, and the reliability calculation unit 21 calculates the angle variance σ by the following equation: ² d (x, y) is calculated.
[0096]
[Equation 9]

<Modification 4>
As described in the description of the reliability calculation unit 21 of the embodiment and the modified examples, the index of the flow between images is used in addition to the index regarding the angle of the flow between images in the reliability of the flow between images. A method using an index is also conceivable.
[0097]
The size of the flow between images is inversely proportional to the number of frames until the size of the flow between images reaches the index distance D. For example, when the index distance D = 2, the following equation is used instead of the equation (6). It is also possible.
[0098]
[Expression 10]

For example, when D = 2, n _r Since (x, y) is the number of frames when the flow between images of size 1 is observed, the flow between images of size 2 is observed when the number of frames from the reference frame is 2 × n. _r The fact that (x, y) is highly likely is used.
[0099]
N _r N instead of (x, y) ₁ It is conceivable to use an average value or an intermediate value of the elements of (x, y).
[0100]
<Modification 5>
In the movement time determination unit 30, as a frame in which the inter-image flow reaches the index distance D, a set N of frames until the size of the inter-image flow becomes substantially equal to the index distance D. ₂ It is conceivable to use an average value, an intermediate value of the elements of (x, y), or a central value between the minimum number of frames and the maximum number of frames.
[0101]
<Modification 6>
It has been described that the speed output unit 31 outputs the speed in various forms.
[0102]
For example, it is conceivable that the reliability calculation unit 21 outputs the reliability of the flow between images of each pixel as it is without performing threshold processing, and outputs it together with the speed calculated by the speed determination unit 3.
[0103]
Further, since the reliability of the speed calculated by the speed determination unit 3 depends on the reliability of the calculation of the flow between images, the correlation value of the flow between images calculated by the flow detection unit 1 is output together as the reliability of the speed. It is also possible to do.
[0104]
<Modification 7>
In the above embodiment, since a high-speed image consisting of 1000 frames per second is targeted, the movement trajectory of each pixel can be estimated as a substantially linear motion. When performing a regular motion other than (for example, a spiral motion), the reliability calculation unit 21 may calculate the reliability of each pixel in consideration of the motion. It should be noted that the movement of each pixel may be considered in the same manner when an image is acquired at a lower speed than the high-speed image.
[0105]
【The invention's effect】
According to the present invention, for example, a high-speed time-series image (moving image) is input, and the movement amount of each pixel is obtained by obtaining the movement time and movement direction until the movement amount of each pixel reaches a certain movement amount. In determining the speed, the reliability of the flow vector of each pixel is obtained, and only the reliable pixel is processed, thereby improving the detection accuracy of the movement time and direction, and the movement speed of each pixel is accurately determined. This contributes to improving the accuracy of image processing using the pixel moving speed.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.
FIG. 2 is an explanatory diagram of a concept of an inter-image flow.
FIG. 3 is a block diagram showing an embodiment of the flow detection unit of FIG. 1;
4 is a block diagram illustrating an embodiment of the flow calculation unit of FIG. 3;
FIG. 5 is a block diagram illustrating an embodiment of the flow verification unit of FIG. 1;
6 is a block diagram illustrating an embodiment of an inter-image speed determination unit in FIG. 1. FIG.
FIG. 7 is an explanatory diagram showing an embodiment of a reliability calculation method for an inter-image flow.
FIG. 8 is an explanatory diagram of a method for determining a travel time.
FIG. 9 is an explanatory diagram of a method for improving the accuracy of the moving direction.
[Explanation of symbols]
0 ... Image input section
1 ... Flow detector
10: Reference frame setting section
11: Index distance setting section
12 ... Flow calculation part
120: Local mask setting section
121 ... correlation value calculation unit
122 ... Correspondence determination unit
2. Flow verification unit
20 ... Flow accumulation part
21 ... Reliability calculation section
30 ... Travel time determination unit
31 ... Speed information output unit

Claims (6)

Image input means for capturing images in time series;
A flow detection means for detecting an optical flow having information relating to movement of each pixel between images from the time-series image captured by the image input means;
Flow verification means for verifying the reliability of the flow vector for each pixel in the optical flow detected by the flow detection means for each pixel;
Based on the flow vector recognized as reliable by the flow verification means, a movement time in which the movement amount of the pixel becomes a constant movement amount is determined, and the determined movement time, the constant movement amount, and the pixel movement amount are determined. Speed determining means for determining the moving speed of the pixel based on the direction component of the flow vector;
Comprising
The flow verification means includes
Flow accumulation means for accumulating information on a flow vector for each pixel detected by the flow detection means in time series;
Using the time-series information about the flow vector for each pixel accumulated by the flow accumulation means, the time-series change of the flow vector for each pixel is obtained, and based on this change, the reliability of the flow vector for each pixel is determined. A reliability calculation means for calculating the sex;
A moving image processing apparatus comprising: The flow accumulation means includes
Time-series data relating to the flow vector of each pixel between the reference image at an arbitrary time detected by the flow detection unit and the temporally previous image or the subsequent image compared to the reference image remains as it is. Storing or processing the time-series data into information for determining the reliability of the pixel or information for determining a movement time at which the movement amount of the pixel is a constant movement amount, and storing the information. The moving image processing apparatus according to claim 1 . The reliability calculation means includes
Using the time series information on the flow vector of each pixel accumulated by the flow accumulation means, the time series change of the movement direction of each pixel, the time series change of the movement amount of each pixel, or each pixel Evaluating whether or not the time-series changes in the movement amount and the movement direction of the pixel change based on a predetermined rule, and calculate the reliability of the flow vector detected by the pixel based on the evaluation. The moving image processing apparatus according to claim 1, wherein: The speed determining means includes
Using the time-series information about the flow vector of each pixel stored in the flow storage means, a movement time determination means for calculating a movement time for the movement amount of each pixel to reach the certain movement amount;
Speed information output means for obtaining and outputting the moving speed of the pixel from the determined moving time, the constant moving amount, and the direction component of the flow vector of the pixel;
Moving image processing apparatus according to claim 1, characterized by including the. An image input step for capturing images in time series,
A flow detection step for detecting an optical flow having information on movement of each pixel between images from the time-series image captured in the image input step;
A flow verification step for verifying the reliability of the flow vector for each pixel in the optical flow detected in the flow detection step for each pixel;
Based on the flow vector recognized as reliable in this flow verification step, a movement time in which the movement amount of the pixel becomes a constant movement amount is determined, and the determined movement time, the constant movement amount, and the pixel movement amount are determined. A speed determining step for determining a moving speed of the pixel based on a direction component of the flow vector;
Comprising
The flow verification step includes:
A flow accumulation step for accumulating information on the flow vector for each pixel detected in the flow detection step in time series;
Using the time-series information about the flow vector for each pixel accumulated in this flow accumulation step, the time-series change of the flow vector for each pixel is obtained, and based on this change, the reliability of the flow vector for each pixel is determined. A reliability calculation step for calculating the reliability;
A moving image processing method comprising: An image input function that captures images in time series,
A flow detection function for detecting an optical flow having information on movement of each pixel between images from a time-series image captured in the image input function;
A flow verification function for verifying, for each pixel, the reliability of the flow vector for each pixel in the optical flow detected by the flow detection function;
Based on the flow vector recognized as reliable by the flow verification function, a moving time in which the moving amount of the pixel becomes a fixed moving amount is determined, and the determined moving time, the fixed moving amount, and the pixel A speed determining function for determining a moving speed of the pixel based on a direction component of the flow vector;
Realized,
The flow verification function is
A flow accumulation function for accumulating information on a flow vector for each pixel detected in the flow detection function in time series;
Using the time-series information regarding the flow vector for each pixel accumulated by this flow accumulation function, the time-series change of the flow vector for each pixel is obtained, and the reliability of the flow vector for each pixel is determined based on this change. Reliability calculation function to calculate
The recording medium of the moving image processing method characterized by recording the program which implement | achieves.

Images