JP4049702B2 - VIDEO GENERATION METHOD, DEVICE, PROGRAM, AND RECORDING MEDIUM CONTAINING THE PROGRAM - Google Patents

Description

【００３２】
ここで、
【００３３】
【外１】

【００３４】
は、入力する２次元画像における、２次元平面内での各画素の濃淡値を示している。同様に、
【００３５】
【外２】

【００３６】
は、入力画像における各画素ごとの２次元速度を示している。∇は空間１次微分であり、ｔは時刻である。Ｉ_x，Ｉ_y，Ｉ_tをそれぞれ、画素濃淡値の、空間（ｘ方向及びｙ方向）と時間の１次微分とする。すなわち本実施形態では、移流方程式は、画像の濃淡値を主変数とするものであって、画素単位に、時間と空間とで記述された時間依存の偏微分方程式である。
【００３７】
本実施形態では、入力画像すなわち数値演算の初期値となるべき初期画像に基づいて、方法１または方法２によって得られた速度を画素単位で移流方程式に与えて、時間発展を計算することにより、未来または過去再現が容易に画像として次々と生成される。言い換えれば、入力画像として与えられた初期の画像パターンが、時間と空間で変化して、出力画像となるパターンが生成される。移流方程式において、速度は、移流項に代入される。
【００３８】
次に、時間の１次微分について、未来再現、過去再現及びオプティカルフローでの速度計算を行う方法について示す。移流方程式から、１つの先の時刻の未来の濃淡値を計算するためには、時間項で、時刻ｎ＋１での濃淡値を未知数として、時刻ｎまでの画像を事前知識として考えればよい。速度を自動計算する場合は、時刻ｎ，ｎ−１での画像を用いる。これを数式でまとめると、
【００３９】
【数２】

【００４０】
となる。過去再現の場合も同様に、時刻ｎ，ｎ−１での事前知識から、１つ過去の時刻に溯って、ｎ−２での濃淡値は、
【００４１】
【数３】

【００４２】
と表現される。未来の任意の時刻におけるパターンの生成については、
【００４３】
【数４】

【００４４】
として、再帰的に求めていけばよい。計算アルゴリズムとしては、ｎ，ｎ＋１での値を繰り返し入れ替えればよい。過去の任意の時刻におけるパターンの再現の場合も、同様である。
【００４５】
ここで、移流基本方程式について、その導出過程を説明しておく。
【００４６】
ある変量Ｆを考え、Ｆの微小時間における時間と１次元移動の変化するものとして、２次オーダまでテイラー展開すると、式(A1)が得られる。
【００４７】
【数５】

【００４８】
ここで、変量Ｆに発達や衰退などの変化がない場合、Ｆに対する１次元の移流基本方程式を導出できる。特に、空間微分を拡張すれば、式(A2)に示すように、２次元移流方程式を得る。
【００４９】
【数６】

【００５０】
次に、式(A2)を風上差分法により離散化すると、式(A3)が得られる。この方法は、流れの方向に応じて、前進差分と後退差分を適応的選択して計算することで、誤差伝播の影響による差分誤差を抑制することを特徴とする。式(A3)は１次オーダである。
【００５１】
【数７】

【００５２】
この式(A3)をテイラー展開してその特性を解析する（式(A4)）。
【００５３】
【数８】

【００５４】
式(A4)の右辺第２項は、数値拡散項であり、数値解を平滑化する効果を有する。右辺第３項は数値分散項であり、数値解を振動させる。次に、式(A3)から３次オーダの風上差分式を導出する。空間上の１点を近傍の３点を用いて、２次式で補間をすると、式(A5)を得る。
【００５５】
【数９】

【００５６】
速度の絶対値を用いて式(A5)を１つにまとめると、式(A6)を得る。
【００５７】
【数１０】

【００５８】
式(A6)の特性を見るために、テイラー展開すると、数値分散項の
【００５９】
【数１１】

【００６０】
が含まれる。安定化のために、第１項を
【００６１】
【数１２】

【００６２】
で置き換えて相殺効果をつくる（河村スキーム）。さらに、式(A6)の第２項に含まれる４階微分の数値拡散項の効果を緩和させるために、
【００６３】
【数１３】

【００６４】
で置き換える。これらをまとめて、本実施形態で適用している３次オーダの風上差分式(A7)を得る。ここでは１次元での式を示しているが、２次元の場合も同様に導出できる。
【００６５】
【数１４】

【００６６】
式(A7)をテイラー展開で解析する。αの大きさによって、４階微分項に起因する数値拡散効果が大きいことがわかる。
【００６７】
【数１５】

【００６８】
以上、本実施形態で用いる移流基本方程式の導出を説明した。ここで「基本方程式」としているのは、以下に説明するように、本実施形態では、移流項のほかに、必要に応じて、拡散項、発達項、衰退項及び停滞項の少なくとも１つを含む移流方程式が使用されるからである。
【００６９】
本実施形態では、移流方程式は、差分法により、離散化近似で解けばよい。また、移流方程式の移流項は、計算機上の離散格子点において、例えば、差分法により、１次、２次及び３次のオーダで近似することが好ましい。
【００７０】
次に、本実施形態で用いる速度場編集ツールについて説明する。
【００７１】
速度場編集ツールは、入力画像を表示しつつ、ユーザーからのその入力画像に適用すべき速度場パターンの指定や修正の入力を受け付けるものであり、入力画像上に編集中の速度場パターンを重畳することにより、どのような速度場が設定されるかをユーザーにわかりやすく表示する。速度場編集ツールは、速度場編集部１３としてこの映像生成装置に実装されている。すなわち、速度場の入力や修正のためにユーザーが使用するマウスやキーボードなどが速度場入力部１８に対応し、速度場変更部１９は、速度場編集ツールのうち、ユーザー入力にあわせて速度パターンを変更する部分に対応する。速度場設定部１７は、現在編集中の速度場パターンを表示するとともに、各画素ごとの速度値を計算する速度場の連続化の処理も実行する。この速度場編集ツールでは、基本的な速度場、例えば、渦、拡散、波、傾斜などの形態の速度場パターンが、速度場フィルタとして予め速度場フィルタ記憶部１６に記憶されており、ユーザーは、所望の速度場フィルタを画像上の所望の領域に設定することにより、簡単に速度場の設定を行えるようになっている。
【００７２】
図３は、速度場編集ツールの操作の概要を示す図である。ここでは、雲を示した１枚の静止画像が入力し、この画像に対して、ユーザーがさまざまな速度場パターンを入力し設定して、動画を生成する場合を説明する。
【００７３】
入力画像に対する速度の基本場（すなわち基本速度場）を示すために、ユーザーに対して速度場パレット３１が表示される。図示した例では、速度場パレット３１には、ユーザーが入力した、渦、拡散、波（振動）、傾斜状などの基本速度場が表示されている。また、速度場における速度の大きさや方向を適宜に調整するために、速度場ブラシ３２がいわゆるフローティングパレットとして表示されている。ユーザーは、速度場パレット３１及び速度場ブラシ３２に対しては、マウスなどのポインティングデバイスによって、メニュー選択の要領で指定などを行うことができる。なお、傾斜状の速度場とは、関節物体の支点まわりの回転を表現するものである。
【００７４】
ユーザーが入力した速度場は、そのままでは空間的には不連続場であって、そこから出力画像を生成すると、不自然なパターンが生成してしまう。そこで、初期速度として用いられる連続した速度場をつくるために、ユーザーによって入力された速度場に対して、移動平均フィルタを何回か作用させる。次に、必要に応じてであるが、流体力学の分野で広く用いられているナビエ・ストークス(Navier-Stokes)方程式を適用する。その結果、時間的な変化が自然に近い時系列画像３３を計算することができる。
【００７５】
一方、図３の画像３４は、従来法により市販の画像編集ツールを適用して動画を生成する際の手順を示している。従来法により動画を生成する場合には、例えば、画像に対してメッシュをあてはめて、マウスなどを用いてメッシュ自体を手作業で移動、変形することによって、少しずつ雲の変形を行うことになる。さらに、ぼかしなどのフィルタをかけるなどの加工処理を加える。しかしながら従来法では、連続した表現を実現することできない。これは、メッシュやフィルタなどをユーザーが感覚的に調整することによって画像間の変形の度合いを設定することに起因している。それに対して本実施形態の方法は、数理方程式に基づいて画像間の変形の度合いを設定しているために、各画素単位ごとに数学的かつ物理的に変化するとともに連続した表現を実現した時系列画像を得ることができるのが特徴である。
【００７６】
以下、図４、図５及び図６を用いて、本実施形態において用いられる速度場ブラシの例を説明する。
【００７７】
図４は、渦状の速度場パターンに対応する速度場ブラシを示している。速度ブラシの中央のほぼ正方形の領域内には、大きさと方向とを有する多数の矢印によって、速度場パターンとして、各点での速度の大きさと方向とが示されている。図４の（ａ）は、渦の加速度成分の調整を示している。加速度成分は、速度における変化を表現する。この速度場ブラシによれば、速度場ブラシの右端にあるスライダーバーを操作することによって、渦について、中心から外周に向けて速度場における速度の大きさを変化させることができ、加速度表現を実現することができる。図４の（ｂ）は、渦の強度の調整を示しており、速度場パターンの表示領域の左に隣接するスライダーバーによって渦の強度を調整できることを示している。図４の（ｃ）は、渦の広がりの調整を示しており、速度場パターンの表示領域の下に隣接するスライダーバーによって渦の広がりを調整できることを示している。
【００７８】
図５は、振動を表現する速度場パターンに対応する速度場ブラシを示している。この図に示されるように、振動表現についても、その振幅や振動数（波長）をスライダーバーによって調整できる。例示していないが、速度の大きさもスライターバーによって調整できる。
【００７９】
図６の（ａ）は、発散場（発達）の速度場パターンにおける加速度調整を示しており、ここでも、速度場パターンの表示領域の左に隣接するスライダーバーによって、発散場の加速度を調整できる。ここでは例示していないが、速度場パターンの表示領域の下に隣接するスライダーバーによって、発散場の速度の大きさを調整できる。図６の（ｂ）は、収束場（衰退）の速度場パターンにおける速度調整を示しており、ここでも、速度場パターンの表示領域の下に隣接するスライダーバーによって、収束場の速度の大きさを調整できる。ここでは例示していないが、速度場パターンの表示領域の左に隣接するスライダーバーによって、収束場の加速度を調整できる。
【００８０】
さらに本実施形態では、拡散効果を表わす速度場パターンも用意される。拡散効果は、文字通り、初期状態での画素濃淡値が画像の全体にわたって、広がっていく効果である。出力される画像としてはボケが生じる。この拡散には、等方と異方の２つのタイプが知られており、図７は、等方拡散法と異方拡散法の相違を示している。
【００８１】
等方拡散は、図７の（ａ）に示すように、濃淡値を変数として、その２次の空間微分項と時間項からなる方程式で記述される。拡散係数の調整により、単位時間ステップ当りの、ぼやけの度合いが異なる。濃淡値領域７１において、等方的に初期の濃淡値が薄まるように広がっていくのが特徴である。
【００８２】
一方、異方拡散では、図７の（ｂ）に示すように、特定の方向だけボケが生じるように計算が進められる。異方拡散は、次式のように表現される。
【００８３】
【数１６】

【００８４】
この式において、右辺第１項のｃ(ｘ,ｙ,ｚ)は、拡散係数に相当する。右辺第２項では、濃淡値についての空間での１次微分が表わされている。異方拡散の表現についての詳細は、Ｐｅｒｏｎａらの論文（P. Perona and J. Malik, "Scale-space and edge detection using anisotropic diffusion", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 12, No. 7, pp. 629-639, 1990；非特許文献５）に記載されている。。この異方拡散の基本的な効果は、エッジ構造７３を保存しながら、緩やかな濃淡値領域７２のみをぼけさせるというものである。なお、等方拡散の場合には、エッジ構造を含めてすべての領域、画像特徴量にボケが生じる。画像生成においては、ボケを生じることなくノイズを削減することが必要な場合があるが、異方拡散は、このような目的において有効である。
【００８５】
さらに、このような等方拡散、異方拡散を速度場に加えることのもう一つの重要な効果として、数値計算が安定化することが挙げられる。すなわち、上述した移流基本方程式のみでは数値計算に不安定性が生じるが、等方拡散及び異方拡散の少なくとも一方を表わす拡散項を移流基本方程式に追加することで、数値計算が安定化することが知られている。
【００８６】
次に、時系列に連続する２枚の画像があったときに画素ごとにその一方の濃淡値から他方の濃淡値を減算して得られる差分画像について説明する。このような差分画像では、発達（発散）、衰退（収束）及び停滞の３つの状態がある。図８は、これらの３つの状態の相違を示している。図８に示すように、画像における特定の領域７１が、時間の経過に伴って図示矢印に示すように移動したものとする（移動には、移動距離がゼロすなわち移動していない場合も含まれるものとする）。すると、その特定の領域８１の面積が、移動前と移動後とでは変化することがあるが、ここで移動後の方が面積が大きくなっている場合には「発達」であり、移動後の方が面積が小さければ「衰退」であり、移動前後で面積が変化していなければ「停滞」である。
【００８７】
これら３つの状態は、連続する２つの画像からの差分値の大きさと正負により、しきい値により分類される。図８に示すように移動前の領域と移動後の領域とが一部で重なっているとすると、移動後の領域には含まれるが移動前の領域には含まれない部分８２と、移動前の領域には含まれるが移動後の領域には含まれない部分８３とが生じる。ここで部分８２の面積の方が部分８３の面積よりも大きければ「発達」ということになり、部分８２の面積の方が部分８３の面積よりも小さければ「衰退」ということになる。領域８１がその周囲から濃淡値によって明確に区別されるとすれば、部分８２，８３は、濃淡値の差分値から識別することができる。したがって、２つの画像からの差分値の大きさ及び正負に基づいて、上述した３つの状態を識別することができる。
【００８８】
また、時系列に連続する２枚の画像を入力画像とする場合には、これらの画像間の差分画像の濃淡値を移流方程式の主変数として代入することによって、移動を伴った発達、衰退あるいは停滞の表現を行うこともできる。また、入力画像が１枚である場合においても、上述したように速度場編集ツールにおいて、ユーザーが、発達、衰退あるいは停滞について、マウス操作により領域を特定して指定することもできる。
【００８９】
このように発達、衰退あるいは停滞がある場合の移流方程式は、
【００９０】
【数１７】

【００９１】
において、変数Ｉに、差分画像が入力される。移動速度については、パターン全体からオプティカルフローによる計算結果が適用される。
【００９２】
以上、本実施形態において用いられる効果すなわち移流、拡散及び３つの状態（発達、衰退、停滞）について説明したが、これらについては、数式表現としては線形和で表わすことができる。すなわち、（移流）＋（拡散）＋（３つの状態）を各画素について表現し、その時間発展を計算すればよい。このような時間発展は、画像生成部１４において計算される。
【００９３】
以上説明した実施形態においては、パターン予測に必要な速度場を形成するために、移動平均フィルタを使用している。しかしながら、速度場の形成には、移動平均フィルタ以外の種々のものも使用することができる。例えば、ナビエ・ストークス方程式を用いることができる。図９は、ナビエ・ストークス方程式による流体的な速度場の一例を示している。移動平均フィルタは、速度場の低周波数成分が伝播する点はいいのだが、流体的な性質は含まれていない。そこで、流体力学の分野で広く適用されている、ナビエ・ストークス方程式を適用することにより、流体的性質を有するように画像を生成することができる。
【００９４】
ナビエ・ストークス方程式は、速度と気圧（圧力）の２つを主変数にもつ。本実施形態では、速度場を速度の初期値として、ナビエ・ストークス方程式を解いている。以下、本実施形態におけるナビエ・ストークス方程式（以下、ＮＳ方程式と記述する）の解法を説明する。
【００９５】
ここでは、ＨＳＭＡＣ法と呼ばれる方法でＮＳ方程式を解いている。この方法では、ＮＳ方程式と連続方程式を圧力（もしくは気圧）に関して調整をしながら、非線形連立方程式を反復的に解いていく。ＮＳ方程式は、速度と圧力の２つの独立変数を含む。流れの対象を非圧縮流体と仮定すると、離散化した連続方程式は、式(A8)に示すように、ゼロが速度場に対する条件となる。計算格子点上、圧力変数は１メッシュごとに１変数、速度変数については、その垂直・水平成分を格子点上に配置するものとする。
【００９６】
【数１８】

【００９７】
また、ここでは、式(A9)で示されるように、降水パターンへの外部力はないものとする。
【００９８】
【数１９】

【００９９】
次に、未来の速度場を得るために、時間項に関して、式(A10)で示すように、前進差分化を行う。
【０１００】
【数２０】

【０１０１】
この方法では、オプティカルフローにより推定された速度成分をＮＳ方程式を解くときの初期値として与え、圧力については全領域ゼロとして推定している。境界条件は、画像輪郭部に連続条件を課している。また、粘性係数と密度については経験的に決定した。
【０１０２】
各メッシュごとに、圧力の傾きを考えた場合、その傾きに沿って速度の方向と大きさが決定される。式(A8)が各メッシュごとに満たされるように、圧力を調整するような解法をとる。式(A11)，(A12)を用いて、メッシュごとに速度成分と圧力を微小量ずつ、全メッシュについて反復的に計算を進める。この反復計算は、条件式(A8)が一定の微小値未満となるまで続けられる。
【０１０３】
【数２１】

【０１０４】
収束した結果から、ある離散時間から次の時間ステップまでの、速度成分と圧力が予測できる。したがって、時間積分を所定回数繰り返すことによって、任意時刻に対応する画像を得ることができる。
【０１０５】
図１０は、１枚の画像から動画を生成した例を示している。図示左側に示すように、テクスチャーを有する１枚のパターンがあり、そこには、上述した本実施形態での手法に基づいて、速度場が書き込まれている。そして、このような速度場に基づいて画像を生成することにより、図示右側に示すように、意図通りにテクスチャー模様を変化させることができ、移動変形された画像を生成することができた。
【０１０６】
以上説明した本実施形態の映像生成装置は、それを実現するための計算機プログラムを、パーソナルコンピュータなどの計算機に読み込ませ、そのプログラムを実行させることによっても実現できる。映像生成装置を実現するためのプログラムは、ＣＤ−ＲＯＭなどの記録媒体によって、あるいはネットワークを介して計算機に読み込まれる。
【０１０７】
このような計算機は、一般に、ＣＰＵ（中央処理装置）と、プログラムやデータを格納するためのハードディスク装置と、主メモリと、キーボードやマウスなどの入力装置と、ＣＲＴなどの表示装置と、ＣＤ−ＲＯＭ等の記録媒体を読み取る読み取り装置と、ネットワークに接続するための通信インタフェースと、画像を読取るための画像入力装置（スキャナ装置など）から構成されている。ネットワークを介して、あるいは記録媒体から画像データを読込む場合には、スキャナ装置などの画像入力装置は必ずしも設けなくてもよい。ハードディスク装置、主メモリ、入力装置、表示装置、読み取り装置、通信インタフェース及び画像入力装置は、いずれもＣＰＵに接続している。この計算機では、映像生成装置を実現するためのプログラムを格納した記録媒体を読み取り装置に装着して記録媒体からそのプログラムを読み出してハードディスク装置に格納し、あるいはネットワークを介してそのようなプログラムをダウンロードしてハードディスク装置に格納し、ハードディスク装置に格納されたプログラムをＣＰＵが実行することにより、上述したような映像生成装置として機能することになる。なお、生成された静止画像や動画は、ＣＲＴなどの表示装置上で表示してもよいし、あるいは、画像データとして、ハードディスク装置に格納したり、取外し可能な記録媒体に記録したり、あるいは、ネットワークを介して外部に送信してもよい。
【０１０８】
【発明の効果】
以上説明したように本発明は、元の時系列画像から自動的に、連続した別の時系列画像を生成したり、１枚の静止画像に基づいて、速度場編集ツールを介して、連続した動画を効率的に生成したりすることができるという効果がある。本発明は、画像の中の特定の対象領域の変形、移動や、背景などのテクスチャーのテクスチャー変形や移動など、いずれも画素を最小単位した細かい変化、微妙な変化をも許容するような動画生成に関して有効である。さらに本発明では、実画像をそのまま変化させるので、その実画像のようなパターンを有する画像を容易に得ることができる。
【図面の簡単な説明】
【図１】本発明の実施の一形態の映像生成装置の構成を示すブロック図である。
【図２】図１に示した映像生成装置の動作原理を説明する図であって、１枚の入力画像が与えられたとき（方法１）及び２枚の入力画像が与えられたとき（方法２）のそれぞれの場合における、過去及び未来方向への動画生成を説明している。
【図３】速度場編集ツールの操作を説明する図である。
【図４】速度場ブラシの例を示す図である。
【図５】速度場ブラシの例を示す図である。
【図６】速度場ブラシの例を示す図である。
【図７】等方拡散と異方拡散との相違を示す図である。
【図８】差分画像の３つの状態の相違を示す図である。
【図９】ナビエ・ストークス方程式による流体的な速度場の例を示す図である。
【図１０】映像シーンでの生成画像の一例を示す図である。
【符号の説明】
１１ 画像入力部
１２ 画像蓄積部
１３ 速度場編集部
１４ 画像生成部
１５ 画像出力部
１６ 速度場フィルタ記憶部
１７ 速度場設定部
１８ 速度場入力部
１９ 速度場変更部
３１ 速度場パレット
３２ 速度場ブラシ
３３ 時系列画像
３４ 画像
７１，７２ 濃淡値領域
７３ エッジ構造
８１ 領域
８２，８３ 部分[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a video generation method and apparatus for editing an image, and in particular, freely edits an image obtained from various observation sources such as a camera or a web site on the Internet as a user intends. The present invention relates to a video generation method and apparatus for generating a video scene.
[0002]
[Prior art]
With the diversification of video, such as CM (commercial), animation, movie, and personal video, it is desired to generate video from one photo (still image), generate video from two still images, etc. ing. In particular, it is thought that the spread of old photographs etc. will be digitized through scanners and cameras and used as materials to change facial expressions, landscapes, and even express river flows as videos. .
[0003]
There have been many commercial editing tools for editing images (eg, Adobe).^(R)The company's Photoshop^(R)And Illustrator). Most of these commercially available editing tools change the color and texture of an image by a function prepared in advance for artificial processing of a still image. As for deformation, it has a simple mesh deformation function. When creating a video (video) from multiple still images, change the still image one by one and change the adjustment parameters little by little, and play back these multiple still images as continuous images, that is, videos. ing. In this case, since the slight change amount between the still images constituting the moving image is also set based on the user's sense, the obtained moving image tends to be discontinuous.
[0004]
In addition, there are many tools called moving image generation in which a user sets a movement locus of a center of gravity such as a sphere, and the sphere moves according to the set movement locus. In the face deformation operation using these teeth, a basic wire frame of the face is prepared in advance, and the user manually aligns the basic wire-frame with a human face image or the like. After that, texture mapping is performed on the wire frame, and the mesh points of the wire frame are moved to create the required moving image deformation expression.
[0005]
In SIGGRAPH: Leonard McMillan and Gary Bishop, "Plenoptic modeling an image-based rendering system", pp. 39-46, 1995 Therefore, methods for artificially creating depth maps and removing shadows have been proposed. However, Non-Patent Document 1 does not describe a method for creating a moving image using a fluid pattern from a single photograph.
[0006]
ICCV: S.Soatto, G.Doretto, and YNWu, "Dynamic texture", 2001 (Non-Patent Document 2) uses a dozen or more past images and applies them to an autoregressive model to create a fluid future A method of creating an image has been proposed. However, in this method, when generating a future image from only one image, it is necessary to input the speed separately. However, since the autoregressive model does not have a speed term explicitly, it matches the speed field input. It may not be good, and appropriate future images may not be obtained. In addition, even if several tens or more past images are used, since the high-frequency component is removed from the generated future image, only a blurred image can be generated.
[0007]
ECCV: Yizhou Wang and Song-Chun Zhu, "A generative method for textured motion: analysis and synthesis", pp. 583-598, 2002 (Non-Patent Document 3) Individually, a method of designing a deformable image element called “movon” and approximating each video pattern by a necessary number has been proposed. The trajectory along which each image element moves is described by a second-order Markov model. In this method, the design of the image element and the number of elements for each image pattern must be artificially designed in advance, and the minimum unit is the size of a dozen pixels by the image element. There is a problem that becomes a blurred pattern.
[0008]
In the prior art, there is no tool that can easily generate a complex texture of an image, a background of a scene, and a full view from a still image to a moving image. In particular, there is no method that leaves clearness.
[0009]
[Non-Patent Document 1]
SIGGRAPH: Leonard McMillan and Gary Bishop, "Plenoptic modeling an image-based rendering system", pp. 39-46, 1995
[0010]
[Non-Patent Document 2]
ICCV: S. Soatto, G. Doretto, and Y. N. Wu, "Dynamic texture", 2001
[0011]
[Non-Patent Document 3]
ECCV: Yizhou Wang and Song-Chun Zhu, "A generative method for textured motion: analysis and synthesis", pp. 583-598, 2002
[0012]
[Non-Patent Document 4]
B. K. P. Horn and B. G. Schunck, "Determining Optical Flow", Artifical Intelligence, Vol. 17, No. 1, pp. 185-203, 1981
[0013]
[Non-Patent Document 5]
P. Perona and J. Malik, "Scale-space and edge detection using anisotropic diffusion", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 12, No. 7, pp. 629-639, 1990
[0014]
[Problems to be solved by the invention]
After all, when trying to generate a moving image with a conventional image editing tool, the user has manually performed several steps such as filtering one by one on the still image, It was forced to adjust the amount of movement and deformation sensuously. For this reason, it takes a lot of time and cost to create a moving image that exceeds several hundred still images.
[0015]
  Therefore, the purpose of the present invention is1A method and apparatus capable of efficiently generating a continuous moving image based on a still image of a sheet through a speed field editing tool, and allowing fine changes and subtle changes with a minimum unit of pixels. It is to provide. Examples of the fine change and subtle change with the smallest unit of pixel include deformation and movement of a specific object in the image, and texture deformation and movement of a texture such as a background.
[0016]
[Means for Solving the Problems]
  The video generation method of the present invention includes:
  Video generation for generating an output image in units of pixels using an advection equation including at least one of a diffusion term, a development term, a decay term and a stagnation term and an advection term based on the input image as a single still image In the method
  A presentation step of presenting a plurality of types of velocity field patterns prepared in advance to the user;,
  The plurality of typesBy the user from within the velocity field patternAt least two typesThe selected velocity field pattern is superimposed on each of the different partial areas over the entire display area of the input image by a symbol indicating the direction, and input by the user via the selected velocity field pattern and the graphical user interface. A velocity field generation step of calculating a velocity for each pixel based on the magnitude and direction of the velocity and generating a velocity field representing the velocity for each pixel;
  A continuous process for generating a continuous velocity field by applying a moving average filter to the generated velocity field;,
  SaidConsecutiveApply the advection equation to the input image based on the velocity field to generate the output imageImage generationAnd a process.
[0017]
  In the video generation method of the present invention,,singleOne still imageTeyoYes.thisIn this case, the input image is displayed to the user, the designation of the velocity field is accepted from the user, and the velocity field is set based on the designated velocity field.TheFurther, in the video generation method of the present invention, an output image that is a time-series image composed of a plurality of still images that are continuous in time series is obtained by repeatedly executing the process of generating an output image corresponding to different times. It is preferable to do.
[0018]
  In the present invention, the advection equation is, for example, a time-dependent partial differential equation described in time and space in units of pixels with the gray value of an image as a main variable. The advection equation includes at least one term selected from a diffusion term, a development term, a decay term and a stagnation term in addition to the advection term as a term for expressing the velocity field.Ru. Here, the diffusion term is a term expressing at least one of isotropic diffusion and anisotropic diffusion, for example.The
[0019]
Furthermore, in the present invention, a step of predicting a velocity field using the Navier-Stokes equation and a moving average filter may be provided.
[0020]
  The video generation apparatus of the present invention
  Video generation that generates an output image pixel by pixel using an advection equation including at least one of a diffusion term, a development term, a decay term and a stagnation term and an advection term based on the input image as a single still image In the device
  An image input unit for inputting the input image;
  A presentation unit for presenting a plurality of types of velocity field patterns prepared in advance to the user;,
  The plurality of typesBy the user from within the velocity field patternAt least two typesThe selected velocity field pattern is superimposed on each of the different partial areas over the entire display area of the input image by a symbol indicating the direction, and the selected velocity field pattern and the user via the graphical user interface are displayed by the user. A speed field editing unit that calculates a speed for each pixel based on the magnitude and direction of the input speed and generates a speed field representing the speed for each pixel;
  A continuous unit that generates a continuous velocity field by applying a moving average filter to the generated velocity field;,
  SaidConsecutiveAn image generation unit configured to generate the output image by applying the advection equation to the input image based on a velocity field.
[0021]
This video generation device changes a speed field based on an input to a speed field input unit that receives a speed field setting instruction from a user and a correction to the speed field set by the speed field setting unit. And a speed field changing unit to be provided. It is preferable that the user can set a velocity field for each region in the input image with a pointing device such as a mouse. The video generation apparatus may include a speed field filter storage unit that stores a plurality of speed field filters indicating the basic speed field in advance, and sets the speed field based on the speed field filter. In this case, the user can select the velocity field to be applied from various basic velocity field patterns, and can set the velocity field on the input image to be edited while adjusting the size and shape of the velocity field. It is preferable to make it. Examples of the basic velocity field include vibration (wave), vortex, divergence, convergence, and inclined velocity. The parameters that can be adjusted in such a basic velocity field are, for example, the frequency and amplitude in the case of vibration, and the velocity and acceleration in vortex, divergence, and convergence. Furthermore, it is preferable that the user can specify and specify a region for development, decline, and stagnation.
[0022]
DETAILED DESCRIPTION OF THE INVENTION
Next, a preferred embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a video generation apparatus according to an embodiment of the present invention.
[0023]
This video generation apparatus receives a time-series image (that is, a moving image) composed of a single two-dimensional still image or a plurality of time-sequential two-dimensional still images as input, and the time zone of the input still image Is to generate continuous time-series images in different time zones. The video generation device determines the value of each pixel of each image in the time-series image, thereby generating a time-series image in units of pixels.
[0024]
The video generation apparatus is set or changed by an image input unit 11 for inputting an image, an image storage unit 12 for storing the input image, a speed field editing unit 13 for setting a speed field filter and changing a speed field. Based on the velocity field thus generated, the image accumulated in the image accumulation unit 12 is subjected to processing based on the advection equation, and an image generation unit 14 that generates an image, an image output unit 15 that outputs the generated image, The image generation unit 14 generates a moving image by repeatedly executing image generation corresponding to different times. Here, the velocity field determines how each pixel of the still image is moving in the newly generated image. For each position of the image, the direction and size of the movement are defined. Yes. The speed field editing unit 13 includes a speed field filter storage unit 16 that stores a plurality of filters (speed field filters) representing such a speed field in advance, and a speed field stored in the speed field filter storage unit 16. One of the filters is selected, or the velocity field is estimated from a plurality of input images and set as a velocity field to be applied to the input image, and set by the velocity field setting unit 17 A speed field input unit 18 that accepts a correction from the user with respect to the speed field, and a speed field change unit 19 that changes the speed field based on an input from the user. When the speed field is not corrected by the user, the speed field input unit 18 and the speed field changing unit 19 may not be provided.
[0025]
Next, a video generation procedure using the above-described video generation apparatus will be described. This generation procedure is based on the video generation method of the present invention. According to this generation procedure, a continuous time series is obtained from a camera (so-called webcam) installed on a website on a network such as the Internet, a still image on the website, or a still image obtained from a scanner or a digital camera. An image (moving image) is generated for each pixel. Further, a time series image in a different time zone such as the past or the future is generated from a time series image (including at least two still images) in a certain time zone.
[0026]
A still image or a moving image is input to the image input unit 11 and stored in the image storage unit 12. The speed field editing unit 13 automatically estimates the speed field from the input image and sets the speed field based on the estimation result, or sets the speed field according to the scenario assumed by the user. When setting the velocity field according to the scenario assumed by the user, for example, the velocity field represented by vortex, divergence, convergence, vibration, wave shape, inclination, etc. A speed field is prepared in advance in the storage unit 16 and interactively presented to the user according to an input from the user, and the speed field is corrected according to a correction input from the user. Then, the image generation unit 14 generates an output image from the input image based on the set speed field. In this case, an output image is obtained by performing numerical calculation on the input image by an advection equation which is a time-dependent partial differential equation. This output image is an image in which the input image changes based on the velocity field for a specified time (past direction or future direction) from the time corresponding to the input image. Therefore, by repeating the numerical calculation based on the advection equation until the past and the future, a time-series image in a time zone different from the time (or time zone) corresponding to the input image can be obtained. The image generated in this way is output from the image output unit 15 and displayed, for example, as a moving image.
[0027]
As described above, in this embodiment, when there are two input images and the speed field is automatically estimated from there (method 1), there is only one input image, and the user has a speed corresponding thereto. An output time-series image can be obtained by two methods of specifying a field (method 2). FIG. 2 shows a comparison of the processing in Method 1 and Method 2.
[0028]
In the case of method 1, that is, when there are two input images and both are continuous time-series images, the times corresponding to these images are assumed to be t−1 and t. In the present embodiment, for example, the corresponding pixels in the two input images are determined by an automatic velocity estimation method such as an optical flow method, and how much the coordinates of the corresponding pixels are shifted in both the input images is determined. Based on the deviation, the moving speed (direction and size) of the pixel is determined for each pixel. At this time, by replacing the times t-1 and t and obtaining the difference, the future direction and the past direction can be easily interchanged. That is, the future can be represented by the order of times t-1 and t and the past can be represented by the order of times t and t-1. Details of the optical flow method are described in, for example, BKP Horn and BG Schunck, “Determining Optical Flow”, Artifical Intelligence, Vol. 17, No. 1, pp. 185-203, 1981 (Non-patent Document 4). Yes.
[0029]
On the other hand, in the method 2, the user sets a velocity field corresponding to the scenario from one input image via the velocity field editing tool. At this time, as will be described later, the input image is presented to the user, and the speed field is set by the user drawing a desired speed field on the presented image. Since the designation of the velocity field by the user is not made for each pixel, the velocity field editing tool calculates the velocity for each pixel based on the user input.
[0030]
In either case of method 1 or method 2, since the speed for each pixel is set as described above, a speed field file representing the speed for each pixel is generated. Then, the image generation unit 15 inputs a speed in a pixel unit to the pattern generation equation based on the speed field file, and applies the pattern generation equation to the input image to generate an output image. An advection equation is used as the pattern generation equation. The advection equation used in this embodiment is shown in Equation (1). Only the advection term is shown here.
[0031]
[Expression 1]

[0032]
here,
[0033]
[Outside 1]

[0034]
Indicates the gray value of each pixel in the two-dimensional plane in the input two-dimensional image. Similarly,
[0035]
[Outside 2]

[0036]
Indicates the two-dimensional velocity for each pixel in the input image. ∇ is a spatial first derivative, and t is time. I_x, I_y, I_tAre the first derivative of space (x direction and y direction) and time of the pixel gray value. That is, in this embodiment, the advection equation is a time-dependent partial differential equation described with time and space for each pixel, with the gray value of the image as the main variable.
[0037]
In this embodiment, based on the input image, that is, the initial image to be the initial value of the numerical operation, the speed obtained by the method 1 or 2 is given to the advection equation in units of pixels, and the time evolution is calculated, Future or past reproduction is easily generated one after another as an image. In other words, an initial image pattern given as an input image changes in time and space, and a pattern that becomes an output image is generated. In the advection equation, velocity is substituted into the advection term.
[0038]
Next, a method for performing velocity calculation in the future reproduction, past reproduction, and optical flow for the first derivative of time will be described. In order to calculate the future gray value at one previous time from the advection equation, the gray value at time n + 1 is an unknown number in the time term, and the image up to time n may be considered as prior knowledge. When the speed is automatically calculated, images at times n and n−1 are used. Summarizing this with a formula,
[0039]
[Expression 2]

[0040]
It becomes. Similarly, in the case of past reproduction, from the prior knowledge at time n and n−1, the gray value at n−2 over the past time is
[0041]
[Equation 3]

[0042]
It is expressed. For pattern generation at any time in the future,
[0043]
[Expression 4]

[0044]
As long as it is obtained recursively. As a calculation algorithm, the values at n and n + 1 may be repeatedly replaced. The same applies to the case of pattern reproduction at an arbitrary past time.
[0045]
Here, the derivation process of the basic advection equation will be described.
[0046]
When a certain variable F is considered and Taylor expansion is performed up to the second order assuming that the time and the one-dimensional movement in a minute time of F change, Expression (A1) is obtained.
[0047]
[Equation 5]

[0048]
Here, when there is no change such as development or decline in the variable F, a one-dimensional advection basic equation for F can be derived. In particular, if the spatial differentiation is expanded, a two-dimensional advection equation is obtained as shown in equation (A2).
[0049]
[Formula 6]

[0050]
Next, when equation (A2) is discretized by the upwind difference method, equation (A3) is obtained. This method is characterized in that the difference error due to the influence of error propagation is suppressed by adaptively selecting and calculating the forward difference and the backward difference according to the flow direction. Formula (A3) is the primary order.
[0051]
[Expression 7]

[0052]
This equation (A3) is Taylor expanded and its characteristics are analyzed (equation (A4)).
[0053]
[Equation 8]

[0054]
The second term on the right side of Equation (A4) is a numerical diffusion term, and has the effect of smoothing the numerical solution. The third term on the right side is a numerical dispersion term that vibrates the numerical solution. Next, an upwind difference equation of the third order is derived from the equation (A3). When one point on the space is interpolated by a quadratic equation using three neighboring points, equation (A5) is obtained.
[0055]
[Equation 9]

[0056]
When the formula (A5) is combined into one using the absolute value of the velocity, the formula (A6) is obtained.
[0057]
[Expression 10]

[0058]
To see the characteristics of equation (A6), Taylor expansion
[0059]
[Expression 11]

[0060]
Is included. For stabilization, the first term
[0061]
[Expression 12]

[0062]
To create an offset effect (Kawamura scheme). Furthermore, in order to mitigate the effect of the numerical diffusion term of the fourth derivative included in the second term of the formula (A6),
[0063]
[Formula 13]

[0064]
Replace with. These are put together to obtain the third order upwind difference formula (A7) applied in the present embodiment. Here, a one-dimensional expression is shown, but a two-dimensional expression can be similarly derived.
[0065]
[Expression 14]

[0066]
Analyze equation (A7) with Taylor expansion. It can be seen that the numerical diffusion effect due to the fourth-order differential term is large depending on the magnitude of α.
[0067]
[Expression 15]

[0068]
The derivation of the basic advection equation used in the present embodiment has been described above. In this embodiment, as described below, “basic equation” is used in addition to the advection term, and at least one of a diffusion term, a development term, a decay term, and a stagnation term as necessary. This is because the containing advection equation is used.
[0069]
In this embodiment, the advection equation may be solved by discretization approximation by the difference method. Moreover, it is preferable that the advection term of the advection equation is approximated on the order of the first, second and third order by, for example, a difference method at discrete grid points on the computer.
[0070]
Next, the speed field editing tool used in this embodiment will be described.
[0071]
The speed field editing tool accepts the input of the speed field pattern to be applied to the input image from the user while displaying the input image, and superimposes the speed field pattern being edited on the input image. By doing so, the user can easily see what kind of speed field is set. The speed field editing tool is mounted on the video generation apparatus as the speed field editing unit 13. That is, a mouse or a keyboard used by the user for inputting or correcting the speed field corresponds to the speed field input unit 18, and the speed field changing unit 19 is a speed pattern according to the user input in the speed field editing tool. Corresponds to the part to be changed. The speed field setting unit 17 displays a speed field pattern that is currently being edited, and also executes a speed field continuation process for calculating a speed value for each pixel. In this speed field editing tool, a basic speed field, for example, a speed field pattern in the form of vortex, diffusion, wave, inclination or the like is stored in the speed field filter storage unit 16 in advance as a speed field filter, and the user can By setting a desired velocity field filter in a desired region on the image, the velocity field can be easily set.
[0072]
FIG. 3 is a diagram showing an outline of the operation of the speed field editing tool. Here, a case will be described in which a single still image showing a cloud is input, and a user inputs and sets various velocity field patterns for the image to generate a moving image.
[0073]
A speed field palette 31 is displayed to the user to indicate the basic speed field (ie, the basic speed field) for the input image. In the illustrated example, the velocity field palette 31 displays basic velocity fields such as vortices, diffusion, waves (vibrations), and inclined shapes input by the user. Further, the speed field brush 32 is displayed as a so-called floating palette in order to appropriately adjust the magnitude and direction of the speed in the speed field. The user can designate the speed field palette 31 and the speed field brush 32 in the manner of menu selection using a pointing device such as a mouse. The inclined velocity field expresses rotation around the fulcrum of the joint object.
[0074]
The velocity field input by the user is a spatially discontinuous field as it is, and when an output image is generated therefrom, an unnatural pattern is generated. Therefore, in order to create a continuous velocity field used as the initial velocity, the moving average filter is applied several times to the velocity field input by the user. Next, the Navier-Stokes equation widely used in the field of fluid dynamics is applied as necessary. As a result, it is possible to calculate the time series image 33 whose temporal change is close to natural.
[0075]
On the other hand, an image 34 in FIG. 3 shows a procedure for generating a moving image by applying a commercially available image editing tool by a conventional method. When a moving image is generated by the conventional method, for example, a mesh is applied to an image, and the mesh itself is manually moved and deformed by using a mouse or the like, so that the cloud is gradually changed. . Furthermore, processing such as applying a filter such as blurring is added. However, the conventional method cannot realize continuous expression. This is because the degree of deformation between images is set by the user's sensuous adjustment of meshes, filters, and the like. On the other hand, since the method of the present embodiment sets the degree of deformation between images based on mathematical equations, it realizes a continuous expression while changing mathematically and physically for each pixel unit. A feature is that a series image can be obtained.
[0076]
Hereinafter, an example of the velocity field brush used in the present embodiment will be described with reference to FIGS. 4, 5, and 6.
[0077]
FIG. 4 shows a velocity field brush corresponding to a vortex velocity field pattern. In a substantially square area in the center of the speed brush, the speed magnitude and direction at each point are shown as a speed field pattern by a number of arrows having magnitude and direction. FIG. 4A shows the adjustment of the acceleration component of the vortex. The acceleration component represents a change in speed. According to this velocity field brush, by operating the slider bar at the right end of the velocity field brush, the magnitude of the velocity in the velocity field can be changed from the center to the outer periphery of the vortex, realizing acceleration expression can do. FIG. 4B shows adjustment of the strength of the vortex, and shows that the strength of the vortex can be adjusted by a slider bar adjacent to the left of the display area of the velocity field pattern. FIG. 4C shows the adjustment of the vortex spread, and shows that the vortex spread can be adjusted by the slider bar adjacent below the display area of the velocity field pattern.
[0078]
FIG. 5 shows a velocity field brush corresponding to a velocity field pattern representing vibration. As shown in this figure, the amplitude and frequency (wavelength) of the vibration expression can be adjusted by the slider bar. Although not illustrated, the magnitude of the speed can also be adjusted by the lighter bar.
[0079]
FIG. 6A shows acceleration adjustment in the velocity field pattern of the divergent field (development). Here, too, the acceleration of the divergent field can be adjusted by the slider bar adjacent to the left of the display area of the velocity field pattern. . Although not illustrated here, the magnitude of the divergence field speed can be adjusted by a slider bar adjacent below the display area of the speed field pattern. FIG. 6B shows the speed adjustment in the speed field pattern of the convergence field (decay). Here again, the magnitude of the speed of the convergence field is indicated by the slider bar adjacent below the display area of the speed field pattern. Can be adjusted. Although not illustrated here, the acceleration of the convergence field can be adjusted by a slider bar adjacent to the left of the display area of the velocity field pattern.
[0080]
Furthermore, in this embodiment, a velocity field pattern representing a diffusion effect is also prepared. The diffusion effect is literally an effect that the pixel gray value in the initial state spreads over the entire image. The output image is blurred. There are two known types of diffusion, isotropic and anisotropic, and FIG. 7 shows the difference between the isotropic diffusion method and the anisotropic diffusion method.
[0081]
As shown in FIG. 7A, the isotropic diffusion is described by an equation including a second-order spatial differential term and a time term with a gray value as a variable. The degree of blur per unit time step varies depending on the adjustment of the diffusion coefficient. In the gray value area 71, the initial gray value is isotropically spread so as to be thinned.
[0082]
On the other hand, in anisotropic diffusion, as shown in FIG. 7B, the calculation proceeds so that blurring occurs only in a specific direction. Anisotropic diffusion is expressed as:
[0083]
[Expression 16]

[0084]
In this equation, c (x, y, z) in the first term on the right side corresponds to a diffusion coefficient. The second term on the right side represents the first-order differentiation in the space for the gray value. For details on the expression of anisotropic diffusion, see Perona et al. (P. Perona and J. Malik, "Scale-space and edge detection using anisotropic diffusion", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 12, No. 7, pp. 629-639, 1990; Non-Patent Document 5). . The basic effect of this anisotropic diffusion is that only the gentle gray value region 72 is blurred while preserving the edge structure 73. In the case of isotropic diffusion, blurring occurs in all regions and image feature amounts including the edge structure. In image generation, it may be necessary to reduce noise without blurring, but anisotropic diffusion is effective for such purposes.
[0085]
Furthermore, another important effect of adding such isotropic and anisotropic diffusion to the velocity field is that the numerical calculation is stabilized. That is, instability occurs in the numerical calculation only with the above-mentioned basic advection equation, but the numerical calculation can be stabilized by adding a diffusion term representing at least one of isotropic diffusion and anisotropic diffusion to the basic advection equation. Are known.
[0086]
Next, a difference image obtained by subtracting the other gray value from one gray value for each pixel when there are two time-series images will be described. Such a difference image has three states of development (divergence), decline (convergence), and stagnation. FIG. 8 shows the difference between these three states. As shown in FIG. 8, it is assumed that a specific area 71 in the image has moved as shown by the arrow in the drawing with the passage of time (the movement includes a case where the movement distance is zero, that is, not moving). Suppose). Then, the area of the specific area 81 may change before and after the movement. However, when the area after the movement is larger, the area is “development”. If the area is smaller, it is “decline”, and if the area does not change before and after movement, it is “stagnation”.
[0087]
These three states are classified by threshold values according to the magnitude and positive / negative of the difference value from two consecutive images. As shown in FIG. 8, assuming that the area before movement and the area after movement partially overlap, a part 82 that is included in the area after movement but not included in the area before movement, and A portion 83 that is included in the region but not included in the moved region is generated. Here, if the area of the part 82 is larger than the area of the part 83, it means “development”, and if the area of the part 82 is smaller than the area of the part 83, it means “decay”. If the region 81 is clearly distinguished from its surroundings by the gray value, the portions 82 and 83 can be identified from the gray value difference value. Therefore, the three states described above can be identified based on the magnitude and positive / negative of the difference value from the two images.
[0088]
In addition, when two consecutive images in time series are used as input images, by substituting the gray value of the difference image between these images as the main variable of the advection equation, It is also possible to express stagnation. Even when there is only one input image, the user can also specify and specify a region by mouse operation for development, decline, or stagnation in the speed field editing tool as described above.
[0089]
The advection equation when there is development, decline or stagnation is
[0090]
[Expression 17]

[0091]
The difference image is input to the variable I. For the moving speed, the calculation result by the optical flow is applied from the entire pattern.
[0092]
As described above, the effects used in the present embodiment, that is, advection, diffusion, and three states (development, decline, stagnation) have been described. These can be expressed by a linear sum as a mathematical expression. That is, (advection) + (diffusion) + (three states) may be expressed for each pixel, and its time evolution may be calculated. Such time evolution is calculated in the image generation unit 14.
[0093]
In the embodiment described above, a moving average filter is used to form a velocity field necessary for pattern prediction. However, various types other than the moving average filter can be used for forming the velocity field. For example, the Navier-Stokes equation can be used. FIG. 9 shows an example of a fluid velocity field according to the Navier-Stokes equation. The moving average filter is good in that the low frequency component of the velocity field propagates, but does not include fluid properties. Therefore, by applying the Navier-Stokes equation, which is widely applied in the field of fluid dynamics, an image can be generated so as to have fluid properties.
[0094]
The Navier-Stokes equation has two main variables: velocity and atmospheric pressure (pressure). In this embodiment, the Navier-Stokes equation is solved using the velocity field as an initial value of velocity. Hereinafter, a method for solving the Navier-Stokes equations (hereinafter referred to as NS equations) in this embodiment will be described.
[0095]
Here, the NS equation is solved by a method called the HSMAC method. In this method, the nonlinear simultaneous equations are iteratively solved while adjusting the NS equation and the continuity equation with respect to pressure (or atmospheric pressure). The NS equation includes two independent variables, speed and pressure. Assuming that the target of flow is an incompressible fluid, the discretized continuity equation is zero for the velocity field as shown in equation (A8). On the calculation grid point, the pressure variable is one variable for each mesh, and the vertical and horizontal components of the velocity variable are arranged on the grid point.
[0096]
[Formula 18]

[0097]
Here, it is assumed that there is no external force to the precipitation pattern, as shown by the equation (A9).
[0098]
[Equation 19]

[0099]
Next, in order to obtain a future velocity field, forward differentiation is performed with respect to the time term as shown by the equation (A10).
[0100]
[Expression 20]

[0101]
In this method, the velocity component estimated by the optical flow is given as an initial value when the NS equation is solved, and the pressure is estimated as zero in the entire region. The boundary condition imposes a continuous condition on the image outline. The viscosity coefficient and density were determined empirically.
[0102]
When the pressure gradient is considered for each mesh, the direction and magnitude of the speed are determined along the gradient. A solution is taken such that the pressure is adjusted so that equation (A8) is satisfied for each mesh. Using formulas (A11) and (A12), calculation is repeatedly performed for all meshes by minute amounts of velocity component and pressure for each mesh. This iterative calculation is continued until the conditional expression (A8) becomes less than a certain minute value.
[0103]
[Expression 21]

[0104]
From the converged result, the velocity component and pressure from a certain discrete time to the next time step can be predicted. Therefore, an image corresponding to an arbitrary time can be obtained by repeating the time integration a predetermined number of times.
[0105]
FIG. 10 shows an example in which a moving image is generated from one image. As shown on the left side of the figure, there is one pattern having a texture, and a velocity field is written in the pattern based on the method in the present embodiment described above. Then, by generating an image based on such a velocity field, as shown on the right side of the figure, the texture pattern can be changed as intended, and a moving and deformed image can be generated.
[0106]
The video generation apparatus according to the present embodiment described above can also be realized by causing a computer program such as a personal computer to read a computer program for realizing it and executing the program. A program for realizing the video generation apparatus is read into a computer by a recording medium such as a CD-ROM or via a network.
[0107]
Such a computer generally includes a CPU (Central Processing Unit), a hard disk device for storing programs and data, a main memory, an input device such as a keyboard and a mouse, a display device such as a CRT, and a CD- It comprises a reading device that reads a recording medium such as a ROM, a communication interface for connecting to a network, and an image input device (such as a scanner device) for reading an image. When reading image data via a network or from a recording medium, an image input device such as a scanner device is not necessarily provided. The hard disk device, main memory, input device, display device, reading device, communication interface, and image input device are all connected to the CPU. In this computer, a recording medium storing a program for realizing a video generation device is loaded into a reading device, the program is read from the recording medium and stored in a hard disk device, or such a program is downloaded via a network. When the CPU executes the program stored in the hard disk device and stored in the hard disk device, it functions as the video generation device as described above. The generated still image or moving image may be displayed on a display device such as a CRT, or may be stored as image data in a hard disk device, recorded on a removable recording medium, or You may transmit outside via a network.
[0108]
【The invention's effect】
As described above, the present invention automatically generates another continuous time-series image from the original time-series image, or continuously through the velocity field editing tool based on one still image. There is an effect that a moving image can be generated efficiently. The present invention generates a moving image that allows fine and subtle changes in the smallest unit of pixels, such as deformation and movement of a specific target area in an image, and texture deformation and movement of a texture such as a background. It is effective with respect to. Furthermore, in the present invention, since the actual image is changed as it is, an image having a pattern like the actual image can be easily obtained.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a video generation apparatus according to an embodiment of the present invention.
FIGS. 2A and 2B are diagrams for explaining the operation principle of the video generation apparatus shown in FIG. 1, when one input image is given (method 1) and when two input images are given (method). The generation of moving images in the past and future directions in each case of 2) is described.
FIG. 3 is a diagram for explaining an operation of a speed field editing tool.
FIG. 4 is a diagram illustrating an example of a velocity field brush.
FIG. 5 is a diagram illustrating an example of a velocity field brush.
FIG. 6 is a diagram illustrating an example of a velocity field brush.
FIG. 7 is a diagram showing a difference between isotropic diffusion and anisotropic diffusion.
FIG. 8 is a diagram illustrating differences in three states of a difference image.
FIG. 9 is a diagram showing an example of a fluid velocity field by the Navier-Stokes equation.
FIG. 10 is a diagram illustrating an example of a generated image in a video scene.
[Explanation of symbols]
11 Image input section
12 Image storage unit
13 Speed field editor
14 Image generator
15 Image output unit
16 Speed field filter storage unit
17 Speed field setting section
18 Speed field input section
19 Speed field change part
31 Speed field palette
32 Speed field brush
33 Time-series images
34 images
71, 72 Gray value area
73 Edge structure
81 area
82,83 parts

Claims (12)

Video generation that generates an output image pixel by pixel using an advection equation including at least one of a diffusion term, a development term, a decay term and a stagnation term and an advection term based on the input image as a single still image In the method
A presentation step of presenting a plurality of types of velocity field patterns prepared in advance to the user ;
Two or more types of velocity field patterns selected by the user from among the plurality of velocity field patterns are superimposed and displayed with symbols indicating directions on different partial areas over the entire display area of the input image, and the selected A velocity field that calculates a velocity per pixel and generates a velocity field representing the velocity per pixel based on the velocity field pattern and the magnitude and direction of the velocity input by the user via the graphical user interface. Generation process;
A continuous process for generating a continuous velocity field by applying a moving average filter to the generated velocity field ;
An image generating step of generating the output image by applying the advection equation to the input image based on the continuous velocity field;
A video generation method comprising:   2. The video generation according to claim 1, wherein the output image is a time-series image composed of a plurality of still images that are continuous in time series by repeatedly executing the step of generating the output image corresponding to different times. Method.   The video generation method according to claim 1, wherein the diffusion term is a term representing at least one of isotropic diffusion and anisotropic diffusion.   The video generation method according to claim 1, further comprising a step of solving the advection equation by a discretization approximation by a difference method.   4. The video generation method according to claim 1, wherein the advection term is approximated in a first-order, second-order, or third-order order at a discrete grid point on a computer by a difference method. The video generation method according to claim 1, wherein the velocity field pattern prepared in advance represents a form of vortex, diffusion, wave, and inclination . Video generation that generates an output image pixel by pixel using an advection equation including at least one of a diffusion term, a development term, a decay term and a stagnation term and an advection term based on the input image as a single still image In the device
An image input unit for inputting the input image;
A presentation unit for presenting a plurality of types of speed field patterns prepared in advance to the user ;
Two or more types of velocity field patterns selected by the user from among the plurality of velocity field patterns are superimposed and displayed with symbols indicating directions on different partial areas over the entire display area of the input image, and the selected A velocity that calculates a velocity per pixel and generates a velocity field that represents the velocity per pixel based on the velocity field pattern and the magnitude and direction of velocity entered by the user via the graphical user interface. The field editing department,
A continuous unit that generates a continuous velocity field by applying a moving average filter to the generated velocity field ;
An image generating unit that generates the output image by applying the advection equation to the input image based on the continuous velocity field;
A video generation apparatus having The video generation device according to claim 7 , wherein the image generation unit generates the output image as a time-series image composed of a plurality of still images continuous in time series. The video generation device according to claim 7 or 8 , wherein the diffusion term is a term expressing at least one of isotropic diffusion and anisotropic diffusion. The video generation device according to claim 7 , wherein the velocity field pattern prepared in advance represents a form of vortex, diffusion, wave, and inclination . A video generation program for causing a computer to execute the video generation method according to claim 1 . A recording medium readable by a computer, on which the video generation program according to claim 11 is recorded.

Images