JP2004173013A - Motion vector detector and detection method, image signal processor using the method and processing method, coeeficient data generator for use in the processor and generation method, and program for implementing these methods - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To quickly and accurately detect a motion vector at a specified position in a first frame with motion vectors existing between this frame and a second frame every block. <P>SOLUTION: A selector circuit 21 selects a motion compensating vector MI in four macroblocks located around a specified position as a motion vector on an upper hierarchical layer. A reference block circuit 23 reads pixel data of a reference block with the center at a specified position. A search block circuit 24 reads pixel data corresponding to candidate blocks one after another within a search range. The search range is a range with a position to which a position corresponding to the specified position moves based on the motion vector in the upper hierarchical layer as a reference position. An evaluation value calculating circuit 25 obtains a difference absolute value sum between the reference block and each candidate block as an evaluation value. A motion vector detector circuit 26 takes the position information of the candidate block giving a minimum evaluation value as a motion vector MV at a specified position. <P>COPYRIGHT: (C)2004,JPO

Description

【０１１４】
この第１の処理部１２１の動作を説明する。
バッファメモリ１０８に記憶されているＩピクチャの画像信号Ｖａ（Ｉ）に基づいて、クラスタップ選択回路１３２で、作成すべきＩピクチャの画像信号Ｖｂ（Ｉ）における注目位置に対して空間方向の周辺に位置するクラスタップの画素データが選択的に取り出される。このクラスタップの画素データはクラス生成回路１３３に供給される。このクラス生成回路１３３では、クラスタップの画素データに１ビットのＡＤＲＣ等の処理が施されて、空間クラスを示すクラスコードＣＬ１が生成される。
【０１１５】
このクラスコードＣＬ１は、画像信号Ｖｂ（Ｉ）における注目位置の画素データが属するクラスの検出結果を表している。このクラスコードＣＬ１は、係数メモリ１３４に読み出しアドレス情報として供給される。この係数メモリ１３４からクラスコードＣＬ１に対応した係数データＷｉが読み出されて、推定予測演算回路１３５に供給される。
【０１１６】
また、バッファメモリ１０８に記憶されているＩピクチャの画像信号Ｖａ（Ｉ）に基づいて、予測タップ選択回路１３１で、作成すべきＩピクチャの画像信号Ｖｂ（Ｉ）における注目位置に対して空間方向の周辺に位置する予測タップの画素データが選択的に取り出される。
【０１１７】
推定予測演算回路１３５では、予測タップの画素データｘｉと、係数メモリ１３４より読み出される係数データＷｉとを用いて、上述の（１）式に示す推定式に基づいて、作成すべきＩピクチャの画像信号Ｖｂ（Ｉ）における注目位置の画素データｙが求められる。
【０１１８】
このように第１の処理部１２１では、Ｉピクチャの画像信号Ｖａ（Ｉ）から係数データＷｉを用いてＩピクチャの画像信号Ｖｂ（Ｉ）が得られる。この場合、画像信号Ｖａ（Ｉ）に基づいて選択された、画像信号Ｖｂ（Ｉ）における注目位置の周辺に位置する複数の画素データ（予測タップの画素データ）、およびこの画像信号Ｖｂ（Ｉ）における注目位置の画素データが属するクラスＣＬ１に対応した係数データＷｉを用いて、推定式に基づいて画像信号Ｖｂ（Ｉ）における注目位置の画素データｙを生成するものである。
【０１１９】
したがって、係数データＷｉとして、画像信号Ｖａ（Ｉ）に対応しこの画像信号Ｖａ（Ｉ）と同様の符号化雑音を含む生徒信号と画像信号Ｖｂ（Ｉ）に対応した符号化雑音を含まない教師信号とを用いた学習によって得られた係数データＷｉを用いることで、画像信号Ｖｂ（Ｉ）として画像信号Ｖａ（Ｉ）に比べて符号化雑音が軽減されたものを得ることができる。
【０１２０】
図４は、第２の処理部１２２の構成を示している。この第２の処理部１２２は、予測に使用する予測タップの複数の画素データを選択的に取り出す予測タップ選択回路１４１と、クラス分類に使用するクラスタップの複数の画素データを選択的に取り出すクラスタップ選択回路１４２とを有している。
【０１２１】
タップ選択回路１４１，１４２には、システムコントローラ１０１から、ＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置に対応した画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）の画素データと対となっているピクチャ情報ＰＩと、後述する動きベクトル検出部１４３で検出された当該画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）の画素データが位置する所定位置の動きベクトルＭＶとが、動作制御情報として供給される。予測タップおよびクラスタップの画素データは、それぞれ、画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置に対して空間方向（水平方向、垂直方向）および時間方向（フレーム方向）の周辺に位置する画素データである。
【０１２２】
タップ選択回路１４１，１４２は、それぞれ、バッファメモリ１０８に格納されているＰピクチャまたはＢピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）、およびバッファメモリ１２３に格納されている、そのピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）を得る際に使用されたＩピクチャ、Ｐピクチャの画像信号Ｖａ（Ｉ），Ｖａ（Ｐ）に対応した、既に符号化雑音が軽減されたＩピクチャ、Ｐピクチャの画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ）に基づいて、複数の画素データを選択的に取り出す。
【０１２３】
ここで、タップ選択回路１４１，１４２は、動き補償用ベクトルＭＶに基づき、画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ）に対して、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）を基準とした動き補償を行う。
【０１２４】
図５は、動き補償の一例を示している。この例は、Ｐピクチャの画像信号Ｖｂ（Ｐ）における注目位置に対して空間方向および時間方向の周辺に位置する画素データを取り出す場合であって、Ｐピクチャの画像信号Ｖａ（Ｐ）およびＩピクチャの画像信号Ｖｂ（Ｉ）が、画素データの選択対象である場合を示している。
【０１２５】
この場合、画像信号Ｖｂ（Ｉ）に対して、画像信号Ｖａ（Ｐ）を基準とした動き補償が行われる。ここで、図５に示すように、動きベクトルＭＶが（−２，＋１）であったとき、タップ選択回路は、画像信号Ｖａ（Ｐ）からｘ_１〜ｘ_１３の画素データを選択的に取り出し、画像信号Ｖｂ（Ｉ）から（−２，＋１）だけずれた位置にあるｘ_１′〜ｘ_１３′の画素データを選択的に取り出す。
【０１２６】
このように、予測タップ選択回路１４１においては、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）の他に動き補償が行われた符号化雑音軽減後の画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ）からも予測タップの画素データを選択的に取り出すものである。したがって、この予測タップの画素データを用いて後述するように生成される画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データの品質を高めることができる。
【０１２７】
また、クラスタップ選択回路１４２において、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）の他に動き補償が行われた符号化雑音軽減後の画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ）からもクラスタップの画素データを選択的に取り出すものである。したがって、上述したように予測タップ選択回路１４１で動き補償が行われて選択された予測タップの画素データに対応した時空間クラスを精度よく検出できる。
【０１２８】
また、第２の処理部１２２は、動きベクトル検出部１４３を有している。この動きベクトル検出部１４３は、バッファメモリ１０８よりシステムコントローラ１０１を介して供給される動き補償用ベクトルＭＩを用いて、ＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置に対応した画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）の画素データが位置する所定位置（以下、「注目画素データ位置」という）の動きベクトルＭＶを検出する。
【０１２９】
この動きベクトル検出部１４３では、ブロックマッチング法により動きベクトルが検出される。これは、図６に示すように、探索フレームの候補ブロックを所定の探索範囲内で移動し、参照フレームの参照ブロックと最も合致している候補ブロックを検出することにより、動きベクトルを求めるものである。
【０１３０】
ブロックマッチング法では、図７Ａに示すように、１枚の画像、例えば水平Ｈ画素、垂直Ｖラインの１フレームの画像が図７Ｂに示すように、Ｐ画素×Ｑラインのブロックに細分化される。図７Ｂの例では、Ｐ＝５、Ｑ＝５の例である。ｃがブロックの中心画素位置である。
【０１３１】
図８Ａ〜Ｃは、ｃを中心画素とする参照ブロックとｃ´を中心とする候補ブロックの位置関係を示している。ｃを中心画素とする参照ブロックは、参照フレームの注目しているある参照ブロックであり、それと一致する探索フレームの候補ブロックが探索フレームにおいてｃ´を中心とするブロックの位置にあるものとしている。ブロックマッチング法では、探索範囲内において、参照ブロックと最も合致する候補ブロックを見出すことによって、動きベクトルを検出する。
【０１３２】
図８Ａの場合では、水平方向に＋１画素、垂直方向に＋１ライン、すなわち、（＋１，＋１）の動きベクトルが検出される。図８Ｂでは、（＋３，＋３）の動きベクトルＭＶが検出され、図８Ｃでは、（＋２，−１）の動きベクトルが検出される。動きベクトルは、参照フレームの参照ブロック毎に求められる。
【０１３３】
動きベクトルを探索する範囲を水平方向で±Ｓ画素、垂直方向で±Ｔラインとすると、参照ブロックは、その中心ｃに対して、水平に±Ｓ、垂直に±Ｔずれたところに中心ｃ´を有する候補ブロックと比較される必要がある。
【０１３４】
図９は、参照フレームのある参照ブロックの中心ｃの位置をＲとする時に、比較すべき探索フレームの（２Ｓ＋１）×（２Ｔ＋１）個の候補ブロックとの比較が必要なことを示している。すなわち、この図９のます目の位置にｃ´が存在する候補ブロックの全てが比較対象である。図９は、Ｓ＝４，Ｔ＝３とした例である。
【０１３５】
探索範囲内の比較で得られた評価値（すなわち、フレーム差の絶対値和、このフレーム差の二乗和、あるいはフレーム差の絶対値のｎ乗和等）の中で、最小値を検出することによって、動きベクトルが検出される。図９の探索範囲は、候補ブロックの中心が位置する領域であり、候補ブロックの全体が含まれる探索範囲の大きさは、（２Ｓ＋Ｐ）×（２Ｔ＋Ｑ）となる。
【０１３６】
また、動きベクトル検出部１４３では、上位階層および下位階層の２階層で動きベクトルＭＶを検出する。この場合、動きベクトル検出部１４３は、注目画素データ位置の周辺に位置する複数のマクロブロックの動き補償用ベクトルＭＩを、上位階層の動きベクトルとして用いる。ここで、注目画素データ位置の周辺に位置する複数のマクロブロックには、この注目画素データ位置を含むマクロブロックを含み、また周辺は空間方向（水平方向および垂直方向）および時間方向（フレーム方向）のいずれかまたは双方の周辺である。
【０１３７】
本実施の形態においては、例えば空間方向の周辺に位置する４個のマクロブロックの動きベクトルＭＩを上位階層の動きベクトルとして用いる。図１０は、注目画素データ位置を含むマクロブロックをＭＢ０とし、このマクロブロックＭＢ０に隣接する８個のマクロブロックをＭＢ１〜ＭＢ８としている。周知のように、マクロブロックの大きさは１６×１６画素であると共に、このマクロブロックは４個の４×４画素のブロックで構成されている。
【０１３８】
注目画素データ位置がａのブロックに含まれるとき、マクロブロックＭＢ０，ＭＢ１，ＭＢ２，ＭＢ８の動き補償用ベクトルＭＩが上位階層の動きベクトルとして選択される。同様に、注目画素データ位置がｂ，ｃ，ｄのブロックに含まれるとき、それぞれ［ＭＢ０，ＭＢ２，ＭＢ３，ＭＢ４］，［ＭＢ０，ＭＢ４，ＭＢ５，ＭＢ６］，［ＭＢ０，ＭＢ６，ＭＢ７，ＭＢ８］の動き補償用ベクトルＭＩが上位階層の動きベクトルとして選択される。
【０１３９】
このように、注目画素データ位置を含むマクロブロックＭＢ０の他にこのマクロブロックＭＢ０に隣接する３個のマクロブロックの動き補償用ベクトルＭＩを上位階層の動きベクトルとして選択するのは、マクロブロックＭＢ０に実際には複数の動きベクトルが存在する場合、注目画素データ位置によっては、マクロブロックＭＢ０の動き補償用ベクトルより、このマクロブロックＭＢ０に隣接するマクロブロックの動き補償用ベクトルの方が、当該注目画素データ位置に適合したものとなる場合があるからである。
【０１４０】
例えば、図１１に示すように、動き領域Ｍ１，Ｍ２が存在する場合（矢印は動き方向を示している）、マクロブロックＭＢ０内の注目画素データ位置がＱであるときには、マクロブロックＭＢ０の動き補償用ベクトルＭＩよりマクロブロックＭＢ１の動き補償用ベクトルＭＩの方が、当該注目画素データ位置に適合したものとなる。
【０１４１】
動きベクトル検出部１４３では、上述したように選択された上位階層の複数の動きベクトルを用いて、下位階層でブロックマッチング法による動きベクトル検出を行う。ブロックマッチングの評価関数は、参照フレームをＴで表し、探索フレームをＴ−１で表すと、（２）式で求められる。ただし、（ｕ_ｎ，ｖ_ｎ）は、下位階層ｎでの動きベクトル、（ｕ_ｎ＋１，ｖ_ｎ＋１）は上位階層での動きベクトルを示している。
【０１４２】
【数２】

【０１４３】
この評価関数Ｅ（Ｙ）_ｎで得られる相関情報としての評価値の最小を与える（ｕ_ｎ，ｖ_ｎ）を、下位階層の動きベクトルＶ０とする。最終的な動きベクトルＭＶは、（３）式により求められる。ここで、Ｖ１は、評価値の最小が得られた際に使用された上位階層の動きベクトルである。
ＭＶ＝Ｖ０＋Ｖ１ ・・・（３）
【０１４４】
図１２のフローチャートは、動きベクトルＭＶの検出処理の手順を示している。
まず、ステップＳＴ１で、処理を開始し、ステップＳＴ２で、Ｎ＝１に設定する。そして、ステップＳＴ３で、上述したように上位階層の動きベクトルとして選択された複数の動きベクトルＭＩのうち、Ｎ番目の動きベクトルＭＩを上位階層の動きベクトルとして、下位階層の探索を開始する。
【０１４５】
この場合、ＰピクチャまたはＢピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）の注目画素データ位置が存在するフレームが参照フレームとされ、そのピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）を得る際に使用されたＩピクチャ、Ｐピクチャの画像信号Ｖａ（Ｉ），Ｖａ（Ｐ）のフレームが探索フレームとされる。
【０１４６】
そして、参照フレーム（Ｔフレーム）に、注目画素データ位置に対応した画素データを中心とした参照ブロックを設定し、探索フレーム（Ｔ−１フレーム）の探索範囲内での探索を開始する。ここで、探索フレームは、上位階層の動きベクトルによって動き補償されている。つまり、探索フレームの探索範囲は、注目位置（注目画素データ位置）に対応した位置が上位階層の動きベクトルに基づいて移動された位置を基準位置とした範囲とされる。
【０１４７】
次に、ステップＳＴ４で、探索範囲内での全ての探索が終了したか否かを判定する。探索が終了していないときは、ステップＳＴ５に進み、画素データを用いて、（２）式に基づいて、評価関数Ｅ（Ｙ）_ｎによる相関情報としての評価値を求め、ステップＳＴ４に戻る。一方、探索が終了したときは、ステップＳＴ６に進む。
【０１４８】
ステップＳＴ６では、Ｎ＝Ｎｍａｘであるか否かを判定する。Ｎｍａｘは、上位階層の動きベクトルとして選択された複数の動きベクトルの個数を示している。本実施の形態においては、Ｎｍａｘ＝４である。Ｎ＝Ｎｍａｘでないときは、ステップＳＴ７でＮをインクリメントし、その後にステップＳＴ３に戻って、次の上位階層の動きベクトルを用いて、下位階層の探索を行う。一方、Ｎ＝Ｎｍａｘであるときは、ステップＳＴ８に進む。
【０１４９】
ステップＳＴ８では、相関情報としての評価値より動きベクトルＶ０を決定する。つまり、最小の評価値を与える（ｕ_ｎ，ｖ_ｎ）を動きベクトルＶ０とする。そして、ステップＳＴ９で、（３）式に基づいて、下位階層の動きベクトルＶ０に、それに対応する上位階層の動きベクトルＶ１を加算して最終的な動きベクトルＭＶを算出する。この場合、参照ブロックと最も相関の高い候補ブロックの位置情報を、注目画素データ位置の動きベクトルＭＶとしたことになる。ステップＳＴ９の処理の後に、ステップＳＴ１０で、処理を終了する。
【０１５０】
図１３は、動きベクトル検出部１４３の構成を示している。
動きベクトル検出部１４３は、注目画素データ位置の周辺に位置する４個のマクロブロックの動き補償用ベクトルＭＩを、上位階層の動きベクトルとして選択する動きベクトル選択回路２１と、この選択回路２１で選択された４個の動きベクトルＭＩ_１〜ＭＩ_４を蓄積しておくための動きベクトル蓄積部２２とを有している。動きベクトル選択回路２１には、注目画素データ位置の位置情報が、動作制御情報として供給される。
【０１５１】
また、動きベクトル検出部１４３は、バッファメモリ１０８（図８参照）に格納されている参照フレームから、注目画素データ位置を中心とした参照ブロックの画素データを選択的に読み出す参照ブロック回路２３と、バッファメモリ１０８に格納されている探索フレームから、探索範囲内の各候補ブロックに対応した画像データを順次読み出すサーチブロック回路２４とを有している。
【０１５２】
ここで、参照フレームは、ＰピクチャまたはＢピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）の注目画素データ位置が存在するフレームである。また、探索フレームは、そのそのピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）を得る際に使用されたＩピクチャ、Ｐピクチャの画像信号Ｖａ（Ｉ），Ｖａ（Ｐ）のフレームである。
【０１５３】
ここで、探索フレームの探索範囲は、注目画素データ位置に対応した位置が上位階層の動きベクトルに基づいて移動された位置を基準位置とした範囲とされる。サーチブロック回路２４は、上位階層の動きベクトルとして、動きベクトル蓄積部２２に蓄積された４個の動きベクトルＭＩ_１〜ＭＩ_４を順次使用していく。そして、サーチブロック回路２４は、各上位階層の動きベクトルにより決定された探索範囲で、それぞれ複数の候補ブロックの画素データを順次読み出す。
【０１５４】
また、動きベクトル検出部１４３は、参照ブロック回路２３からの参照ブロックの画素データと、サーチブロック回路２４からの各候補ブロックの画像データとを用いて、上述した（２）式の評価関数Ｅ（Ｙ）_ｎに基づいて、各候補ブロックの評価値を求める評価値算出回路２５を有している。ここで、評価値は参照ブロックと候補ブロックとの相関の度合いを示す相関情報であり、評価値算出回路２５は相関検出手段を構成している。
【０１５５】
なお、サーチブロック回路２４から評価値算出回路２５には、候補ブロックの画素データと共に、その候補ブロックが存在する探索範囲を決定した上位階層の動きベクトルも供給される。評価値算出回路２５は、算出した各候補ブロックの評価値を、上位階層の動きベクトル別に評価値メモリに記憶する。
【０１５６】
また、動きベクトル検出部１４３は、動きベクトル検出回路２６を有している。この動きベクトル検出回路２６は、評価値算出回路２５で算出された各候補ブロックの評価値を順次評価し、最小の評価値を与える候補ブロックに対応した動きベクトル（ｕ_ｎ、ｖ_ｎ）を、下位階層の動きベクトルＶ０とする。そして、最終的な動きベクトルＭＶを、上述した（３）式により求める。
【０１５７】
つまり、下位階層の動きベクトルＶ０に上位階層の動きベクトルＶ１を加算して、動きベクトルＭＶとする。この動きベクトルＶ１は、評価値の最小が得られた候補ブロックが存在する探索範囲を決定した上位階層の動きベクトルである。上述したように各候補ブロックの評価値は上位階層の動きベクトル別に評価値メモリに記憶されるので、この動きベクトルＶ１の情報は評価値メモリにおける最小の評価値の記憶位置情報から容易に得ることができる。
【０１５８】
この動きベクトル検出回路２６は、動きベクトルＭＶの他に、下位階層の動きベクトルＶ０および最小の評価値ＥＹｍｉｎも出力する。動きベクトルＭＶは、上述したように、予測タップおよびクラスタップの画素データを選択する際の動き補償に使用されるが、動きベクトルＶ０および評価値ＥＹｍｉｎは後述するようにクラス分類のために使用される。
【０１５９】
図１４は、評価値算出回路２５の構成を示している。
参照ブロックメモリ４１には、参照ブロック回路２３（図１３参照）で選択的に読み出された参照ブロックの画素データが格納される。候補ブロックメモリ４２には、サーチブロック回路２４（図１３参照）で選択的に読み出された候補ブロックの画素データが格納される。
【０１６０】
参照ブロックメモリ４１および候補ブロックメモリ４２の内容が、メモリコントローラ４３で指定されたアドレスの順に読み出され、それぞれレジスタ４４およびレジスタ４５を通じて減算回路４６で減算される。この結果得られる差分データは絶対値化回路４７で絶対値化され、加算回路４８およびレジスタ４９で累積加算される。この累積加算結果は、当該候補ブロックの評価値となる。
【０１６１】
評価値メモリコントローラ５１には、また、当該候補ブロックが存在する探索範囲を決定した上位階層の動きベクトルＭＩ_１〜ＭＩ_４も供給される。評価値メモリ５０には、上位階層の動きベクトルＭＩ_１〜ＭＩ_４でそれぞれ決定される探索範囲内の各候補ブロックの評価値が、上位階層の動きベクトル別に、下位階層の動きベクトル（ｕ_ｎ，ｖ_ｎ）（（２）式参照）の順に記憶される。
【０１６２】
この場合、ある候補ブロックの評価値は、その候補ブロックの下位階層の動きベクトル（ｕ_ｎ，ｖ_ｎ）を下位アドレスとし、その候補ブロックの上位階層の動きベクトル（ｕ_ｎ＋１，ｖ_ｎ＋１）に対応した上位アドレスを持つ、評価値メモリ５０のアドレスに記憶される。
【０１６３】
図１５は、動きベクトル検出回路２６の構成を示している。
評価値メモリ５０から、評価値メモリコントローラ５１の制御によって、上述したアドレス（上位アドレスおよび下位アドレスからなる）の順に、評価値が読み出される。この場合、評価値メモリコントローラ５１からは、下位アドレスである下位階層の動きベクトル（ｕ_ｎ，ｖ_ｎ）と、この下位階層の動きベクトルおよび上位アドレスに対応した上位階層の動きベクトル（ｕ_ｎ＋１，ｖ_ｎ＋１）を加算した動きベクトル（ｕ，ｖ）とが出力される。
【０１６４】
評価値メモリ５０から読み出された評価値は、比較器６１およびレジスタ６２に供給される。また、評価値メモリコントローラ５１より出力される動きベクトル（ｕ，ｖ）および下位階層の動きベクトル（ｕ_ｎ，ｖ_ｎ）は、それぞれレジスタ６３およびレジスタ６４に供給される。
【０１６５】
比較器６１は他方の入力と評価値メモリ５０より読み出された評価値を順次比較し、このうち評価値メモリ５０より読み出された評価値が小さいとき、レジスタ６２〜６４の内容を更新する信号を出力する。レジスタ６２には、最終的に最小の評価値ＥＹｍｉｎが保持される。また、レジスタ６３には、最小の評価値ＥＹｍｉｎを与える候補ブロックに対応した動きベクトル（ｕ，ｖ）が、動きベクトルＭＶとして保持される。さらに、レジスタ６４には、最小の評価値ＥＹｍｉｎを与える候補ブロックに対応した下位階層の動きベクトル（ｕ_ｎ，ｖ_ｎ）が、動きベクトルＶ０として保持される。
【０１６６】
図１３に示す動きベクトル検出部１４３の動作を説明する。
動きベクトル選択回路２１は、注目画素データ位置の位置情報に基づいて、注目画素データ位置の周辺に位置する４個のマクロブロックの動き補償用ベクトルＭＩを、上位階層の動きベクトルとして選択する。このように選択された４個の動きベクトルＭＩ_１〜ＭＩ_４は動きベクトル蓄積部２２に供給されて蓄積される。
【０１６７】
参照ブロック回路２３は、バッファメモリ１０８（図８参照）に格納されている参照フレームから、注目画素データ位置を中心とした参照ブロックの画素データを選択的に読み出す。
【０１６８】
また、サーチブロック回路２４は、バッファメモリ１０８に格納されている探索フレームから、探索範囲内の各候補ブロックに対応した画像データを順次読み出す。この場合、探索フレームの探索範囲は、注目画素データ位置に対応した位置が上位階層の動きベクトルに基づいて移動された位置を基準位置とした範囲である。サーチブロック回路２４は、上位階層の動きベクトルとして動きベクトル蓄積部２２に蓄積された４個の動きベクトルＭＩ_１〜ＭＩ_４を順次使用し、各上位階層の動きベクトルにより決定された探索範囲でそれぞれ複数の候補ブロックの画素データを順次読み出す。
【０１６９】
参照ブロック回路２３で読み出された参照ブロックの画素データと、サーチブロック回路２４で読み出された各候補ブロックの画像データは評価値算出回路２５に供給される。評価値算出回路２５は、上述した（２）式の評価関数Ｅ（Ｙ）_ｎに基づいて、各候補ブロックの評価値を求める。算出された各候補ブロックの評価値は、上位階層の動きベクトル別に、評価値メモリ５０（図１４参照）に記憶される。
【０１７０】
動きベクトル検出回路２６は、評価値算出回路２５で算出された各候補ブロックの評価値を順次評価し、最小の評価値を与える候補ブロック、すなわち参照ブロックと最も相関の高い候補ブロックに対応した動きベクトル（ｕ_ｎ、ｖ_ｎ）を、下位階層の動きベクトルＶ０とする。そして、この下位階層の動きベクトルＶ０に、その候補ブロックの上位階層の動きベクトルＶ１を加算してなる動きベクトル（ｕ，ｖ）を、最終的な動きベクトルＭＶとして出力する。なお、動きベクトル検出回路２６は、動きベクトルＭＶを出力する他に、動きベクトルＶ０および評価値ＥＹｍｉｎも出力する。
【０１７１】
このように、動きベクトル検出部１４３では、マクロブロック毎に有する動き補償用ベクトルＭＩを上位階層の動きベクトルとして使用して、注目画素データ位置の動きベクトルＭＶを階層的に検出するものであり、動きベクトルＭＶを素早く検出できる。また、動きベクトル検出部１４３では、注目画素データ位置が存在するマクロブロックの動き補償用ベクトルＭＩだけでなく、そのマクロブロックの周辺のマクロブロックの動き補償用ベクトルＭＩをも上位階層の動きベクトルとして使用するものであり、注目画素データ位置の動きベクトルＭＶを精度よく検出できる。
【０１７２】
なお、図１３に示す動きベクトル検出部１４３においては、バッファメモリ１０８に格納されている画像信号Ｖａから参照フレームおよび探索フレームの双方を取得するものであるが、その画像信号Ｖａから参照フレームを取得すると共にバッファメモリ１２３に格納されている画像信号Ｖｂから探索フレームを取得するようにしてもよい。
【０１７３】
また、図１３に示す動きベクトル検出部１４３においては、注目画素データ位置の周辺に位置するマクロブロックの動き補償用ベクトルＭＩを上位階層の動きベクトルとして選択するものであるが、さらに注目画素データ位置を含むフレームの各マクロブロックの動き補償用ベクトルＭＩの度数分布を生成し、度数の高い方から所定数の動きベクトルも上位階層の動きベクトルとして選択するようにしてもよい。
【０１７４】
図１６は、その場合における動きベクトル検出部１４３の要部を示している。その場合、動きベクトル検出部１４３は、注目画素データ位置を含むフレームの各マクロブロックの動き補償用ベクトルＭＩの度数分布を生成する度数分布生成回路２７と、その度数分布に基づいて、度数の高い方から所定数の動きベクトルを上位階層の動きベクトルとして選択する動きベクトル選択回路２８を備えることになる。そして、選択回路２８で選択された所定数の動きベクトルＭＩ_ａ〜ＭＩ_ｎは動きベクトル蓄積部２２に供給されて蓄積され、動きベクトル選択回路２１で選択されたＭＩ_１〜ＭＩ_４と同様に使用される。
【０１７５】
注目画素データ位置の周辺に位置する複数のマクロブロックの動き補償用ベクトルＭＩに、注目画素データ位置に適合した動きベクトルが存在しないことも考えられる。その場合、上述したように度数分布に基づいて選択された動きベクトルの中に注目画素データ位置に適合した動きベクトルがある場合も想定される。したがって、上述したように度数分布に基づいて選択された動き補償用ベクトルＭＩを上位階層の動きベクトルに加えることで、注目画素データ位置の動きベクトルＭＶの検出精度をさらに高めることができる。
【０１７６】
図４に戻って、また、第２の処理部１２２は、クラス生成回路１４４，１４５を有している。クラス生成回路１４４は、クラスタップ選択回路１４２で選択的に取り出されるクラスタップの画素データから時空間クラスを示すクラスコードＣＬａを生成する。このクラス生成回路１４４は、上述した第１の処理部１２１（図３参照）のクラス生成回路１３３と同様に、クラスタップの画素データに１ビットのＡＤＲＣ等の処理を施すことによって、時空間クラスを示すクラスコードＣＬａを生成する。
【０１７７】
クラス生成回路１４５は、動きベクトル検出部１４３から出力される、注目画素データ位置の動きベクトルＭＶに関連した動きベクトルＶ０および評価値ＥＹｍｉｎをそれぞれ閾値判定して、動き関連クラスを示すクラスコードＣＬｂを生成する。
【０１７８】
また、第２の処理部１２２は、クラス生成回路１４４，１４５で生成されるクラスコードＣＬａ，ＣＬｂを統合して１個のクラスコードＣＬ２とするクラス統合回路１４６を有している。このクラスコードＣＬ２は、ＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データが属するクラスを示すものとなる。
【０１７９】
また、第２の処理部１２２は、係数メモリ１４７を有している。この係数メモリ１４７は、後述する推定予測演算回路１４８で使用される推定式で用いられる係数データＷｉ（ｉ＝１〜ｎ、ｎは予測タップの個数）を、クラス毎に、格納するものである。
【０１８０】
この係数メモリ１４７には、Ｐピクチャの画像信号Ｖａ（Ｐ）をＰピクチャの画像信号Ｖｂ（Ｐ）に変換するためのＰピクチャ用の係数データＷｉと、Ｂピクチャの画像信号Ｖａ（Ｂ）をＢピクチャの画像信号Ｖｂ（Ｂ）に変換するためのＢピクチャ用の係数データＷｉとが別個のアドレスに格納されている。この係数メモリ１４７には、システムコントローラ１０１から、ピクチャ情報ＰＩが読み出し制御情報として供給される。
【０１８１】
この係数メモリ１４７に格納される係数データＷｉは、予め画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）に対応した生徒信号と画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）に対応した教師信号との間の学習によって生成される。この係数メモリ１４７には上述したクラス統合回路１４６で得られたクラスコードＣＬ２が読み出しアドレス情報として供給される。この係数メモリ１４７からはクラスコードＣＬ２に対応した推定式の係数データＷｉが読み出されて、推定予測演算回路１４８に供給される。係数データＷｉの生成方法については後述する。
【０１８２】
また、第２の処理部１２２は、予測タップ選択回路１４１で選択的に取り出される予測タップの画素データｘｉと、係数メモリ１４７より読み出される係数データＷｉとから、上述の（１）式の推定式によって、作成すべき画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データｙを演算する推定予測演算回路１４８を有している。
【０１８３】
この第２の処理部１２２の動作を説明する。
バッファメモリ１０８に格納されているＰピクチャまたはＢピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）、およびバッファメモリ１２３に格納されているその画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）を得る際に使用されたＩピクチャ、Ｐピクチャの画像信号Ｖａ（Ｉ），Ｖａ（Ｐ）に対応した、Ｉピクチャ、Ｐピクチャの画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ）に基づいて、クラスタップ選択回路１４２で、作成すべきＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置に対して空間方向および時間方向の周辺に位置するクラスタップの画素データが選択的に取り出される。
【０１８４】
この場合、動きベクトル検出部１４３で検出された注目画素データ位置の動きベクトルＭＶに基づき、画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ）に対して、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）を基準とした動き補償が行われる。なお、注目画素データ位置とは、上述したように、ＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置に対応した画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）の画素データが位置する所定位置をいう。
【０１８５】
クラスタップ選択回路１４２で取り出されるクラスタップの画素データはクラス生成回路１４４に供給される。このクラス生成回路１４４は、クラスタップの画素データに１ビットのＡＤＲＣ等の処理が施されて、時空間クラスを示すクラスコードＣＬａを生成する。
【０１８６】
動きベクトル検出部１４３より出力される、動きベクトルＭＶに関連した動きベクトルＶ０および評価値ＥＹｍｉｎはクラス生成回路１４５に供給される。このクラス生成回路１４５は、動きベクトルＶ０および評価値ＥＹｍｉｎをそれぞれ閾値判定して、動き関連クラスを示すクラスコードＣＬｂを生成する。
【０１８７】
そして、クラス生成回路１４４，１４５で生成されるクラスコードＣＬａ，ＣＬｂはクラス統合回路１４６で統合され、ＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データが属するクラスを示す、１個のクラスコードＣＬ２が得られる。このクラスコードＣＬ２は、係数メモリ１４７に読み出しアドレス情報として供給される。
【０１８８】
この係数メモリ１４７からクラスコードＣＬ２に対応した係数データＷｉが読み出されて、推定予測演算回路１４８に供給される。この場合、ピクチャ情報ＰＩに基づいて、Ｐピクチャの画像信号Ｖｂ（Ｐ）を作成する場合にはＰピクチャ用の係数データＷｉが読み出され、Ｂピクチャの画像信号Ｖｂ（Ｂ）を作成する場合にはＢピクチャ用の係数データＷｉが読み出される。
【０１８９】
また、バッファメモリ１０８に格納されているＰピクチャまたはＢピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）、およびバッファメモリ１２３に格納されているその画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）を得る際に使用されたＩピクチャ、Ｐピクチャの画像信号Ｖａ（Ｉ），Ｖａ（Ｐ）に対応した、Ｉピクチャ、Ｐピクチャの画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ）に基づいて、予測タップ選択回路１４１で、作成すべきＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置に対して空間方向および時間方向の周辺に位置する予測タップの画素データが選択的に取り出される。この場合、動きベクトル検出部１４３で検出された注目画素データ位置の動きベクトルＭＶに基づき、画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ）に対して、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）を基準とした動き補償が行われる。
【０１９０】
推定予測演算回路１４５では、予測タップの画素データｘｉと、係数メモリ１４７より読み出される係数データＷｉとを用いて、上述の（１）式に示す推定式に基づいて、作成すべきＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｉ）における注目位置の画素データｙが求められる。
【０１９１】
このように第２の処理部１２２では、ＰピクチャまたはＢピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）から、係数データＷｉを用いて、ＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）が得られる。この場合、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）およびＶｂ（Ｉ）、Ｖｂ（Ｐ）に基づいて選択された、画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の周辺に位置する複数の画素データ（予測タップの画素データ）、およびこの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データが属するクラスＣＬ２に対応した係数データＷｉを用いて、推定式に基づいて画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データｙを生成するものである。
【０１９２】
したがって、係数データＷｉとして、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）に対応しこの画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）と同様の符号化雑音を含む生徒信号と画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）に対応した符号化雑音を含まない教師信号とを用いた学習によって得られた係数データＷｉを用いることで、画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）として画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）に比べて符号化雑音が軽減されたものを得ることができる。
【０１９３】
また、予測タップの画素データは、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）から選択される他に、動きベクトル検出部１４３で精度よく検出された注目画素データ位置の動きベクトルＭＶにより動き補償された、符号化雑音軽減後の画像信号Ｖｂ（Ｉ）、Ｖｂ（Ｐ）からも選択されるものであり、画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）の品質の向上を図ることができ、それに含まれる符号化雑音を良好に軽減できる。
【０１９４】
また、クラスタップの画素データは、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）から選択される他に、動きベクトル検出部１４３で精度よく検出された注目画素データ位置の動きベクトルＭＶにより動き補償された、符号化雑音軽減後の画像信号Ｖｂ（Ｉ）、Ｖｂ（Ｐ）からも選択されるものであり、クラス生成回路１４４で生成されるクラスコードＣＬａは、予測タップ選択回路１４１で動き補償が行われて選択された予測タップの画素データに対応した時空間クラスを精度よく示すものとなる。したがって、クラス分類の精度を高めることができ、ひいては画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）の品質の向上を図ることができる。
【０１９５】
また、動きベクトル検出部１４３より出力される、動きベクトルＭＶに関連した下位階層の動きベクトルＶ０および最小の評価値ＥＹｍｉｎを用いて、動き関連クラスを示すクラスコードＣＬｂが生成される。そして、ＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データが属するクラスを示すクラスコードＣＬ２として、このクラスコードＣＬｂを統合したものが使用される。このようにクラス分類情報として、符号化雑音軽減後の画像信号Ｖｂ（Ｉ）、Ｖｂ（Ｐ）を動き補償する動きベクトルＭＶに関連した情報を用いているので、的確なクラス分類を行うことができ、ひいては画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）の品質の向上を図ることができる。
【０１９６】
図１に戻って、また、画像信号処理部１１０は、第１の処理部１２１より出力されるＩピクチャの画像信号Ｖｂ（Ｉ）および第２の処理部１２２より出力されるＰピクチャ、Ｂピクチャの画像信号Ｖｂ（Ｐ），Ｖｂ（Ｂ）を、実際のフレーム／フィールドの順番に並べ換え、画像信号Ｖｂとして出力する並べ換え部１２４を有している。並べ換え部１２４は、バッファメモリ１２５を利用して並べ換え処理を行う。
【０１９７】
画像信号処理部１１０の動作を説明する。
バッファメモリ１０８に格納されている画像信号ＶａのうちＩピクチャの画像信号Ｖａ（Ｉ）は第１の処理部１２１に供給される。第１の処理部１２１は、このＩピクチャの画像信号Ｖａ（Ｉ）から、係数データＷｉを用いて、符号化雑音を軽減したＩピクチャの画像信号Ｖｂ（Ｉ）を生成する。
【０１９８】
この場合、画像信号Ｖａ（Ｉ）に基づいて選択された、画像信号Ｖｂ（Ｉ）における注目位置の周辺に位置する複数の画素データ（予測タップの画素データ）、およびこの画像信号Ｖｂ（Ｉ）における注目位置の画素データが属するクラスＣＬ１に対応した係数データＷｉを用いて、推定式に基づいて画像信号Ｖｂ（Ｉ）における注目位置の画素データｙを生成する。
【０１９９】
第１の処理部１２１で生成されたＩピクチャの画像信号Ｖｂ（Ｉ）は、並べ換え部１２４に供給されると共に、バッファメモリ１２３に供給されて一時的に格納される。このようにバッファメモリ１２３に格納されたＩピクチャの画像信号Ｖｂ（Ｉ）は、第２の処理部１２２で、Ｐピクチャの画像信号Ｖｂ（Ｐ）およびＢピクチャの画像信号Ｖｂ（Ｂ）を生成する際に用いられる。
【０２００】
また、バッファメモリ１０８に格納されている画像信号ＶａのうちＰピクチャまたはＢピクチャの画像信号Ｖａ（Ｐ）／Ｖａ（Ｉ）は第２の処理部１２２に供給される。第２の処理部１２２は、この画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）およびバッファメモリ１２３に格納されているＩピクチャ、Ｐピクチャの画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ）から、係数データＷｉを用いて、符号化雑音を軽減したＰピクチャまたはＢピクチャの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）を生成する。
【０２０１】
この場合、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）およびＶｂ（Ｉ）、Ｖｂ（Ｐ）に基づいて選択された、画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の周辺に位置する複数の画素データ（予測タップの画素データ）、およびこの画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データが属するクラスＣＬ２に対応した係数データＷｉを用いて、推定式に基づいて画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データｙを生成する。
【０２０２】
第２の処理部１２２で生成されたＰピクチャ、Ｂピクチャの画像信号Ｖｂ（Ｐ），Ｖｂ（Ｂ）は、並べ換え部１２４に供給される。また、第２の処理部１２２で生成されたＰピクチャの画像信号Ｖｂ（Ｐ）は、バッファメモリ１２３に供給されて一時的に格納される。このようにバッファメモリ１２３に格納されたＰピクチャの画像信号Ｖｂ（Ｐ）は、第２の処理部１２２で、Ｂピクチャの画像信号Ｖｂ（Ｂ）を生成する際に用いられる。
【０２０３】
並べ換え部１２４は、第１の処理部１２１より出力されるＩピクチャの画像信号Ｖｂ（Ｉ）および第２の処理部１２２より出力されるＰピクチャ、Ｂピクチャの画像信号Ｖｂ（Ｐ），Ｖｂ（Ｂ）を、符号化の順番から実際のフレーム／フィールドの順番に並べ換え、画像信号Ｖｂとして順次出力する。
【０２０４】
このように画像信号処理部１１０では、画像信号Ｖａから係数データＷｉを用いて画像信号Ｖｂが得られる。したがって、係数データＷｉとして、画像信号Ｖａに対応しこの画像信号Ｖａと同様の符号化雑音を含む生徒信号と画像信号Ｖｂに対応した符号化雑音を含まない教師信号とを用いた学習によって得られた係数データＷｉを用いることで、画像信号Ｖｂとして画像信号Ｖａに比べて符号化雑音が軽減されたものを得ることができる。
【０２０５】
また、Ｐピクチャ、Ｂピクチャの画像信号Ｖｂ（Ｐ），Ｖｂ（Ｂ）を得る際に、第２の処理部１２２では、予測タップの画素データを画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）から選択する他に、符号化雑音が軽減された画像信号Ｖｂ（Ｉ）、Ｖｂ（Ｐ）からも選択するものである。したがって、Ｐピクチャ、Ｂピクチャの画像信号Ｖｂ（Ｐ），Ｖｂ（Ｂ）の品質の向上を図ることができ、それらに含まれる符号化雑音を良好に軽減できる。
【０２０６】
また、第２の処理部１２２では、画像信号Ｖｂ（Ｉ）、Ｖｂ（Ｐ）に対して画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）を基準とした動き補償を行って予測タップの画素データを選択的に取り出すものである。したがって、画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）における注目位置の画素データを生成するための予測タップの画素データは相関が高いものとなり、画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）の品質を高めることができる。
【０２０７】
次に、第１の処理部１２１（図３参照）の係数メモリ１３４、第２の処理部１２２（図４参照）の係数メモリ１４７に記憶される係数データＷｉの生成方法について説明する。この係数データＷｉは、予め学習によって生成されたものである。
【０２０８】
まず、この学習方法について説明する。上述の、（１）式において、学習前は係数データＷ_１ ，Ｗ_２，‥‥，Ｗ_ｎ は未定係数である。学習は、クラス毎に、複数の信号データに対して行う。学習データ数がｍの場合、（１）式に従って、以下に示す（４）式が設定される。ｎは予測タップの数を示している。
ｙ_ｋ ＝Ｗ_１ ×ｘ_ｋ１＋Ｗ_２ ×ｘ_ｋ２＋‥‥＋Ｗ_ｎ ×ｘ_ｋｎ ・・・（４）
（ｋ＝１，２，‥‥，ｍ）
【０２０９】
ｍ＞ｎの場合、係数データＷ_１ ，Ｗ_２，‥‥，Ｗ_ｎは、一意に決まらないので、誤差ベクトルｅの要素ｅ_ｋを、以下の式（５）で定義して、（６）式のｅ^２を最小にする係数データを求める。いわゆる最小２乗法によって係数データを一意に定める。
ｅ_ｋ＝ｙ_ｋ−｛Ｗ_１×ｘ_ｋ１＋Ｗ_２×ｘ_ｋ２＋‥‥＋Ｗ_ｎ×ｘ_ｋｎ｝ ・・・（５）
（ｋ＝１，２，‥‥ｍ）
【０２１０】
【数３】

【０２１１】
（６）式のｅ^２を最小とする係数データを求めるための実際的な計算方法としては、まず、（７）式に示すように、ｅ^２を係数データＷｉ（ｉ＝１，２，・・・，ｎ）で偏微分し、ｉの各値について偏微分値が０となるように係数データＷｉを求めればよい。
【０２１２】
【数４】

【０２１３】
（７）式から係数データＷｉを求める具体的な手順について説明する。（８）式、（９）式のようにＸｊｉ，Ｙｉ を定義すると、（７）式は、（１０）式の行列式の形に書くことができる。
【０２１４】
【数５】

【０２１５】
【数６】

【０２１６】
（１０）式は、一般に正規方程式と呼ばれるものである。この正規方程式を掃き出し法（Ｇａｕｓｓ−Ｊｏｒｄａｎの消去法）等の一般解法で解くことにより、係数データＷｉ（ｉ＝１，２，・・・，ｎ）を求めることができる。
【０２１７】
図１７は、第１の処理部１２１の係数メモリ１３４に格納すべき係数データＷｉを生成する係数データ生成装置１５０の構成を示している。
この係数データ生成装置１５０は、画像信号Ｖｂに対応した教師信号ＳＴが入力される入力端子１５１と、この教師信号ＳＴに対して符号化を行ってＭＰＥＧ２ストリームを得るＭＰＥＧ２符号化器１５２と、このＭＰＥＧ２ストリームに対して復号化を行って画像信号Ｖａに対応した生徒信号ＳＳを得るＭＰＥＧ２復号化器１５３とを有している。ここで、ＭＰＥＧ２復号化器１５３は、図１に示すデジタル放送受信機１００におけるＭＰＥＧ２復号化器１０７およびバッファメモリ１０８に対応したものである。
【０２１８】
また、係数データ生成装置１５０は、復号化器１５３より出力されるＩピクチャの生徒信号ＳＳ（Ｉ）に基づいて、予測に使用する予測タップの複数の画素データを選択的に取り出す予測タップ選択回路１５４と、同様に、復号化器１５３より出力されるＩピクチャの生徒信号ＳＳ（Ｉ）に基づいて、クラス分類に使用するクラスタップの複数の画素データを選択的に取り出すクラスタップ選択回路１５４とを有している。これらのタップ選択回路１５４，１５５は、第１の処理部１２１のタップ選択回路１３１，１３２と同様に構成される。予測タップおよびクラスタップの画素データは、それぞれ、Ｉピクチャの教師信号ＳＴ（Ｉ）における注目位置に対して空間方向（水平方向、垂直方向）の周辺に位置する画素データである。
【０２１９】
また、係数データ生成装置１５０は、クラスタップ選択回路１５４で選択的に取り出されるクラスタップの画素データから空間クラスを示すクラスコードＣＬ１を生成するクラス生成回路１５６を有している。このクラス生成回路１５６は、第１の処理部１２１のクラス生成回路１３３と同様に構成される。
【０２２０】
また、係数データ生成装置１５０は、入力端子１５１に供給される教師信号ＳＴの時間調整を行うための遅延回路１５７と、この遅延回路１５７で時間調整されたＩピクチャの教師信号ＳＴ（Ｉ）より得られる各注目位置の画素データｙと、この各注目位置の画素データｙにそれぞれ対応して予測タップ選択回路１５５で選択的に取り出される予測タップの画素データｘｉと、各注目位置の画素データｙにそれぞれ対応してクラス生成回路１５６で生成されるクラスコードＣＬ１とから、クラス毎に、係数データＷｉ（ｉ＝１〜ｎ）を得るための正規方程式（上述の（１０）式参照）を生成する正規方程式生成部１５８を有している。
【０２２１】
この場合、１個の画素データｙとそれに対応するｎ個の予測タップの画素データｘｉとの組み合わせで１個の学習データが生成されるが、教師信号ＳＴ（Ｉ）と生徒信号ＳＳ（Ｉ）との間で、クラス毎に、多くの学習データが生成されていく。これにより、正規方程式生成部１５８では、クラス毎に、係数データＷｉ（ｉ＝１〜ｎ）を得るための正規方程式が生成される。
【０２２２】
また、係数データ生成装置１５０は、正規方程式生成部１５８で生成された正規方程式のデータが供給され、その正規方程式を解いて、各クラスの係数データＷｉを求める係数データ決定部１５９と、この求められた各クラスの係数データＷｉを格納する係数メモリ１６０とを有している。
【０２２３】
次に、図１７に示す係数データ生成装置１５０の動作を説明する。
入力端子１５１には画像信号Ｖｂに対応した教師信号ＳＴが供給され、そしてＭＰＥＧ２符号化器１５２で、この教師信号ＳＴに対して符号化が施されて、ＭＰＥＧ２ストリームが生成される。このＭＰＥＧ２ストリームは、ＭＰＥＧ２復号化器１５３に供給される。ＭＰＥＧ２復号化器１５３で、このＭＰＥＧ２ストリームに対して復号化が施されて、画像信号Ｖａに対応した生徒信号ＳＳが生成される。
【０２２４】
復号化器１５３で生成されたＩピクチャの生徒信号ＳＳ（Ｉ）に基づいて、クラスタップ選択回路１５５で、Ｉピクチャの教師信号ＳＴ（Ｉ）における注目位置に対して空間方向の周辺に位置するクラスタップの画素データが選択的に取り出される。このクラスタップの画素データはクラス生成回路１５６に供給される。このクラス生成回路１５６では、クラスタップの画素データに１ビットのＡＤＲＣ等の処理が施されて、空間クラスを示すクラスコードＣＬ１が生成される。
【０２２５】
また、復号化器１５３で生成されたＩピクチャの生徒信号ＳＳ（Ｉ）に基づいて、予測タップ選択回路１５４で、Ｉピクチャの教師信号ＳＴ（Ｉ）における注目位置に対して空間方向の周辺に位置する予測タップの画素データが選択的に取り出される。
【０２２６】
そして、遅延回路１５７で時間調整された教師信号ＳＴ（Ｉ）から得られる各注目位置の画素データｙと、この各注目位置の画素データｙにそれぞれ対応して予測タップ選択回路１５４で選択的に取り出される予測タップの画素データｘｉと、各注目位置の画素データｙにそれぞれ対応してクラス生成回路１５６で生成されるクラスコードＣＬ１とを用いて、正規方程式生成部１５８では、クラス毎に、係数データＷｉ（ｉ＝１〜ｎ）を得るための正規方程式（（１０）式参照）が生成される。この正規方程式は係数データ決定部１５９で解かれて各クラスの係数データＷｉが求められ、その係数データＷｉは係数メモリ１６０に格納される。
【０２２７】
このように、図１７に示す係数データ生成装置１５０においては、第１の処理部１２１の係数メモリ１３４に格納される各クラスの係数データＷｉを生成することができる。
【０２２８】
生徒信号ＳＳは、教師信号ＳＴに対して符号化を施してＭＰＥＧ２ストリームを生成し、その後このＭＰＥＧ２ストリームに対して復号化を施して得たものである。したがって、この生徒信号ＳＳは、画像信号Ｖａと同様の符号化雑音を含んだものとなる。そのため、第１の処理部１２１において、画像信号Ｖａ（Ｉ）からこの係数データＷｉを用いて得られる画像信号Ｖｂ（Ｉ）は、画像信号Ｖａ（Ｉ）に比べて符号化雑音が軽減されたものとなる。
【０２２９】
図１８は、第２の処理部１２２の係数メモリ１４７に格納すべき係数データＷｉを生成する係数データ生成装置１７０の構成を示している。
この係数データ生成装置１７０は、画像信号Ｖｂに対応した教師信号ＳＴが入力される入力端子１７１と、この教師信号ＳＴを一時的に格納するバッファメモリ１７２とを有している。
【０２３０】
また、係数データ生成装置１７０は、バッファメモリ１７２に一時的に格納さた教師信号ＳＴに対して符号化を行ってＭＰＥＧ２ストリームを得るＭＰＥＧ２符号化器１７３と、このＭＰＥＧ２ストリームに対して復号化を行って画像信号Ｖａに対応した生徒信号ＳＳを得るＭＰＥＧ２復号化器１７４とを有している。ここで、ＭＰＥＧ２復号化器１７４は、図１に示すデジタル放送受信機１００におけるＭＰＥＧ２復号化器１０７およびバッファメモリ１０８に対応したものである。
【０２３１】
また、係数データ生成装置１７０は、予測に使用する予測タップの複数の画素データを選択的に取り出す予測タップ選択回路１７５と、クラス分類に使用するクラスタップの複数の画素データを選択的に取り出すクラスタップ選択回路１７６とを有している。これらのタップ選択回路１７５，１７６は、上述した第２の処理部１２２のタップ選択回路１４１，１４２と同様に構成される。
【０２３２】
タップ選択回路１７５，１７６には、復号化器１７４から、ＰピクチャまたはＢピクチャの教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）における注目位置に対応した生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）の画素データと対となっているピクチャ情報ＰＩと、後述する動きベクトル検出部１７７で検出された当該生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）の画素データが位置する所定位置（以下、「注目画素データ位置」という）の動きベクトルＭＶとが、動作制御情報として供給される。予測タップおよびクラスタップの画素データは、それぞれ、ＰピクチャまたはＢピクチャの教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）における注目位置に対して空間方向（水平方向、垂直方向）および時間方向（フレーム方向）の周辺に位置する画素データである。
【０２３３】
タップ選択回路１７５，１７６は、それぞれ、復号化器１７４で得られるＰピクチャまたはＢピクチャの生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）、およびバッファメモリ１７２に格納されている、そのピクチャの生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）を得る際に使用されたＩピクチャ、Ｐピクチャの生徒信号ＳＳ（Ｉ），ＳＳ（Ｐ）に対応した、Ｉピクチャ、Ｐピクチャの教師信号ＳＴ（Ｉ），ＳＴ（Ｐ）に基づいて、複数の画素データを選択的に取り出す。ここで、タップ選択回路１７５，１７６は、上述した動きベクトルＭＶに基づき、教師信号ＳＴ（Ｉ），ＳＴ（Ｐ）に対して、生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）を基準とした動き補償を行う。
【０２３４】
また、係数データ生成装置１７０は、上述した動きベクトルＭＶを検出する動きベクトル検出部１７７を有している。この動きベクトル検出部１７７には、復号化器１７４から生徒信号ＳＳが供給される。この動きベクトル検出部１７７は、上述した第２の処理部１２２の動きベクトル検出部１４３と同様に構成される。この動きベクトル検出部１７７は、注目画素データ位置の動きベクトルＭＶを検出する。
【０２３５】
この動きベクトル検出部１７７は、動きベクトルＭＶの他に、下位階層の動きベクトルＶ０および最小の評価値ＥＹｍｉｎも出力する。動きベクトルＭＶは、上述したように、予測タップおよびクラスタップの画素データを選択する際の動き補償に使用されるが、動きベクトルＶ０および評価値ＥＹｍｉｎは後述するようにクラス分類のために使用される。
【０２３６】
また、係数データ生成装置１７０は、クラス生成回路１７８，１７９を有している。クラス生成回路１７８は、クラスタップ選択回路１７６で選択的に取り出されるクラスタップの画素データから時空間クラスを示すクラスコードＣＬ２を生成する。このクラス生成回路１７８は、第２の処理部１２２のクラス生成回路１４４と同様に構成される。
【０２３７】
クラス生成回路１７９は、動きベクトル検出部１７７から出力される、注目画素データ位置の動きベクトルＭＶに関連した動きベクトルＶ０および評価値ＥＹｍｉｎをそれぞれ閾値判定して、動き関連クラスを示すクラスコードＣＬｂを生成する。このクラス生成回路１７８は、第２の処理部１２２のクラス生成回路１４５と同様に構成される。
【０２３８】
また、係数データ生成装置１７０は、クラス生成回路１７８，１７９で生成されるクラスコードＣＬａ，ＣＬｂを統合して１個のクラスコードＣＬ２とするクラス統合回路１８０を有している。このクラスコードＣＬ２は、ＰピクチャまたはＢピクチャの教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）における注目位置の画素データが属するクラスを示すものとなる。
【０２３９】
また、係数データ生成装置１７０は、バッファメモリ１７２より読み出される教師信号ＳＴの時間調整を行うための遅延回路１８１と、この遅延回路１８１で時間調整されたＰピクチャまたはＢピクチャの教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）より得られる各注目位置の画素データｙと、この各注目位置の画素データｙにそれぞれ対応して予測タップ選択回路１７５で選択的に取り出される予測タップの画素データｘｉと、各注目位置の画素データｙにそれぞれ対応してクラス統合回路１８０で得られたクラスコードＣＬ２とから、クラス毎に、係数データＷｉ（ｉ＝１〜ｎ）を得るための正規方程式（上述の（１０）式参照）を生成する正規方程式生成部１８２を有している。
【０２４０】
この場合、１個の画素データｙとそれに対応するｎ個の予測タップの画素データｘｉとの組み合わせで１個の学習データが生成されるが、教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）と生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）との間で、クラス毎に、多くの学習データが生成されていく。これにより、正規方程式生成部１８２では、クラス毎に、係数データＷｉ（ｉ＝１〜ｎ）を得るための正規方程式が生成される。
【０２４１】
また、係数データ生成装置１７０は、正規方程式生成部１８２で生成された正規方程式のデータが供給され、その正規方程式を解いて、各クラスの係数データＷｉを求める係数データ決定部１８３と、この求められた各クラスの係数データＷｉを格納する係数メモリ１８４とを有している。
【０２４２】
次に、図１８に示す係数データ生成装置１７０の動作を説明する。
入力端子１７１には画像信号Ｖｂに対応した教師信号ＳＴが供給され、この教師信号ＳＴはバッファメモリ１７２に一時的に格納される。そして、このようにバッファメモリ１７２に一時的に格納された教師信号ＳＴに対して、ＭＰＥＧ２符号化器１７３で符号化が施されて、ＭＰＥＧ２ストリームが生成される。このＭＰＥＧ２ストリームは、ＭＰＥＧ２復号化器１７４に供給される。ＭＰＥＧ２復号化器１７４で、このＭＰＥＧ２ストリームに対して復号化が施されて、画像信号Ｖａに対応した生徒信号ＳＳが生成される。
【０２４３】
予測タップ選択回路１７５およびクラスタップ選択回路１７６には、教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）の注目位置に対応した生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）の画素データと対となっているピクチャ情報ＰＩが復号化器１７４より供給されると共に、動きベクトル検出部１７７で検出された注目画素データ位置の動きベクトルＭＶが供給される。
【０２４４】
復号化器１７４で生成されたＰピクチャまたはＢピクチャの生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）、およびバッファメモリ１７２に格納されている、そのピクチャの生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）を得る際に使用されたＩピクチャ、Ｐピクチャの生徒信号ＳＳ（Ｉ），ＳＳ（Ｐ）に対応した、Ｉピクチャ、Ｐピクチャの教師信号ＳＴ（Ｉ），ＳＴ（Ｐ）に基づいて、クラスタップ選択回路１７６で、ＰピクチャまたはＢピクチャの教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）における注目位置に対して空間方向および時間方向の周辺に位置するクラスタップの画素データが選択的に取り出される。この場合、動きベクトルＭＶに基づき、教師信号ＳＴ（Ｉ），ＳＴ（Ｐ）に対して、生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）を基準とした動き補償が行われる。
【０２４５】
クラスタップ選択回路１７６で選択的に取り出されたクラスタップの画素データはクラス生成回路１７８に供給される。このクラス生成回路１７８では、クラスタップの画素データに１ビットのＡＤＲＣ等の処理が施されて、時空間クラスを示すクラスコードＣＬａが生成される。
【０２４６】
また、動きベクトル検出部１７７より出力される、動きベクトルＭＶに関連した動きベクトルＶ０および評価値ＥＹｍｉｎはクラス生成回路１７９に供給される。このクラス生成回路１７９では、動きベクトルＶ０および評価値ＥＹｍｉｎそれぞれ閾値判定されて、動き関連クラスを示すクラスコードＣＬｂを生成される。
【０２４７】
そして、クラス生成回路１７８，１７９で生成されるクラスコードＣＬａ，ＣＬｂはクラス統合回路１８０で統合され、ＰピクチャまたはＢピクチャの教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）における注目位置の画素データが属するクラスを示す、１個のクラスコードＣＬ２が得られる。
【０２４８】
また、復号化器１７４で生成されたＰピクチャまたはＢピクチャの生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）、およびバッファメモリ１７２に格納されている、そのピクチャの生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）を得る際に使用されたＩピクチャ、Ｐピクチャの生徒信号ＳＳ（Ｉ），ＳＳ（Ｐ）に対応した、Ｉピクチャ、Ｐピクチャの教師信号ＳＴ（Ｉ），ＳＴ（Ｐ）に基づいて、予測タップ選択回路１７５で、ＰピクチャまたはＢピクチャの教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）における注目位置に対して空間方向および時間方向の周辺に位置する予測タップの画素データが選択的に取り出される。この場合、動きベクトルＭＶに基づき、教師信号ＳＴ（Ｉ），ＳＴ（Ｐ）に対して、生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）を基準とした動き補償が行われる。
【０２４９】
そして、遅延回路１８１で時間調整された教師信号ＳＴ（Ｐ）／ＳＴ（Ｂ）から得られる各注目位置の画素データｙと、この各注目位置の画素データｙにそれぞれ対応して予測タップ選択回路１７５で選択的に取り出される予測タップの画素データｘｉと、各注目位置の画素データｙにそれぞれ対応してクラス統合回路１８０で得られるクラスコードＣＬ２とを用いて、正規方程式生成部１８２では、クラス毎に、係数データＷｉ（ｉ＝１〜ｎ）を得るための正規方程式（（１０）式参照）が生成される。この正規方程式は係数データ決定部１８３で解かれて各クラスの係数データＷｉが求められ、その係数データＷｉは係数メモリ１８４に格納される。
【０２５０】
このように、図１８に示す係数データ生成装置１７０においては、第２の処理部１２２の係数メモリ１４７に格納される各クラスの係数データＷｉを生成することができる。
【０２５１】
生徒信号ＳＳは、教師信号ＳＴに対して符号化を施してＭＰＥＧ２ストリームを生成し、その後このＭＰＥＧ２ストリームに対して復号化を施して得たものである。したがって、この生徒信号ＳＳは、画像信号Ｖａと同様の符号化雑音を含んだものとなる。そのため、第２の処理部１２２において、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）等からこの係数データＷｉを用いて得られる画像信号Ｖｂ（Ｐ）／Ｖｂ（Ｂ）は、画像信号Ｖａ（Ｐ）／Ｖａ（Ｂ）に比べて符号化雑音が軽減されたものとなる。
【０２５２】
なお、図１の画像信号処理部１１０における処理を、例えば図１９に示すような画像信号処理装置３００によって、ソフトウェアで実現することも可能である。
【０２５３】
まず、図８に示す画像信号処理装置３００について説明する。この画像信号処理装置３００は、装置全体の動作を制御するＣＰＵ３０１と、このＣＰＵ３０１の制御プログラムや係数データ等が格納されたＲＯＭ（Ｒｅａｄ Ｏｎｌｙ Ｍｅｍｏｒｙ）３０２と、ＣＰＵ３０１の作業領域を構成するＲＡＭ（Ｒａｎｄｏｍ Ａｃｃｅｓｓ Ｍｅｍｏｒｙ）３０３とを有している。これらＣＰＵ３０１、ＲＯＭ３０２およびＲＡＭ３０３は、それぞれバス３０４に接続されている。
【０２５４】
また、画像信号処理装置３００は、外部記憶装置としてのハードディスクドライブ（ＨＤＤ）３０５と、フロッピー（登録商標）ディスク３０６をドライブするドライブ（ＦＤＤ）３０７とを有している。これらドライブ３０５，３０７は、それぞれバス３０４に接続されている。
【０２５５】
また、画像信号処理装置３００は、インターネット等の通信網４００に有線または無線で接続する通信部３０８を有している。この通信部３０８は、インタフェース３０９を介してバス３０４に接続されている。
【０２５６】
また、画像信号処理装置３００は、ユーザインタフェース部を備えている。このユーザインタフェース部は、リモコン送信機２００からのリモコン信号ＲＭを受信するリモコン信号受信回路３１０と、ＬＣＤ（Ｌｉｑｕｉｄ Ｃｒｙｓｔａｌ Ｄｉｓｐｌａｙ）等からなるディスプレイ３１１とを有している。受信回路３１０はインタフェース３１２を介してバス３０４に接続され、同様にディスプレイ３１１はインタフェース３１３を介してバス３０４に接続されている。
【０２５７】
また、画像信号処理装置３００は、画像信号Ｖａを入力するための入力端子３１４と、画像信号Ｖｂを出力するための出力端子３１５とを有している。入力端子３１４はインタフェース３１６を介してバス３０４に接続され、同様に出力端子３１５はインタフェース３１７を介してバス３０４に接続される。
【０２５８】
ここで、上述したようにＲＯＭ３０２に制御プログラムや係数データ等を予め格納しておく代わりに、例えばインターネットなどの通信網４００より通信部３０８を介してダウンロードし、ハードディスクやＲＡＭ３０３に蓄積して使用することもできる。また、これら制御プログラムや係数データ等をフロッピー（登録商標）ディスク３０６で提供するようにしてもよい。
【０２５９】
また、処理すべき画像信号Ｖａを入力端子３１４より入力する代わりに、予めハードディスクに記録しておき、あるいはインターネットなどの通信網４００より通信部３０８を介してダウンロードしてもよい。また、処理後の画像信号Ｖｂを出力端子３１５に出力する代わり、あるいはそれと並行してディスプレイ３１１に供給して画像表示をしたり、さらにはハードディスクに格納したり、通信部３０８を介してインターネットなどの通信網４００に送出するようにしてもよい。
【０２６０】
図２０のフローチャートを参照して、図１９に示す画像信号処理装置３００における、画像信号Ｖａより画像信号Ｖｂを得るため処理手順を説明する。
【０２６１】
まず、ステップＳＴ２１で、処理を開始し、ステップＳＴ２２で、例えば入力端子３１４より装置内に１ＧＯＰ（Ｇｒｏｕｐ Ｏｆ Ｐｉｃｔｕｒｅ）分の画像信号Ｖａを入力する。この場合、画像信号Ｖａの画素データと対となっている動き補償用ベクトルＭＩおよびピクチャ情報ＰＩも入力する。
【０２６２】
このように入力端子３１４より入力される画像信号Ｖａ等はＲＡＭ３０３に一時的に格納される。なお、この画像信号Ｖａ等が装置内のハードディスクドライブ３０５に予め記録されている場合には、このドライブ３０５からこの画像信号Ｖａ等を読み出し、この画像信号Ｖａ等をＲＡＭ３０３に一時的に格納する。
【０２６３】
そして、ステップＳＴ２３で、画像信号Ｖａの全てのＧＯＰの処理が終わっているか否かを判定する。処理が終わっているときは、ステップＳＴ２４で、処理を終了する。一方、処理が終わっていないときは、ステップＳＴ２５に進む。
【０２６４】
このステップＳＴ２５では、ステップＳＴ２２で入力されたＩピクチャの画像信号Ｖａ（Ｉ）に基づいて、Ｉピクチャの画像信号Ｖｂ（Ｉ）における注目位置に対して空間方向の周辺に位置する複数の画素データ（クラスタップの画素データ）を選択的に取り出し、このクラスタップの画素データを用いてＩピクチャの画像信号Ｖｂ（Ｉ）における注目位置の画素データが属するクラスを示すクラスコードＣＬ１を生成する。
【０２６５】
次に、ステップＳＴ２６で、ステップＳＴ２２で入力されたＩピクチャの画像信号Ｖａ（Ｉ）に基づいて、Ｉピクチャの画像信号Ｖｂ（Ｉ）における注目位置に対して空間方向の周辺に位置する複数の画素データ（予測タップの画素データ）を取得する。そして、ステップＳＴ２７で、ステップＳＴ２５で生成されたクラスコードＣＬ１に対応した係数データＷｉとステップＳＴ２６で取得された予測タップの画素データｘｉを使用して、上述した（１）式の推定式に基づいて、Ｉピクチャの画像信号Ｖｂ（Ｉ）における注目位置の画素データｙを生成する。このように生成される画像データからなるＩピクチャの画像信号Ｖｂ（Ｉ）は、ＲＡＭ３０３に一時的に格納される。
【０２６６】
そして、ステップＳＴ２８で、Ｉピクチャの処理が終了したか否かを判定する。終了していないときは、ステップＳＴ２５に戻って、次の注目位置についての処理に移る。一方処理が終了しているときは、ステップＳＴ２９に進み、ＰピクチャおよびＢピクチャの画像信号Ｖｂ（Ｐ），Ｖｂ（Ｂ）を得る処理に移る。
【０２６７】
これらの画像信号Ｖｂ（Ｐ），Ｖｂ（Ｂ）を得る処理は、上述したＩピクチャの画像信号Ｖｂ（Ｉ）を得るステップＳＴ２５〜ステップＳＴ２８までの処理と同様に行われる。
【０２６８】
ただし、予測タップおよびクラスタップの画素データは、ステップＳＴ２２で入力されたＰピクチャ、Ｂピクチャの画像信号Ｖａ（Ｐ），Ｖａ（Ｂ）の他に、生成されたＩピクチャ、Ｐピクチャの画像信号Ｖｂ（Ｉ）、Ｖｂ（Ｐ）からも選択的に取り出される。
【０２６９】
また、マクロブロック毎の動き補償用ベクトルＭＩが上位階層の動きベクトルとして使用され、注目画素データ位置の動きベクトルＭＶが階層的に検出される。そして、画像信号Ｖａ（Ｐ），Ｖａ（Ｂ）を基準として、画像信号Ｖｂ（Ｉ）、Ｖｂ（Ｐ）に対して、動きベクトルＭＶに基づいて動き補償が行われる。
【０２７０】
ステップＳＴ２９で、所定のＰピクチャまたはＢピクチャの処理が終了したとき、ステップＳＴ３０で、処理すべき次のピクチャがあるか否かを判定する。次のピクチャがあるときは、ステップＳＴ２９に戻って、そのピクチャの画像信号を得る処理に移る。一方、処理すべき次のピクチャがないときは、ステップＳＴ３１に進む。
【０２７１】
ステップＳＴ３１では、生成された１ＧＯＰ分の画像信号Ｖｂ（Ｉ），Ｖｂ（Ｐ），Ｖｂ（Ｂ）を、符号化の順番から、実際のフレーム／フィールドの順番に並べ換えて、出力端子３１５に出力する。そして、その後、ステップＳＴ２２に戻って、次の１ＧＯＰ分の画像信号Ｖａの入力処理に移る。
【０２７２】
このように、図２０に示すフローチャートに沿って処理をすることで、入力された画像信号Ｖａの画素データを処理して、画像信号Ｖｂの画素データを得ることができる。上述したように、このように処理して得られた画像信号Ｖｂは出力端子３１５に出力されたり、ディスプレイ３１１に供給されてそれによる画像が表示されたり、さらにはハードディスクドライブ３０５に供給されてハードディスクに記録されたりする。
【０２７３】
また、処理装置の図示は省略するが、図１７の係数データ生成装置１５０における処理も、ソフトウェアで実現可能である。
【０２７４】
図２１のフローチャートを参照して、係数データを生成するための処理手順を説明する。
まず、ステップＳＴ３１で、処理を開始し、ステップＳＴ３２で、教師信号ＳＴを１ＧＯＰ分だけ入力する。そして、ステップＳＴ３３で、教師信号ＳＴの全てのＧＯＰの処理が終了したか否かを判定する。終了していないときは、ステップＳＴ３４で、ステップＳＴ３２で入力された教師信号ＳＴから生徒信号ＳＳを生成する。
【０２７５】
次に、ステップＳＴ３５で、ステップＳＴ３４で生成されたＩピクチャの生徒信号ＳＳ（Ｉ）に基づいて、Ｉピクチャの教師信号ＳＴ（Ｉ）における注目位置に対して空間方向の周辺に位置する複数の画素データ（クラスタップの画素データ）を選択的に取り出し、このクラスタップの画素データを用いてＩピクチャの教師信号ＳＴ（Ｉ）における注目位置の画素データが属するクラスを示すクラスコードＣＬ１を生成する。
【０２７６】
次に、ステップＳＴ３６で、ステップＳＴ３４で生成されたＩピクチャの生徒信号ＳＳ（Ｉ）に基づいて、Ｉピクチャの教師信号ＳＴ（Ｉ）における注目位置に対して空間方向の周辺に位置する複数の画素データ（予測タップの画素データ）を取得する。
【０２７７】
そして、ステップＳＴ３７で、ステップＳＴ３５で生成されたクラスコードＣＬ１、ステップＳＴ３６で取得された予測タップの画素データｘｉおよび教師信号ＳＴ（Ｉ）における注目位置の画素データｙを用いて、クラス毎に、（１０）式に示す正規方程式を得るための加算をする（（８）式、（９）式参照）。
【０２７８】
次に、ステップＳＴ３８で、ステップＳＴ３２で入力されたＩピクチャの教師信号ＳＴ（Ｉ）の画素データの全領域において学習処理が終了したか否かを判定する。学習処理を終了しているときは、ステップＳＴ３２に戻って、次の１ＧＯＰ分の教師信号ＳＴの入力を行って、上述したと同様の処理を繰り返す。一方、学習処理を終了していないときは、ステップＳＴ３５に戻って、次の注目位置についての処理に移る。
【０２７９】
上述したステップＳＴ３３で、処理が終了したときは、ステップＳＴ３９で、上述のステップＳＴ３７の加算処理によって生成された、各クラスの正規方程式を掃き出し法などで解いて、各クラスの係数データＷｉを算出する。そして、ステップＳＴ４０で、各クラスの係数データＷｉをメモリに保存し、その後にステップＳＴ４１で、処理を終了する。
【０２８０】
このように、図２１に示すフローチャートに沿って処理をすることで、図１７に示す係数データ生成装置１５０と同様の手法によって、Ｉピクチャに係る各クラスの係数データＷｉを得ることができる。
【０２８１】
なお、詳細説明は省略するが、図１８の係数データ生成装置１７０における処理も、上述した図１７の係数データ生成装置１５０における処理と同様に、ソフトウェアで実現可能である。
【０２８２】
その場合、Ｉピクチャの教師信号ＳＴ（Ｉ）、生徒信号ＳＳ（Ｉ）の代わりに、Ｐピクチャの教師信号ＳＴ（Ｐ）、生徒信号ＳＳ（Ｐ）またはＢピクチャの教師信号ＳＴ（Ｂ）、生徒信号ＳＳ（Ｂ）が用いられる。また、クラスタップ、予測タップの画素データを得る際には、生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）の他に、この生徒信号ＳＳ（Ｐ）／ＳＳ（Ｂ）を得る際に使用された生徒信号ＳＳ（Ｉ），ＳＳ（Ｐ）に対応した教師信号ＳＴ（Ｉ），ＳＴ（Ｐ）も用いられる。これにより、ＰピクチャまたはＢピクチャに係る各クラスの係数データＷｉを得ることができる。
【０２８３】
また、マクロブロック毎の動き補償用ベクトルＭＩが上位階層の動きベクトルとして使用され、注目画素データ位置の動きベクトルＭＶが階層的に検出される。そして、生徒信号ＳＳ（Ｐ），ＳＳ（Ｂ）を基準として、教師信号ＳＴ（Ｉ）、ＳＴ（Ｐ）に対して、動きベクトルＭＶに基づいて動き補償が行われる。
【０２８４】
次に、この発明の第２の実施の形態について説明する。図２２は、第２の実施の形態としてのデジタル放送受信機５００の構成を示している。
このデジタル放送受信機５００は、マイクロコンピュータを備え、システム全体の動作を制御するためのシステムコントローラ５０１と、リモートコントロール信号ＲＭを受信するリモコン信号受信回路５０２とを有している。リモコン信号受信回路５０２は、システムコントローラ５０１に接続され、リモコン送信機２００よりユーザの操作に応じて出力されるリモートコントロール信号ＲＭを受信し、その信号ＲＭに対応する操作信号をシステムコントローラ５０１に供給するように構成されている。
【０２８５】
また、デジタル放送受信機５００は、受信アンテナ５０５と、この受信アンテナ５０５で捕らえられた放送信号（ＲＦ変調信号）が供給され、選局処理、復調処理および誤り訂正処理等を行って、所定番組に係る符号化された画像信号としてのＭＰＥＧ２ストリームを得るチューナ部５０６とを有している。
【０２８６】
また、デジタル放送受信機５００は、このチューナ部５０６より出力されるＭＰＥＧ２ストリームを復号化して画像信号Ｖａを得るＭＰＥＧ２復号化器５０７と、このＭＰＥＧ２復号化器５０７より出力される画像信号Ｖａを一時的に格納するバッファメモリ５０８とを有している。
【０２８７】
なお、本実施の形態において、ＭＰＥＧ２復号化器５０７からは、画像信号Ｖａを構成する各画素データの他に、ピクチャ情報ＰＩ，動き補償用ベクトルＭＩも出力される。バッファメモリ５０８には、各画素データと対にしてピクチャ情報ＰＩおよび動き補償ベクトルＭＩも格納される。
【０２８８】
ピクチャ情報ＰＩは、出力される画素データがＩピクチャ、Ｐピクチャ、Ｂピクチャのいずれのピクチャに係るものであったかを示す情報である。動き補償用ベクトルＭＩは、出力される画素データがＰピクチャ、Ｂピクチャに係るものである場合、その画素データを復号する際のリファレンスデータがどのように動き補償されたかを示す情報である。この動き補償用ベクトルＭＩは、周知のように、ＭＰＥＧ２符号化器で、マクロブロック毎に検出された動きベクトルである。
【０２８９】
ＭＰＥＧ２復号化器５０７は、上述した図１に示すデジタル放送受信機１００のＭＰＥＧ２復号化器１０７と同様に構成されている（図２参照）。この復号化器５０７からは、符号化の順番で各ピクチャの画像信号Ｖａが出力される。本実施の形態においては、上述したバッファメモリ５０８から画像信号Ｖａを読み出す際に、各ピクチャの画像信号が符号化の順番から実際のフレーム／フィールドの順番に並べ直される。
【０２９０】
因みに、ＭＰＥＧ方式の符号化では、従来周知のように、実際のフレーム／フィールドの順番とは異なる順番で符号化が行われている。すなわち、Ｉピクチャ、Ｐピクチャの画像信号が先に符号化され、それらの間に挟まれたＢピクチャの画像信号はその後に符号化されている。
【０２９１】
また、デジタル放送受信機５００は、バッファメモリ５０８に記憶されている画像信号Ｖａを、この画像信号Ｖａとは異なるフレーム数を有する画像信号Ｖｂに変換する画像信号処理部５１０と、この画像信号処理部５１０より出力される画像信号Ｖｂによる画像を表示するディスプレイ部５１１とを有している。ディスプレイ部５１１は、例えばＣＲＴディスプレイ、あるいはＬＣＤ等の表示器で構成されている。
【０２９２】
本実施の形態においては、画像信号Ｖａに対して、画像信号Ｖｂは５倍のフレーム数を有するものとされる。例えば、画像信号Ｖａのフレーム構成が図２３Ａに示すようであるとき、画像信号Ｖｂのフレーム構成は図２３Ｂに示すようになる。この場合、画像信号Ｖｂは、画像信号Ｖａの各ピクチャのフレーム間にフレームｆ_１〜ｆ_４が挿入された構成となっている。なお、ｆ_Ｉ，ｆ_Ｐ，ｆ_Ｂは、それぞれＩピクチャ、Ｐピクチャ、Ｂピクチャのフレームを示している。
【０２９３】
図２２に示すデジタル放送受信機５００の動作を説明する。
チューナ部５０６より出力されるＭＰＥＧ２ストリームはＭＰＥＧ２復号化器５０７に供給されて復号化される。そして、この復号化器５０７より出力される画像信号Ｖａは、バッファメモリ５０８に供給されて一時的に格納される。この場合、復号器５０７からは、画像信号Ｖａの各画素データと対となって、ピクチャ情報ＰＩおよび動き補償用ベクトルＭＩが出力される。これらもバッファメモリ５０８に一時的に格納される。
【０２９４】
このようにバッファメモリ５０８に一時的に格納された画像信号Ｖａは画像信号処理部５１０に供給され、フレーム数が５倍とされた画像信号Ｖｂに変換される。この画像信号処理部５１０では、画像信号Ｖａを構成する画素データから、画像信号Ｖｂのフレームｆ_１〜ｆ_４を構成する画素データが生成される。この画像信号処理部５１０では、バッファメモリ５０８に格納されているピクチャ情報ＰＩおよび動き補償用ベクトルＭＩが用いられて、後述するように変換処理が行われる。
【０２９５】
画像信号処理部５１０より出力される画像信号Ｖｂはディスプレイ部５１１に供給され、このディスプレイ部５１１の画面上にはその画像信号Ｖｂによる画像が表示される。
【０２９６】
次に、画像信号処理部５１０の詳細を説明する。
画像信号処理部５１０は、画像信号Ｖｂのフレームｆ_１〜ｆ_４を得るための画像信号生成部５２１と、バッファメモリ５０８に格納されている各ピクチャのフレームｆ_Ｉ，ｆ_Ｐ，ｆ_Ｂおよび画像信号生成部５２１で生成されたフレームｆ_１〜ｆ_４とを選択的に取り出して画像信号Ｖｂを得る出力選択回路５２２とを有している。
【０２９７】
画像信号生成部５２１について説明する。この画像信号生成部５２１では、画像信号Ｖａの連続した複数フレーム、本実施の形態においては連続した３フレームに基づいて、フレームｆ_１〜ｆ_４を生成する。
【０２９８】
この場合、フレームｆ_１，ｆ_２は、当該フレームｆ_１，ｆ_２に対して時間方向位置が前にある２つのフレームと、当該フレームｆ_１，ｆ_２に対して時間方向位置が後にある１つのフレームとを用いて生成される。一方、フレームｆ_３，ｆ_４は、当該フレームｆ_３，ｆ_４に対して時間方向位置が前にある１つのフレームと、当該フレームｆ_３，ｆ_４に対して時間方向位置が後にある２つのフレームとを用いて生成される。
【０２９９】
ただし、Ｉピクチャのフレームｆ_ＩとＢピクチャのフレームｆ_Ｂとの間のフレームｆ_１，ｆ_２に関しては、当該フレームｆ_１，ｆ_２に対して時間方向位置が前にある１つのフレームと、当該フレームｆ_３，ｆ_４に対して時間方向位置が後にある２つのフレームとを用いて生成される。
【０３００】
またこの場合、所定フレームにおける注目位置の画素データを生成する場合、その注目位置の、３つのフレームとの間の動きベクトルＭＶ１，ＭＶ２，ＭＶ３を検出し、これらの動きベクトルＭＶ１，ＭＶ２，ＭＶ３に基づき、所定フレームを基準として３つのフレームの動き補償を行って、フレームｆ_１〜ｆ_４の品質の向上を図る。
【０３０１】
例えば、２つのＢピクチャのフレームｆ_Ｂ，ｆ_Ｂの間のフレームｆ_１における注目位置の画素データを生成する場合、図２４Ａに示すように、Ｉピクチャのフレームｆ_Ｉとの間の動きベクトルＭＶ１と、１番目のＢピクチャのフレームｆ_Ｂとの間の動きベクトルＭＶ２と、２番目のＢピクチャのフレームｆ_Ｂとの間の動きベクトルＭＶ３とを検出する。
【０３０２】
また例えば、２つのＢピクチャのフレームｆ_Ｂ，ｆ_Ｂの間のフレームｆ_３における注目位置の画素データを生成する場合、図２４Ｂに示すように、１番目のＢピクチャのフレームｆ_Ｂとの間の動きベクトルＭＶ１と、２番目のＢピクチャのフレームｆ_Ｂとの間の動きベクトルＭＶ２と、Ｐピクチャのフレームｆ_Ｐとの間の動きベクトルＭＶ３とを検出する。
【０３０３】
また、動きベクトルＭＶ１〜ＭＶ３を検出する際には、３つのフレームに関係する第１のフレームに存在する注目位置に対応した所定位置の、第２のフレームとの間の動きベクトル（以下、「利用動きベクトル」という）ＭＶ′を検出し、この利用動きベクトルＭＶ′を線形配分して動きベクトルＭＶ１〜ＭＶ３を求める。
【０３０４】
例えば、図２４Ａに示す動きベクトルＭＶ１〜ＭＶ３を検出する際には、図２５Ａに示すように、１番目のＢピクチャのフレームｆ_Ｂを第１のフレームとし、このフレームｆ_Ｂに存在する所定位置の、第２のフレームであるＩピクチャのフレームｆ_Ｉとの間の動きベクトルＭＶＩと、１番目のＢピクチャのフレームｆ_Ｂを第１のフレームとし、このフレームｆ_Ｂに存在する所定位置の、第２のフレームであるＰピクチャのフレームｆ_Ｐとの間の動きベクトルＭＶＰを検出する。
【０３０５】
そして、動きベクトルＭＶＩをそれぞれ６／５倍、１／５倍して、動きベクトルＭＶ１，ＭＶ２を求める。また、動きベクトルＭＶＰを２／５倍して動きベクトルＭＶ３を求める。なお、このように動きベクトルＭＶＩ，ＭＶＰを線形配分して動きベクトルＭＶ１〜ＭＶ３を求める場合、１画素以下の部分は四捨五入する。
【０３０６】
また例えば、図２４Ｂに示す動きベクトルＭＶ１〜ＭＶ３を検出する際には、図２５Ｂに示すように、２番目のＢピクチャのフレームｆ_Ｂを第１のフレームとし、このフレームｆ_Ｂに存在する所定位置の、第２のフレームであるＩピクチャのフレームｆ_Ｉとの間の動きベクトルＭＶＩと、２番目のＢピクチャのフレームｆ_Ｂを第１のフレームとし、このフレームｆ_Ｂに存在する所定位置の、第２のフレームであるＰピクチャのフレームｆ_Ｐとの間の動きベクトルＭＶＰを検出する。
【０３０７】
そして、動きベクトルＭＶＩを３／１０倍して動きベクトルＭＶ１を求める。また、動きベクトルＭＶＰをそれぞれ２／５倍、７／５倍して動きベクトルＭＶ２，ＭＶ３を求める。
【０３０８】
なお、上述した動きベクトルＭＶＩ、動きベクトルＭＶＰ等の利用動きベクトルＭＶ′を検出する際、上述した第２の処理部１２２（図４参照）の動きベクトル検出部１４３におけると同様に、所定位置の周辺に位置するマクロブロックの動き補償用ベクトルＭＩを上位階層の動きベクトルとして階層的検出を行う。
【０３０９】
図２６は、画像信号生成部５２１の構成を示している。
この画像信号生成部５２１は、予測に使用する予測タップの複数の画素データを選択的に取り出す予測タップ選択回路５４１と、クラス分類に使用するクラスタップの複数の画素データを選択的に取り出すクラスタップ選択回路５４２とを有している。
【０３１０】
タップ選択回路５４１，５４２は、それぞれ、画像信号Ｖａの連続する３つのフレームに基づいて、画像信号Ｖｂの所定フレーム（ｆ_１〜ｆ_４のいずれか）における注目位置に対して空間方向（水平方向、垂直方向）および時間方向（フレーム方向）の周辺に位置する複数の画素データを選択的に取り出す。
【０３１１】
このタップ選択回路５４１，５４２には、後述する動きベクトル検出部５４３で検出された、画像信号Ｖｂの所定フレームにおける注目位置の、上述した３つのフレーム（以下、「利用フレーム」という）との間の動きベクトルＭＶ１，ＭＶ２，ＭＶ３が供給される。タップ選択回路５４１，５４２は、動きベクトルＭＶ１，ＭＶ２，ＭＶ３に基づき、３つの利用フレームに対して、画像信号Ｖｂの所定フレームを基準とした動き補償を行う。
【０３１２】
このように、予測タップ選択回路５４１においては、動き補償された３つの利用フレームから予測タップの画素データを選択的に取り出すものである。したがって、この予測タップの画素データを用いて後述するように生成される画像信号の所定フレームにおける注目位置の画素データの品質を高めることができる。
【０３１３】
また、クラスタップ選択回路５４２においては、動き補償された３つの利用フレームから予測タップの画素データを選択的に取り出すものである。したがって、上述したように予測タップ選択回路５４１で動き補償が行われて選択された予測タップの画素データに対応した時空間クラスを精度よく検出できる。
【０３１４】
また、画像信号生成部５２１は、動きベクトル検出部５４３を有している。この動きベクトル検出部５４３は、動きベクトルＭＶ１〜ＭＶ３を検出する際に、利用動きベクトルＭＶ′を検出すると共に、この検出された利用動きベクトルＭＶ′を線形配分して動きベクトルＭＶ（ＭＶ１〜ＭＶ３）を求める。
【０３１５】
この動きベクトル検出部５４３は、利用動きベクトルＭＶ′を階層的に検出する。その場合、上述した第２の処理部１２２（図４参照）の動きベクトル検出部１４３におけると同様に、所定位置の周辺に位置する複数のマクロブロックの動き補償用ベクトルＭＩを上位階層の動きベクトルとして用いる。
【０３１６】
図２７は、動きベクトル検出部５４３の構成を示している。この図２７において、図１３と対応する部分には同一符号を付し、適宜その詳細説明を省略する。
【０３１７】
動きベクトル検出部５４３は、第１のフレームの所定位置の周辺に位置する４個のマクロブロックの動き補償用ベクトルＭＩを、上位階層の動きベクトルとして選択する動きベクトル選択回路２１と、この選択回路２１で選択された４個の動きベクトルＭＩ_１〜ＭＩ_４を蓄積しておくための動きベクトル蓄積部２２とを有している。動きベクトル選択回路２１には、所定位置の位置情報が、動作制御情報として供給される。所定位置の周辺に位置する４個のマクロブロックをどのマクロブロックとするかは、上述の図１０を用いて説明したと同様にして決める。ただし、その説明中で、「注目画素データ位置」を「所定位置」に置き換えることとする。
【０３１８】
また、動きベクトル検出部５４３は、バッファメモリ５０８（図２２参照）に格納されている参照フレーム（第１のフレーム）から、所定位置を中心とした参照ブロックの画素データを選択的に読み出す参照ブロック回路２３と、バッファメモリ５０８に格納されている探索フレーム（第２のフレーム）から、探索範囲内の各候補ブロックに対応した画像データを順次読み出すサーチブロック回路２４とを有している。
【０３１９】
ここで、探索フレームの探索範囲は、注目画素データ位置に対応した位置が上位階層の動きベクトルに基づいて移動された位置を基準位置とした範囲とされる。サーチブロック回路２４は、上位階層の動きベクトルとして、動きベクトル蓄積部２２に蓄積された４個の動きベクトルＭＩ_１〜ＭＩ_４を順次使用していく。そして、サーチブロック回路２４は、各上位階層の動きベクトルにより決定された探索範囲で、それぞれ複数の候補ブロックの画素データを順次読み出す。
【０３２０】
また、動きベクトル検出部５４３は、参照ブロック回路２３からの参照ブロックの画素データと、サーチブロック回路２４からの各候補ブロックの画像データとを用いて、上述した（２）式の評価関数Ｅ（Ｙ）_ｎに基づいて、各候補ブロックの評価値を求める評価値算出回路２５を有している。ここで、評価値は参照ブロックと候補ブロックとの相関の度合いを示す相関情報であり、評価値算出回路２５は相関検出手段を構成している。
【０３２１】
なお、サーチブロック回路２４から評価値算出回路２５には、候補ブロックの画素データと共に、その候補ブロックが存在する探索範囲を決定した上位階層の動きベクトルも供給される。評価値算出回路２５は、算出した各候補ブロックの評価値を、上位階層の動きベクトル別に評価値メモリ（図示せず）に記憶する。
【０３２２】
また、動きベクトル検出部５４３は、動きベクトル検出回路２６を有している。この動きベクトル検出回路２６は、評価値算出回路２５で算出された各候補ブロックの評価値を順次評価し、最小の評価値を与える候補ブロックに対応した動きベクトル（ｕ_ｎ、ｖ_ｎ）を、下位階層の動きベクトルＶ０とする。そして、最終的な動きベクトルＭＶ′を、上述した（３）式により求める。ただし、（３）式のＭＶがここではＭＶ′である。
【０３２３】
また、動きベクトル検出部５４３は、動きベクトル検出回路２６で検出される利用動きベクトルＭＶ′を線形配分して動きベクトルＭＶを求める線形配分回路２９を有している。動きベクトルＭＶは、画像信号Ｖｂの所定フレームの時間方向位置に対応した動きベクトルである。この場合、このように利用動きベクトルＭＶ′を線形配分して動きベクトルＭＶを求める場合、１画素以下の部分は四捨五入する。
【０３２４】
図２７に示す動きベクトル検出部５４３の動作を説明する。
動きベクトル選択回路２１は、第１のフレームの所定位置の位置情報に基づいて、その所定位置の周辺に位置する４個のマクロブロックの動き補償用ベクトルＭＩを、上位階層の動きベクトルとして選択する。このように選択された４個の動きベクトルＭＩ_１〜ＭＩ_４は動きベクトル蓄積部２２に供給されて蓄積される。
【０３２５】
例えば、図２４Ａに示すようにフレームｆ_１の注目位置の３つのフレームｆ_Ｉ，ｆ_Ｂ，ｆ_Ｂとの間の動きベクトルＭＶ１，ＭＶ２，ＭＶ３を検出する際、図２５Ａに示すような動きベクトルＭＶＩおよび動きベクトルＭＶＰを利用動きベクトルＭＶ′として検出する必要がある。
【０３２６】
動きベクトルＭＶＩを検出する場合、動きベクトル選択部２１は、１番目のＢピクチャのフレームｆ_Ｂを第１のフレームとし、このフレームｆ_Ｂの所定位置の周辺に位置する４個のマクロブロックの、第２のフレームとしてのＩピクチャのフレームｆ_Ｉとの間の動き補償用ベクトルＭＩを動きベクトルＭＩ_１〜ＭＩ_４として選択する。
【０３２７】
動きベクトルＭＶＰを検出する場合、動きベクトル選択部２１は、１番目のＢピクチャのフレームｆ_Ｂを第１のフレームとし、このフレームｆ_Ｂの所定位置の周辺に位置する４個のマクロブロックの、第２のフレームとしてのＰピクチャのフレームｆ_Ｐとの間の動き補償用ベクトルＭＩを動きベクトルＭＩ_１〜ＭＩ_４として選択する。
【０３２８】
また、参照ブロック回路２３は、バッファメモリ５０８（図２２参照）に格納されている参照フレームから、所定位置を中心とした参照ブロックの画素データを選択的に読み出す。また、サーチブロック回路２４は、バッファメモリ５０８に格納されている探索フレームから、探索範囲内の各候補ブロックに対応した画像データを順次読み出す。
【０３２９】
この場合、探索フレームの探索範囲は、所定位置に対応した位置が上位階層の動きベクトルに基づいて移動された位置を基準位置とした範囲である。サーチブロック回路２４は、上位階層の動きベクトルとして動きベクトル蓄積部２２に蓄積された４個の動きベクトルＭＩ_１〜ＭＩ_４を順次使用し、各上位階層の動きベクトルにより決定された探索範囲でそれぞれ複数の候補ブロックの画素データを順次読み出す。
【０３３０】
参照ブロック回路２３で読み出された参照ブロックの画素データと、サーチブロック回路２４で読み出された各候補ブロックの画像データは評価値算出回路２５に供給される。評価値算出回路２５は、上述した（２）式の評価関数Ｅ（Ｙ）_ｎに基づいて、各候補ブロックの評価値を求める。
【０３３１】
動きベクトル検出回路２６は、評価値算出回路２５で算出された各候補ブロックの評価値を順次評価し、最小の評価値を与える候補ブロック、すなわち参照ブロックと最も相関の高い候補ブロックに対応した動きベクトル（ｕ_ｎ、ｖ_ｎ）を、下位階層の動きベクトルＶ０とする。そして、この下位階層の動きベクトルＶ０に、その候補ブロックの上位階層の動きベクトルＶ１を加算してなる動きベクトル（ｕ，ｖ）を、最終的な利用動きベクトルＭＶ′として出力する。
【０３３２】
そして、動きベクトル検出回路２６で得られた利用動きベクトルＭＶ′は、線形配分回路２９に供給される。線形配分回路２９は、利用動きベクトルＭＶ′を線形配分して、画像信号Ｖｂの所定フレームにおける注目位置の動きベクトルＭＶ（ＭＶ１〜ＭＶ３）を求めて出力する。
【０３３３】
ここで、図２４Ａに示すようにフレームｆ_１の注目位置の３つのフレームｆ_Ｉ，ｆ_Ｂ，ｆ_Ｂとの間の動きベクトルＭＶ１，ＭＶ２，ＭＶ３を検出する場合を説明する。
【０３３４】
動きベクトル選択回路２１で１番目のＢピクチャのフレームｆ_Ｂの所定位置の周辺に位置する４個のマクロブロックのＩピクチャのフレームｆ_Ｉとの間の動き補償用ベクトルＭＩが選択され、サーチブロック回路２４ではＩピクチャのフレームｆ_Ｉが探索フレームとされて候補ブロックの読み出しが行われことで、動きベクトル検出回路２６からは、利用動きベクトルＭＶ′として図２５Ａに示す動きベクトルＭＢＩが得られる。線形配分回路２９では、この動きベクトルＭＢＩがそれぞれ６／５倍、１／５倍されて、動きベクトルＭＶ１，ＭＶ２が求められる。
【０３３５】
また、動きベクトル選択回路２１で１番目のＢピクチャのフレームｆ_Ｂの所定位置の周辺に位置する４個のマクロブロックのＰピクチャのフレームｆ_Ｐとの間の動き補償用ベクトルＭＩが選択され、サーチブロック回路２４ではＰピクチャのフレームｆ_Ｐが探索フレームとされて候補ブロックの読み出しが行われことで、動きベクトル検出回路２６からは、利用動きベクトルＭＶ′として図２５Ａに示す動きベクトルＭＢＰが得られる。そして、線形配分回路２９では、この動きベクトルＭＢＩが２／５倍されて動きベクトルＭＶ３を求められる。
【０３３６】
このように、動きベクトル検出部５４３では、マクロブロック毎に有する動き補償用ベクトルＭＩを上位階層の動きベクトルとして使用して、利用動きベクトルＭＶ′を階層的に検出するものであり、動きベクトルＭＶを素早く検出できる。また、動きベクトル検出部５４３では、所定位置が存在するマクロブロックの動き補償用ベクトルＭＩだけでなく、そのマクロブロックの周辺のマクロブロックの動き補償用ベクトルＭＩをも上位階層の動きベクトルとして使用するものであり、利用動きベクトルＭＶ′を精度よく検出できる。
【０３３７】
なお、図２７に示す動きベクトル検出部５４３においては、所定位置の周辺に位置するマクロブロックの動き補償用ベクトルＭＩを上位階層の動きベクトルとして選択するものであるが、さらに所定位置を含むフレームの各マクロブロックの動き補償用ベクトルＭＩの度数分布を生成し、度数の高い方から所定数の動きベクトルも上位階層の動きベクトルとして選択するようにしてもよい。これにより、利用動きベクトルＭＶ′の検出精度をさらに高めることができる。
【０３３８】
図２６に戻って、また、画像信号生成部５２１は、クラス生成回路５４４を有している。クラス生成回路５４４は、クラスタップ選択回路５４２で選択的に取り出されるクラスタップの画素データから時空間クラスを示すクラスコードＣＬを生成する。このクラス生成回路５４４は、クラスタップの画素データに１ビットのＡＤＲＣ等の処理を施すことによって、時空間クラスを示すクラスコードＣＬを生成する。このクラスコードＣＬは、画像信号Ｖｂの所定フレームにおける注目位置の画素データが属する時空間クラスを示すものとなる。
【０３３９】
また、画像信号生成部５２１は、係数メモリ５４５を有している。この係数メモリ５４５は、後述する推定予測演算回路５４６で使用される推定式で用いられる係数データＷｉ（ｉ＝１〜ｎ、ｎは予測タップの個数）を、クラス毎に、格納するものである。この係数メモリ５４５には、画像信号Ｖａを画像信号Ｖｂに変換するための係数データＷｉが格納されている。
【０３４０】
この係数メモリ５４５に格納される係数データＷｉは、予め画像信号Ｖａに対応した生徒信号と画像信号Ｖｂに対応した教師信号との間の学習によって生成される。この係数メモリ５４５には上述したクラス生成回路５４４で生成されたクラスコードＣＬが読み出しアドレス情報として供給される。この係数メモリ５４５からはクラスコードＣＬに対応した推定式の係数データＷｉが読み出されて、推定予測演算回路５４６に供給される。係数データＷｉの生成方法については後述する。
【０３４１】
また、画像信号生成部５２１は、予測タップ選択回路５４１で選択的に取り出される予測タップの画素データｘｉと、係数メモリ５４５より読み出される係数データＷｉとから、上述の（１）式の推定式によって、作成すべき画像信号Ｖｂの所定フレームにおける注目位置の画素データｙを演算する推定予測演算回路５４６を有している。
【０３４２】
この画像信号生成部５２１の動作を説明する。
クラスタップ選択回路５４２で、バッファメモリ５０８に格納されている画像信号Ｖａの連続する３つの利用フレームに基づいて、画像信号Ｖｂの所定フレーム（ｆ_１〜ｆ_４のいずれか）における注目位置に対して空間方向および時間方向の周辺に位置する複数の画素データが選択的に取り出される。
【０３４３】
この場合、動きベクトル検出部５４３で検出された、画像信号Ｖｂの所定フレームにおける注目位置の、上述した３つの利用フレームとの間の動きベクトルＭＶ１，ＭＶ２，ＭＶ３に基づいて、この３つの利用フレームに対して、画像信号Ｖｂの所定フレームを基準とした動き補償が行われる。
【０３４４】
クラスタップ選択回路５４２で取り出されるクラスタップの画素データはクラス生成回路５４４に供給される。このクラス生成回路５４４は、クラスタップの画素データに１ビットのＡＤＲＣ等の処理が施されて、時空間クラスを示すクラスコードＣＬを生成する。
【０３４５】
このクラスコードＣＬは、係数メモリ５４５に読み出しアドレス情報として供給される。係数メモリ５４５からはこのクラスコードＣＬに対応した係数データＷｉが読み出され、読み出された係数データＷｉは推定予測演算回路５４６に供給される。
【０３４６】
また、予測タップ選択回路５４１で、バッファメモリ５０８に格納されている画像信号Ｖａの連続する３つの利用フレームに基づいて、画像信号Ｖｂの所定フレームにおける注目位置に対して空間方向および時間方向の周辺に位置する複数の画素データが選択的に取り出される。
【０３４７】
この場合、動きベクトル検出部５４３で検出された、画像信号Ｖｂの所定フレームにおける注目位置の、上述した３つの利用フレームとの間の動きベクトルＭＶ１，ＭＶ２，ＭＶ３に基づいて、この３つの利用フレームに対して、画像信号Ｖｂの所定フレームを基準とした動き補償が行われる。
【０３４８】
推定予測演算回路５４６では、予測タップの画素データｘｉと、係数メモリ５４５より読み出される係数データＷｉとを用いて、上述の（１）式に示す推定式に基づいて、作成すべき画像信号Ｖｂの所定フレームにおける注目位置の画素データｙが求められる。
【０３４９】
このように画像信号生成部５２１では、画像信号Ｖａの連続する３つの利用フレームを用いて、画像信号Ｖｂの所定フレームが得られる。この場合、画像信号Ｖａの連続した３つの利用フレームに基づいて選択された、画像信号Ｖｂの所定フレームにおける注目位置の周辺に位置する複数の画素データ（予測タップの画素データ）、およびこの画像信号Ｖｂの所定フレームにおける注目位置の画素データが属するクラスＣＬに対応した係数データＷｉを用いて、推定式に基づいて画像信号Ｖｂの所定フレームにおける注目位置の画素データｙを生成するものである。
【０３５０】
この場合、予測タップの複数の画素データは、動きベクトル検出部５４３で検出された動きベクトルＭＶ（ＭＶ１〜ＭＶ３）に基づいて動き補償された、画像信号Ｖａの連続した３つの利用フレームに基づいて選択されるものであり、相関が高いものとなる。したがって、この予測タップの画素データを用いて生成される画像信号Ｖｂの所定フレームにおける注目位置の画素データの品質の向上を図ることができる。
【０３５１】
また、クラスタップの画素データは、動きベクトル検出部５４３で検出された動きベクトルＭＶ（ＭＶ１〜ＭＶ３）に基づいて動き補償された、画像信号Ｖａの連続した３つの利用フレームに基づいて選択されたものである。そのため、クラス生成回路５４４で生成されるクラスコードＣＬは、予測タップ選択回路５４１で動き補償が行われて選択された予測タップの画素データに対応した時空間クラスを精度よく示すものとなる。したがって、クラス分類の精度を高めることができ、ひいては画像信号Ｖｂにおける所定フレームの品質の向上を図ることができる。
【０３５２】
図２２に戻って、画像信号処理部５１０の動作を説明する。
バッファメモリ５０８に格納されている画像信号Ｖａの各ピクチャのフレームｆ_Ｉ，ｆ_Ｐ，ｆ_Ｂは、実際のフレームの順に読み出されて、出力選択回路５２２に供給される。
【０３５３】
また、画像信号生成部５２１は、バッファメモリ５０８に格納されている、画像信号Ｖａの連続した３つの利用フレームを用いて、各ピクチャのフレーム間に、フレームｆ_１〜ｆ_４を生成する。このフレームｆ_１〜ｆ_４は出力選択回路５２２に供給される。
【０３５４】
出力選択回路５２２では、各ピクチャのフレームｆ_Ｉ，ｆ_Ｐ，ｆ_Ｂおよび画像信号生成部５２１で生成されたフレームｆ_１〜ｆ_４とが選択的に取り出される。したがって、この出力選択回路５２２からは、画像信号Ｖａに対してフレーム数が５倍とされた画像信号Ｖｂが得られる。
【０３５５】
図２８は、画像信号生成部５２１の係数メモリ５４５に格納すべき係数データＷｉを生成する係数データ生成装置５７０の構成を示している。
この係数データ生成装置５７０は、画像信号Ｖｂに対応した教師信号ＳＴが入力される入力端子５７１と、この教師信号ＳＴに対してフレーム数を１／５に間引いた後に符号化を行ってＭＰＥＧ２ストリームを得るＭＰＥＧ２符号化器５７２と、このＭＰＥＧ２ストリームに対して復号化を行って画像信号Ｖａに対応した生徒信号ＳＳを得るＭＰＥＧ２復号化器５７３とを有している。ここで、ＭＰＥＧ２復号化器５７３は、図２２に示すデジタル放送受信機５００におけるＭＰＥＧ２復号化器５０７およびバッファメモリ５０８に対応したものである。
【０３５６】
また、係数データ生成装置５７０は、予測に使用する予測タップの複数の画素データを選択的に取り出す予測タップ選択回路５７４と、クラス分類に使用するクラスタップの複数の画素データを選択的に取り出すクラスタップ選択回路５７５とを有している。これらのタップ選択回路５７４，５７５は、上述した画像信号生成部５２１（図２６参照）のタップ選択回路５４１，５４２と同様に構成される。
【０３５７】
タップ選択回路５７４，５７５は、それぞれ、生徒信号ＳＳの連続する３つのフレームに基づいて、教師信号ＳＴの所定フレーム（ｆ_１〜ｆ_４のいずれか）における注目位置に対して空間方向（水平方向、垂直方向）および時間方向（フレーム方向）の周辺に位置する複数の画素データを選択的に取り出す。
【０３５８】
タップ選択回路５７４，５７５には、後述する動きベクトル検出部５７６で検出された、教師信号ＳＴの所定フレームにおける注目位置の、上述した３つのフレーム（以下、「利用フレーム」という）との間の動きベクトルＭＶ１，ＭＶ２，ＭＶ３が供給される。タップ選択回路５７４，５７５は、動きベクトルＭＶ１，ＭＶ２，ＭＶ３に基づき、３つの利用フレームに対して、教師信号ＳＴの所定フレームを基準とした動き補償を行う。
【０３５９】
また、係数データ生成装置５７０は、上述した動きベクトルＭＶ１，ＭＶ２，ＭＶ３を検出する動きベクトル検出部５７６を有している。この動きベクトル検出部５７６は、上述した画像信号生成部５２１の動きベクトル検出部５４３（図２７参照）と同様に構成される。この場合、動きベクトルＭＶ１〜ＭＶ３を検出する際に、利用動きベクトルＭＶ′を検出すると共に、この検出された利用動きベクトルＭＶ′を線形配分して動きベクトルＭＶ（ＭＶ１〜ＭＶ３）を求める。
【０３６０】
また、係数データ生成装置５７０は、クラス生成回路５７７を有している。クラス生成回路５７７は、クラスタップ選択回路５７５で選択的に取り出されるクラスタップの画素データから時空間クラスを示すクラスコードＣＬを生成する。このクラス生成回路５７７は、上述した画像信号生成部５２１のクラス生成回路５４４と同様に構成される。このクラスコードＣＬは、教師信号ＳＴの所定フレームにおける注目位置の画素データが属するクラスを示すものとなる。
【０３６１】
また、係数データ生成装置５７０は、入力端子５７１に入力される教師信号ＳＴの時間調整を行うための遅延回路５７８と、この遅延回路５７８で時間調整された教師信号ＳＴの所定フレームより得られる各注目位置の画素データｙと、この各注目位置の画素データｙにそれぞれ対応して予測タップ選択回路５７４で選択的に取り出される予測タップの画素データｘｉと、各注目位置の画素データｙにそれぞれ対応してクラス生成回路５７７で生成されたクラスコードＣＬとから、クラス毎に、係数データＷｉ（ｉ＝１〜ｎ）を得るための正規方程式（上述の（１０）式参照）を生成する正規方程式生成部５７９を有している。
【０３６２】
この場合、１個の画素データｙとそれに対応するｎ個の予測タップの画素データｘｉとの組み合わせで１個の学習データが生成されるが、教師信号ＳＴの所定フレームと生徒信号ＳＳとの間で、クラス毎に、多くの学習データが生成されていく。これにより、正規方程式生成部５７９では、クラス毎に、係数データＷｉ（ｉ＝１〜ｎ）を得るための正規方程式が生成される。
【０３６３】
また、係数データ生成装置５７０は、正規方程式生成部５７９で生成された正規方程式のデータが供給され、その正規方程式を解いて、各クラスの係数データＷｉを求める係数データ決定部５８０と、この求められた各クラスの係数データＷｉを格納する係数メモリ５８１とを有している。
【０３６４】
次に、図１８に示す係数データ生成装置５７０の動作を説明する。
入力端子５７１には画像信号Ｖｂに対応した教師信号ＳＴが供給される。この教師信号ＳＴに対して、ＭＰＥＧ２符号化器５７２でフレーム数が１／５に間引かれた後に符号化が施されて、ＭＰＥＧ２ストリームが生成される。このＭＰＥＧ２ストリームは、ＭＰＥＧ２復号化器５７３に供給される。ＭＰＥＧ２復号化器５７３で、このＭＰＥＧ２ストリームに対して復号化が施されて、画像信号Ｖａに対応した生徒信号ＳＳが生成される。
【０３６５】
予測タップ選択回路５７４およびクラスタップ選択回路５７５には、動きベクトル検出部５７６で検出された、教師信号ＳＴの所定フレームにおける注目位置の、生徒信号ＳＳの３つの利用フレームとの間の動きベクトルＭＶ１，ＭＶ２，ＭＶ３が供給される。
【０３６６】
復号化器５７３で生成された生徒信号ＳＳの連続する３つの利用フレームに基づいて、クラスタップ選択回路５７５で、教師信号ＳＴの所定フレームにおける注目位置に対して空間方向および時間方向の周辺に位置する複数の画素データが選択的に取り出される。この場合、動きベクトルＭＶ１，ＭＶ２，ＭＶ３に基づき、３つの利用フレームに対して、教師信号ＳＴの所定フレームを基準とした動き補償が行われる。
【０３６７】
クラスタップ選択回路５７５で選択的に取り出されたクラスタップの画素データはクラス生成回路５７７に供給される。このクラス生成回路５７７では、クラスタップの画素データに１ビットのＡＤＲＣ等の処理が施されて、時空間クラスを示すクラスコードＣＬが生成される。
【０３６８】
また、復号化器５７３で生成された生徒信号ＳＳの連続する３つの利用フレームに基づいて、予測タップ選択回路５７４で、教師信号ＳＴの所定フレームにおける注目位置に対して空間方向および時間方向の周辺に位置する複数の画素データが選択的に取り出される。この場合、動きベクトルＭＶ１，ＭＶ２，ＭＶ３に基づき、３つの利用フレームに対して、教師信号ＳＴの所定フレームを基準とした動き補償が行われる。
【０３６９】
そして、遅延回路５７８で時間調整された教師信号ＳＴの所定フレームから得られる各注目位置の画素データｙと、この各注目位置の画素データｙにそれぞれ対応して予測タップ選択回路５７４で選択的に取り出される予測タップの画素データｘｉと、各注目位置の画素データｙにそれぞれ対応してクラス生成回路５７７で生成されたクラスコードＣＬとを用いて、正規方程式生成部５７９では、クラス毎に、係数データＷｉ（ｉ＝１〜ｎ）を得るための正規方程式（（１０）式参照）が生成される。この正規方程式は係数データ決定部５８０で解かれて各クラスの係数データＷｉが求められ、その係数データＷｉは係数メモリ５８１に格納される。
【０３７０】
このように、図２８に示す係数データ生成装置５７０においては、画像信号生成部５２１の係数メモリ５４５に格納される各クラスの係数データＷｉを生成することができる。
【０３７１】
なお、図２２の画像信号処理部５１０における処理を、例えば図１９に示すような画像信号処理装置３００によって、ソフトウェアで実現することも可能である。図２９のフローチャートを参照して、図１９に示す画像信号処理装置３００における、画像信号Ｖａより画像信号Ｖｂを得るため処理手順を説明する。
【０３７２】
まず、ステップＳＴ５１で、処理を開始し、ステップＳ５２で、例えば入力端子３１４より装置内に複数フレーム分の画像信号Ｖａを入力する。この場合、画像信号Ｖａの画素データと対となっている動き補償用ベクトルＭＩも入力する。
【０３７３】
このように入力端子３１４より入力される画像信号Ｖａ等はＲＡＭ３０３に一時的に格納される。なお、この画像信号Ｖａ等が装置内のハードディスクドライブ３０５に予め記録されている場合には、このドライブ３０５からこの画像信号Ｖａ等を読み出し、この画像信号Ｖａ等をＲＡＭ３０３に一時的に格納する。
【０３７４】
そして、ステップＳＴ５３で、画像信号Ｖａの全フレームの処理が終わっているか否かを判定する。処理が終わっているときは、ステップＳＴ５４で、処理を終了する。一方、処理が終わっていないときは、ステップＳＴ５５に進む。
【０３７５】
このステップＳＴ５５では、入力された画像信号Ｖａのフレームｆ_ｎを出力端子３１５に出力する。その後に、ステップＳＴ５６に進み、この画像信号Ｖａのフレームｆ_ｎとその次のフレーム_ｎ＋１との間に挿入すべきフレームｆ_１〜ｆ_４の生成処理に移る。ステップＳＴ５６では、Ｎ＝１に設定する。
【０３７６】
次に、ステップＳＴ５７で、動き補償用ベクトルＭＩを用いて、画像信号Ｖｂのフレームｆ_Ｎにおける注目位置の、画像信号Ｖａの連続した３つの利用フレームとの間の動きベクトルＭＶ１〜ＭＶ３を検出する。フレームｆ_Ｎは、最初はｆ_１である。
【０３７７】
そして、ステップＳＴ５８で、ステップＳＴ５２で入力された画像信号Ｖａの連続した３つの利用フレームに基づいて、画像信号Ｖｂのフレームｆ_Ｎにおける注目位置に対して時間方向および空間方向の周辺に位置する複数の画素データ（クラスタップの画素データ）を取得し、このクラスタップの画素データを用いて、画像信号Ｖｂのフレームｆ_Ｎにおける注目位置の画素データが属するクラスを示すクラスコードＣＬを生成する。
【０３７８】
次に、ステップＳＴ５９で、ステップＳＴ５２で入力された画像信号Ｖａの連続した３つの利用フレームに基づいて、画像信号Ｖｂのフレームｆ_Ｎにおける注目位置に対して時間方向および空間方向の周辺に位置する複数の画素データ（予測タップの画素データ）を取得する。
【０３７９】
なお、ステップＳＴ５８およびステップＳＴ５９では、画像信号Ｖａの３つの利用フレームが、ステップＳＴ５７で検出された動きベクトルＭＶ１〜ＭＶ３に基づき、フレームｆ_Ｎを基準として動き補償された後に、画素データの取得が行われる。
【０３８０】
次に、ステップＳＴ６０で、ステップＳＴ５８で生成されたクラスコードＣＬに対応した係数データＷｉとステップＳＴ５９で取得された予測タップの画素データｘｉを使用して、（１）式の推定式に基づいて、画像信号Ｖｂにおける注目位置の画素データｙを生成する。
【０３８１】
そして、ステップＳＴ６１で、フレームｆ_Ｎを生成する処理が終了したか否かを判定する。終了していないときは、ステップＳＴ５７に戻り、次の注目位置についての処理に移る。一方、処理が終了しているときは、ステップＳＴ６２に進み、フレームｆ_Ｎを出力端子３１５に出力する。
【０３８２】
次に、ステップＳＴ６３で、Ｎ＝Ｎｍａｘであるか否かを判定する。ここで、Ｎｍａｘ＝４である。Ｎ＝Ｎｍａｘでないときは、ステップＳＴ６４で、Ｎをインクリメントし、その後にステップＳＴ５７に戻って、次のフレームｆ_Ｎ＋１の生成処理に移る。一方、Ｎ＝Ｎｍａｘであるときは、フレームｆ_１〜ｆ_４を生成し、その出力が終了したことを意味するので、ステップＳＴ６５に進む。
【０３８３】
ステップＳＴ６５では、ステップＳＴ５２で入力された複数フレーム分の画像信号Ｖａに対応した処理が終了したか否かを判定する。終了しているときは、ステップＳＴ５２に戻り、次の複数フレーム分の画像信号Ｖａの入力処理に移る。一方、処理が終了していないときは、ステップＳＴ５５に戻って、画像信号Ｖａの次のフレームを出力端子３１５に出力し、そのフレームとその次のフレームとの間に挿入すべきフレームｆ_１〜ｆ_４の生成処理に移る。
【０３８４】
このように、図２９に示すフローチャートに沿って処理をすることで、入力された画像信号Ｖａを処理して、５倍のフレーム数を有する画像信号Ｖｂを得ることができる。
【０３８５】
また、処理装置の図示は省略するが、図２８の係数データ生成装置５７０における処理も、ソフトウェアで実現可能である。
【０３８６】
図３０のフローチャートを参照して、係数データを生成するための処理手順を説明する。
まず、ステップＳＴ７１で、処理を開始し、ステップＳＴ７２で、教師信号ＳＴを複数フレーム分だけ入力する。そして、ステップＳＴ７３で、教師信号ＳＴの全フレームの処理が終了したか否かを判定する。終了していないときは、ステップＳＴ７４で、ステップＳＴ７２で入力された教師信号ＳＴから生徒信号ＳＳを生成する。この場合、生徒信号ＳＳの画素データと対となっている動き補償用ベクトルＭＩも得る。
【０３８７】
そして、ステップＳＴ７５で、動き補償用ベクトルＭＩを用いて、教師信号ＳＴの所定フレーム（画像信号Ｖｂのフレームｆ_１〜ｆ_４のいずれかに対応）における注目位置の、生徒信号ＳＳの連続した３つの利用フレームとの間の動きベクトルＭＶ１〜ＭＶ３を検出する。
【０３８８】
そして、ステップＳＴ７６で、ステップＳＴ７４で生成された生徒信号ＳＳの連続した３つの利用フレームに基づいて、教師信号ＳＴの所定フレームにおける注目位置に対して時間方向および空間方向の周辺に位置する複数の画素データ（クラスタップの画素データ）を取得し、このクラスタップの画素データを用いて、教師信号ＳＴの所定フレームにおける注目位置の画素データが属するクラスを示すクラスコードＣＬを生成する。
【０３８９】
次に、ステップＳＴ７７で、ステップＳＴ７４で生成された生徒信号ＳＳの連続した３つの利用フレームに基づいて、教師信号ＳＴの所定フレームにおける注目位置に対して時間方向および空間方向の周辺に位置する複数の画素データ（予測タップの画素データ）を取得する。
【０３９０】
なお、ステップＳＴ７６およびステップＳＴ７７では、生徒信号ＳＳの連続した３つの利用フレームが、ステップＳＴ７５で検出された動きベクトルＭＶ１〜ＭＶ３に基づき、教師信号ＳＴの所定フレームを基準として動き補償された後に、画素データの取得が行われる。
【０３９１】
次に、ステップＳＴ７８で、ステップＳＴ７６で生成されたクラスコードＣＬ、ステップＳＴ７７で取得された予測タップの画素データｘｉおよび教師信号ＳＴの所定フレームにおける注目位置の画素データｙを用いて、クラス毎に、（１０）式に示す正規方程式を得るための加算をする（（８）式、（９）式参照）。
【０３９２】
次に、ステップＳＴ７９で、ステップＳＴ７２で入力された複数フレーム分の教師信号ＳＴに対応した学習処理が終了したか否かを判定する。学習処理を終了しているときは、ステップＳＴ７２に戻って、次の複数フレーム分の教師信号ＳＴの入力を行って、上述したと同様の処理を繰り返す。一方、学習処理を終了していないときは、ステップＳＴ７５に戻って、次の注目位置についての処理に移る。
【０３９３】
上述したステップＳＴ７３で、処理が終了したときは、ステップＳＴ８０で、上述のステップＳＴ７８の加算処理によって生成された、各クラスの正規方程式を掃き出し法などで解いて、各クラスの係数データＷｉを算出する。そして、ステップＳＴ８１で、各クラスの係数データＷｉをメモリに保存し、その後にステップＳＴ８２で、処理を終了する。
【０３９４】
このように、図３０に示すフローチャートに沿って処理をすることで、図２８に示す係数データ生成装置５７０と同様の手法によって、各クラスの係数データＷｉを得ることができる。
【０３９５】
なお、上述実施の形態においては、ＤＣＴを伴うＭＰＥＧ２ストリームを取り扱うものを示したが、この発明は、その他の符号化されたデジタル情報信号を取り扱うものにも同様に適用することができる。また、ＤＣＴの代わりに、ウォーブレット変換、離散サイン変換などのその他の直交変換を伴う符号化であってもよい。
【０３９６】
また、上述実施の形態において、動きベクトル検出部１４３（図１３参照）、動きベクトル検出部５４３（図２７参照）では、相関情報としての評価値を差分絶対値和としたものであるが、差分二乗和であってもよい。
【０３９７】
【発明の効果】
この発明によれば、複数のブロックに分割され、各ブロック毎に第２のフレームとの間の動きベクトルを有する第１のフレームにおける所定位置の動きベクトルを検出する際に、所定位置の周辺に位置する複数のブロックの動きベクトルを上位階層の動きベクトルとして選択し、第１のフレームの所定位置を中心とする参照ブロックと第２のフレームの所定位置に対応した位置が選択された複数の動きベクトルに基づいて移動された位置をそれぞれ基準位置とした複数の探索範囲内の候補ブロックとの間の相関情報を検出し、参照ブロックと最も相関の高い候補ブロックの位置情報を所定位置の動きベクトルとするものであり、所定位置の動きベクトルを素早く、かつ精度よく検出できる。
【０３９８】
また、この発明によれば、符号化されたデジタル画像信号を復号化することによって生成される、複数の画素データからなる第１の画像信号を、複数の画素データからなる第２の画像信号に変換して符号化雑音を軽減する際、第１の画像信号の他に、上述した動きベクトル検出装置等を用いて検出された動きベクトルにより動き補償された、符号化雑音軽減後の第２の画像信号に基づいて、第２の画像信号における注目位置に対して空間方向および時間方向の周辺に位置する複数の第１の画素データを選択し、この複数の第１の画素データを用いて第２の画像信号における注目位置の画素データを生成するものであり、第２の画像信号の品質の向上を図ることができる。
【０３９９】
また、この発明によれば、複数の画素データからなる第１の画像信号を、この第１の画像信号とは異なるフレーム数を有する、複数の画素データからなる第２の画像信号に変換する際、上述した動きベクトル検出装置等を用いて検出された動きベクトルにより動き補償された、第１の画像信号の連続する複数フレームに基づいて、第２の画像信号の所定フレームにおける注目位置に対して空間方向および時間方向の周辺に位置する複数の第１の画素データを選択し、この複数の第１の画素データを用いて第２の画像信号の所定フレームにおける注目位置の画素データを生成するものであり、第２の画像信号の品質の向上を図ることができる。
【図面の簡単な説明】
【図１】第１の実施の形態としてのデジタル放送受信機の構成を示すブロック図である。
【図２】ＭＰＥＧ２復号化器の構成を示すブロック図である。
【図３】第１の処理部の構成を示すブロック図である。
【図４】第２の処理部の構成を示すブロック図である。
【図５】動き補償を説明するための図である。
【図６】ブロックマッチング法を説明するための図である。
【図７】ブロックマッチング法を説明するための図である。
【図８】ブロックマッチング法を説明するための図である。
【図９】ブロックマッチング法を説明するための図である。
【図１０】上位階層の動きベクトルの選択を説明するための図である。
【図１１】上位階層の動きベクトルの選択を説明するための図である。
【図１２】動きベクトル検出処理を説明するためのフローチャートである。
【図１３】動きベクトル検出部の構成を示すブロック図である。
【図１４】評価値算出回路の構成を示すブロック図である。
【図１５】動きベクトル検出回路の構成を示すブロック図である。
【図１６】動きベクトル検出部の変形例を説明するためのブロック図である。
【図１７】係数データ生成装置の構成を示すブロック図である。
【図１８】係数データ生成装置の構成を示すブロック図である。
【図１９】ソフトウェアで実現するための画像信号処理装置の構成例を示すブロック図である。
【図２０】画像信号処理を示すフローチャートである。
【図２１】係数データ生成処理を示すフローチャートである。
【図２２】第２の実施の形態としてのデジタル放送受信機の構成を示すブロック図である。
【図２３】画像信号Ｖａ，Ｖｂのフレーム構成を説明するための図である。
【図２４】動きベクトルＭＶ１〜ＭＶ３を説明するための図である。
【図２５】利用動きベクトルＭＶ′（ＭＶＩ，ＭＶＰ）を説明するための図である。
【図２６】画像信号生成部の構成を示すブロック図である。
【図２７】動きベクトル検出部の構成を示すブロック図である。
【図２８】係数データ生成装置の構成を示すブロック図である。
【図２９】画像信号処理を示すフローチャートである。
【図３０】係数データ生成処理を示すフローチャートである。
【符号の説明】
２１，２８・・・動きベクトル選択回路、２２・・・動きベクトル蓄積部、２３・・・参照ブロック回路、２４・・・サーチブロック回路、２５・・・評価値回路、２６・・・動きベクトル検出回路、２７・・・度数分布生成回路、２９・・・線形配分回路、１００・・・デジタル放送受信機、１０１・・・システムコントローラ、１０２・・・リモコン信号受信回路、１０５・・・受信アンテナ、１０６・・・チューナ部、１０７・・・ＭＰＥＧ２復号化器、１０８，１２３，１２５・・・バッファメモリ、１１０・・・画像信号処理部、１１１・・・ディスプレイ部、１２１・・・第１の処理部、１２２・・・第２の処理部、１２４・・・並べ換え部、１３１，１４１・・・予測タップ選択回路、１３２，１４２・・・クラスタップ選択回路、１３３，１４４，１４５・・・クラス生成回路、１３４，１４７・・・係数メモリ、１３５，１４８・・・推定予測演算回路、１４３・・・動きベクトル検出部、１４６・・・クラス統合回路、１５０，１７０，５７０・・・係数データ生成装置、３００・・・画像信号処理装置、５００・・・デジタル放送受信機、５０１・・・システムコントローラ、５０２・・・リモコン信号受信回路、５０５・・・受信アンテナ、５０６・・・チューナ部、５０７・・・ＭＰＥＧ２復号化器、５０８・・・バッファメモリ、５１０・・・画像信号処理部、５１１・・・ディスプレイ部、５２１・・・画像信号生成部、５２２・・・出力選択回路、５４１・・・予測タップ選択回路、５４２・・・クラスタップ選択回路、５４３・・・動きベクトル検出部、５４４・・・クラス生成回路、５４５・・・係数メモリ、５４６・・・推定予測演算回路[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a motion vector detecting device and a detecting method, an image signal processing device and a processing method using the same, a coefficient data generating device and a generating method used therein, and a program for executing each method.
[0002]
More specifically, the present invention is characterized in that when detecting a motion vector at a predetermined position in a first frame that is divided into a plurality of blocks and has a motion vector between each block and a second frame, the detection method is performed around a predetermined position. Are selected as the motion vector of the upper layer, and the reference block centering on the predetermined position of the first frame and the position corresponding to the predetermined position of the second frame are selected. Correlation information between candidate blocks in a plurality of search ranges, each of which is a reference position based on the position moved based on the motion vector, is detected, and the position information of the candidate block having the highest correlation with the reference block is calculated at a predetermined position. The present invention relates to a motion vector detecting device or the like which can quickly and accurately detect a motion vector at a predetermined position by using a vector. A.
[0003]
In addition, the present invention converts a first image signal composed of a plurality of pixel data, which is generated by decoding an encoded digital image signal, into a second image signal composed of a plurality of pixel data. When the coding noise is reduced by using the above-described motion vector detecting device or the like, in addition to the first image signal, the second image signal after the coding noise reduction, , A plurality of first pixel data located in the space direction and the time direction around the target position in the second image signal are selected, and the second pixel data is selected using the plurality of first pixel data. The present invention relates to an image signal processing device and the like which improve the quality of a second image signal by generating pixel data of a target position in an image signal.
[0004]
Further, according to the present invention, when converting a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal, Based on a plurality of continuous frames of the first image signal, motion-compensated by the motion vector detected using the motion vector detecting device or the like, in the spatial direction with respect to the target position in the predetermined frame of the second image signal. And selecting a plurality of first pixel data located in the periphery in the time direction, and generating pixel data of a target position in a predetermined frame of the second image signal using the plurality of first pixel data, The present invention relates to an image signal processing device and the like which improve the quality of a second image signal.
[0005]
[Prior art]
As a compression coding method of an image signal, there is a coding method based on MPEG2 (Moving Picture Experts Group 2) using DCT (Discrete Cosine Transform). In this encoding method, motion compensation prediction encoding is performed for each block.
[0006]
The DCT performs discrete cosine transform on pixels in a block, requantizes coefficient data obtained by the discrete cosine transform, and further performs variable length encoding on the requantized coefficient data. For this variable length coding, entropy coding such as Huffman coding is often used. The image signal is divided into a large number of frequency data from low frequency to high frequency by orthogonal transformation.
[0007]
When performing requantization on the divided frequency data, in consideration of human visual characteristics, low-frequency data with high importance is finely quantized, and high-frequency data with low importance is coarsely quantized. By performing the conversion, it is possible to maintain high image quality and to realize efficient compression.
[0008]
In conventional decoding using DCT, reproduced data is obtained by converting quantized data for each frequency component into a representative value of the code and performing inverse DCT (IDCT: Inverse DCT) on those components. . When converting to this representative value, the quantization step width at the time of encoding is used.
[0009]
[Problems to be solved by the invention]
As described above, the encoding method based on MPEG using DCT has the advantage that high-quality images can be maintained and high-efficiency compression can be realized by performing encoding in consideration of human visual characteristics.
[0010]
However, since the encoding that performs DCT is a process in units of blocks, block-like noise, so-called block noise (block distortion), may occur as the compression ratio increases. Also, in a portion where there is a sharp change in luminance such as an edge, a rough noise caused by coarsely quantizing a high-frequency component, so-called mosquito noise, is generated.
[0011]
It is conceivable to remove coding noise such as block noise and mosquito noise by a class classification adaptive process. That is, an image signal containing coding noise is set as a first image signal, an image signal from which coding noise is removed is set as a second image signal, and a class to which pixel data of a target position in the second image signal belongs is detected. Then, corresponding to this class, the pixel data of the target position in the second image signal is generated.
[0012]
In this case, based on the second frame of the second image signal in which the coding noise has been reduced, in addition to the first frame of the first image signal, the position of interest in the second image signal is temporally shifted. A plurality of pixel data located in the periphery in the direction and the spatial direction are selected, and pixel data at a position of interest in the second image signal is generated using the plurality of pixel data to improve the quality of the second image signal. It is possible to plan.
[0013]
In this case, it is necessary to detect a motion vector of the first frame at a predetermined position corresponding to the target position in the second image signal, and to perform motion compensation of the second frame of the second image signal using the motion vector. is there.
[0014]
When detecting a motion vector at a predetermined position corresponding to the target position, a motion compensation vector added to a macroblock in which the predetermined position exists is used, and the motion compensation vector is used as a higher-layer motion vector. It is conceivable to detect the motion vector hierarchically by using the motion vector.
[0015]
In this case, the processing for generating the pixel data of the upper layer and the processing of detecting the motion vector of the upper layer from the pixel data of the upper layer are unnecessary, so that the processing time of the motion vector detection can be shortened.
[0016]
However, the motion compensation vector is detected and added for each macroblock. Therefore, when a plurality of motion vectors exist in a macroblock, the motion compensation vector of a certain macroblock does not match all pixel positions in the macroblock. Depending on the predetermined position, the motion vector of a macroblock adjacent to the macroblock in which the predetermined position exists may be more suitable for the predetermined position.
[0017]
Therefore, when detecting a motion vector at a predetermined position corresponding to the target position in the second image signal, only the motion compensation vector added to the macroblock in which the predetermined position exists is set as a higher-layer motion vector. When the motion vector is detected in a targeted manner, the accuracy of the detected motion vector is reduced, and as a result, it is difficult to improve the quality of the second image signal.
[0018]
It should be noted that the classification adaptation process may be applied to the case where the first image signal is converted into a second image signal having a different number of frames from the first image signal. In this case, based on a plurality of continuous frames of the first image signal, a plurality of pieces of pixel data located in the temporal and spatial directions around the target position in the predetermined frame of the second image signal are selected. It is conceivable to generate pixel data of a target position in a predetermined frame of the second image signal using a plurality of pieces of pixel data to improve the quality of the second image signal.
[0019]
In this case, a motion vector between a plurality of frames of the first image signal and a plurality of frames of the first image signal, which are related to the position of interest in the predetermined frame of the second image signal, is detected. It is necessary to perform frame motion compensation. When the motion vector is detected by using the motion compensation vector added to the macroblock as described above, the processing time for the detection can be shortened, but the accuracy of the detected motion vector is reduced, and Deterioration of the quality of the image signal 2 is a problem.
[0020]
An object of the present invention is to enable a motion vector at a predetermined position to be detected quickly and accurately. Further, an object of the present invention is to convert a first image signal generated by decoding an encoded digital image signal into a second image signal to reduce coding noise, An object of the present invention is to improve the quality of an image signal. Another object of the present invention is to improve the quality of a second image signal when converting the first image signal into a second image signal having a different number of frames from the first image signal. is there.
[0021]
[Means for Solving the Problems]
A motion vector detection device according to the present invention is configured to detect a motion vector at a predetermined position in a first frame that is divided into a plurality of blocks and has a motion vector between the blocks and a second frame. A motion vector selecting means for selecting a motion vector of a plurality of blocks located around a predetermined position as a motion vector of an upper layer, a reference block centered on a predetermined position of the first frame, and a second frame. Is detected based on a plurality of motion vectors selected by the motion vector selection means at a position corresponding to the predetermined position, and correlation information between the reference blocks and candidate blocks within a plurality of search ranges is detected. A reference block and a reference block are identified based on correlation detection means and correlation information corresponding to a plurality of candidate blocks detected by the correlation detection means. The position information of high correlation candidate block is one and a motion vector detecting means for the motion vector of a predetermined position.
[0022]
Further, the motion vector detecting method according to the present invention is a motion vector detecting a motion vector at a predetermined position in a first frame which is divided into a plurality of blocks and has a motion vector between each block and a second frame. A first step of selecting a motion vector of a plurality of blocks located around a predetermined position as a motion vector of an upper layer, and a reference block centered on the predetermined position of the first frame and a second block. The correlation information between candidate positions in a plurality of search ranges, each of which is a position corresponding to a predetermined position of the frame of the target frame based on the plurality of motion vectors selected in the first step, is set as a reference position. Based on the second step to be detected and the correlation information corresponding to the plurality of candidate blocks detected in the second step, the reference block and the The position information of the high candidate block of function, in which a third step of a motion vector at a predetermined position.
[0023]
A program according to the present invention is for causing a computer to execute the above-described motion vector detection method.
[0024]
According to the present invention, a motion vector between a predetermined position in the first frame and a second frame is detected. The first frame is divided into a plurality of blocks, and each block has a motion vector between itself and the second frame.
[0025]
The motion vectors of a plurality of blocks located around the predetermined position in the first frame are selected as the motion vectors of the upper hierarchy. Here, the plurality of blocks located around the predetermined position include a block including the predetermined position. The periphery is around either or both of the spatial direction (horizontal direction and vertical direction) and the time direction (frame direction).
[0026]
A reference block centered on a predetermined position of the first frame is set. Further, a plurality of search ranges are set based on the plurality of motion vectors selected as described above for the position corresponding to the predetermined position of the second frame. Then, correlation information between the reference block and the candidate blocks in the plurality of search ranges is detected.
[0027]
Then, a motion vector is detected based on the correlation information corresponding to the plurality of candidate blocks detected by the correlation detecting means. In this case, the position information of the candidate block having the highest correlation with the reference block is set as the motion vector having the predetermined value.
[0028]
As described above, a motion vector at a predetermined position is hierarchically detected by using a motion vector provided for each block as a motion vector of an upper layer, and a motion vector can be quickly detected. In addition, not only the motion vector of the block in which the predetermined position exists but also the motion vector of the block around the block is used as the motion vector of the upper layer, and the motion vector at the predetermined position can be accurately detected.
[0029]
Note that a frequency distribution generating unit that generates a frequency distribution of a motion vector of each block of the first frame is further provided, and the motion vector selecting unit further includes a frequency distribution generating unit that generates a frequency distribution based on the frequency distribution generated by the frequency distribution generating unit. A predetermined number of motion vectors from the highest one may be selected as the motion vector of the upper hierarchy.
[0030]
It is also conceivable that a motion vector suitable for the predetermined position does not exist among the motion vectors of a plurality of blocks located around the predetermined position. In this case, as described above, there may be a case where a motion vector selected at a predetermined position is included in the motion vectors selected based on the frequency distribution. Therefore, by adding the motion vector selected based on the frequency distribution to the motion vector of the upper layer as described above, it is possible to further improve the detection accuracy of the motion vector at the predetermined position.
[0031]
An image signal processing apparatus according to the present invention converts a first image signal composed of a plurality of pixel data, which is generated by decoding an encoded digital image signal, into a second image composed of a plurality of pixel data. An image signal processing apparatus for converting a signal into a plurality of blocks, wherein a first frame having a motion vector between each block and a second frame is a target position in the second image signal. A motion vector detecting unit that detects a motion vector at a predetermined position corresponding to the first image signal, and a second image signal that has been motion-compensated by the first frame of the first image signal and the motion vector detected by the motion vector detecting unit. A plurality of first pixel data located around the position of interest in the second image signal in the spatial direction and the temporal direction based on the second frame And over data selecting means, by using a plurality of first pixel data selected by the data selection means, in which and a pixel data generation means for generating pixel data of the target position in the second image signal.
[0032]
The motion vector detection unit includes a motion vector selection unit that selects a motion vector of a plurality of blocks located around the predetermined position as a motion vector of an upper layer, and a reference block centered on the predetermined position of the first frame. Correlation between a position corresponding to a predetermined position of the second frame based on a plurality of motion vectors selected by the motion vector selection means and a plurality of candidate blocks in a plurality of search ranges, each of which is a reference position. Based on correlation detection means for detecting information and correlation information corresponding to a plurality of candidate blocks detected by the correlation detection means, position information of a candidate block having the highest correlation with the reference block is set as a motion vector at a predetermined position. And a motion vector detecting means.
[0033]
Also, the image signal processing method according to the present invention converts the first image signal composed of a plurality of pixel data generated by decoding the encoded digital image signal into a second image signal composed of a plurality of pixel data. An image signal processing method for converting an image signal into a plurality of blocks, wherein a first frame having a motion vector between the plurality of blocks and a second frame for each block is focused on a second image signal. A first step of detecting a motion vector at a predetermined position corresponding to the position, and a second image in which motion compensation has been performed using the first frame of the first image signal and the motion vector detected in the first step. A second selecting unit that selects a plurality of first pixel data located in a space direction and a time direction around the position of interest in the second image signal based on the second frame of the signal; Step a, using the second plurality of first pixel data selected in step, in which a third step of generating pixel data of the target position in the second image signal.
[0034]
Then, the first step is a step of selecting the motion vectors of a plurality of blocks located around the predetermined position as the motion vectors of the upper layer, and the step of selecting the reference block centered on the predetermined position of the first frame and the second Detecting correlation information between candidate blocks in a plurality of search ranges each having a position corresponding to a predetermined position of the frame based on the plurality of motion vectors selected as a reference position, And setting the position information of the candidate block having the highest correlation with the reference block as a motion vector at a predetermined position based on the obtained correlation information corresponding to the plurality of candidate blocks.
[0035]
Further, a program according to the present invention causes a computer to execute the above-described image signal processing method.
[0036]
According to the present invention, a first image signal including a plurality of pixel data is converted into a second image signal including a plurality of pixel data. The first image signal is generated by decoding an encoded digital image signal. For example, a digital image signal has been subjected to MPEG coding.
[0037]
The first frame of the first image signal is divided into a plurality of blocks, and each block has a motion vector between the first frame and the second frame. In the first frame of the first image signal, a motion vector at a predetermined position corresponding to the target position in the second image signal is detected.
[0038]
Detection of this motion vector is performed by the above-described motion vector detection device and the like, using not only the motion vector of the block where the predetermined position exists but also the motion vector of a block around this block as a higher-layer motion vector. It is detected with high accuracy hierarchically.
[0039]
Based on the first frame of the first image signal and the second frame of the second image signal motion-compensated by the detected motion vector, the time direction and A plurality of first pixel data located on the periphery in the spatial direction are selected. Then, the pixel data of the target position in the second image signal is generated using the plurality of first pixel data.
[0040]
As described above, the pixel data of the target position in the second image signal is generated using the plurality of first image data, and the second image signal in which the coding noise is reduced can be obtained. it can.
[0041]
For example, the class to which the pixel data of the target position in the second image signal belongs is detected. Then, pixel data of the target position in the second image signal is calculated based on the estimation formula using the coefficient data generated in accordance with the detected class and the plurality of first pixel data. Learning using, as coefficient data, a student signal corresponding to the first image signal and including the same coding noise as the first image signal and a teacher signal corresponding to the second image signal and including no coding noise. By using the coefficient data obtained by the above, it is possible to obtain a second image signal in which coding noise is reduced as compared with the first image signal.
[0042]
Here, based on the first frame of the first image signal and the second frame of the second image signal motion-compensated by the detected motion vector, a position corresponding to the target position in the second image signal is determined. A plurality of second pixel data located around the time direction and the spatial direction are selected, and a class to which the pixel data of the target position in the second image signal belongs is detected using at least the second pixel data. . In this case, the plurality of second pixel data are selected not only from the first image signal but also from the second image signal in which coding noise has been reduced, and the accuracy of class classification can be improved. Thus, the quality of the second image signal can be improved.
[0043]
Further, at least, obtained when detecting a motion vector at a predetermined position, using the position information from the reference position of the search range of the candidate block having the highest correlation with the reference block or the correlation information of the candidate block with the highest correlation The class to which the pixel data at the target position in the second image signal belongs is detected. By using information related to a motion vector obtained by motion-compensating the second frame of the second image signal, accurate class classification can be performed.
[0044]
Further, the motion vector at the predetermined position is detected hierarchically using not only the motion vector of the block where the predetermined position exists but also the motion vector of a block around the block as a higher-level motion vector. Yes, a motion vector at a predetermined position is accurately detected. Then, the plurality of first pixel data include, in addition to the first frame of the first image signal, the second image signal of the second image signal after the reduction of the coding noise that has been motion-compensated with the detected motion vector. It is also selected from the frame.
[0045]
By performing motion compensation on the second image signal using the motion compensation vector detected as described above, a plurality of first pixel data selected from the first frame and the second frame existing in the time direction are obtained. The correlation is high. Therefore, it is possible to improve the quality of the pixel data at the position of interest in the second image signal generated using the plurality of first pixel data.
[0046]
A coefficient data generating apparatus according to the present invention converts a first image signal composed of a plurality of pixel data, which is generated by decoding an encoded digital image signal, into a second image composed of a plurality of pixel data. An apparatus for generating coefficient data of an estimation formula used when converting into a signal, the method comprising: decoding a digital image signal obtained by encoding a teacher signal corresponding to a second image signal; A decoding means for obtaining a student signal corresponding to the image signal; and a decoding means corresponding to a target position in the teacher signal of a first frame divided into a plurality of blocks and having a motion vector between each block and a second frame. The motion compensation is performed by the motion vector detection unit that detects the motion vector at the predetermined position and the motion vector detected by the first frame of the student signal and the motion vector detection unit. Data selecting means for selecting a plurality of pixel data located in the spatial and temporal directions with respect to the position of interest in the teacher signal based on the second frame of the teacher signal; and a plurality of pixel data selected by the data selecting means. And calculating means for calculating coefficient data using the pixel data of the position of interest in the teacher signal.
[0047]
The motion vector detection unit includes a motion vector selection unit that selects a motion vector of a plurality of blocks located around the predetermined position as a motion vector of an upper layer, and a reference block centered on the predetermined position of the first frame. Correlation between a position corresponding to a predetermined position of the second frame based on a plurality of motion vectors selected by the motion vector selection means and a plurality of candidate blocks in a plurality of search ranges, each of which is a reference position. Based on correlation detection means for detecting information and correlation information corresponding to a plurality of candidate blocks detected by the correlation detection means, position information of a candidate block having the highest correlation with the reference block is set as a motion vector at a predetermined position. And a motion vector detecting means.
[0048]
Further, the coefficient data generating method according to the present invention converts a first image signal composed of a plurality of pixel data, which is generated by decoding an encoded digital image signal, into a second image composed of a plurality of pixel data. A method for generating coefficient data of an estimation formula used when converting the image signal into an image signal, the method comprising: decoding a digital image signal obtained by encoding a teacher signal corresponding to a second image signal; A first step of obtaining a student signal corresponding to one image signal; and a notice in the teacher signal of a first frame divided into a plurality of blocks and having a motion vector between each block and a second frame. A second step of detecting a motion vector at a predetermined position corresponding to the position, and motion compensation is performed by the first frame of the student signal and the motion vector detected in the second step. A third step of selecting, based on the second frame of the teacher signal, a plurality of pieces of pixel data located in the spatial and temporal directions around the target position in the teacher signal; and selecting the pixel data in the third step. And a fourth step of obtaining coefficient data using the plurality of pixel data and the pixel data at the position of interest in the teacher signal.
[0049]
Then, the second step is a step of selecting motion vectors of a plurality of blocks located around the predetermined position as motion vectors of an upper layer, and a step of selecting a reference block centered on the predetermined position of the first frame and a second block. Detecting correlation information between candidate blocks in a plurality of search ranges each of which has a position corresponding to a predetermined position of the frame based on the plurality of selected motion vectors as a reference position, Setting the position information of the candidate block having the highest correlation with the reference block based on the correlation information corresponding to the plurality of detected candidate blocks as a motion vector at a predetermined position.
[0050]
Further, a program according to the present invention is for causing a computer to execute the above-described coefficient data generation method.
[0051]
According to the present invention, coefficient data of an estimation expression used when converting a first image signal including a plurality of pixel data into a second image signal including a plurality of pixel data is generated. Here, the first image signal is generated by decoding the encoded digital image signal.
[0052]
A digital image signal obtained by encoding the teacher signal corresponding to the second image signal is further decoded to obtain a student signal corresponding to the first image signal.
[0053]
The first frame of the student signal is divided into a plurality of blocks, and each block has a motion vector between the first frame and the second frame. In the first frame of the student signal, a motion vector at a predetermined position corresponding to the target position in the teacher signal is detected.
[0054]
Detection of this motion vector is performed by the above-described motion vector detection device and the like, using not only the motion vector of the block where the predetermined position exists but also the motion vector of a block around this block as a higher-layer motion vector. Detected hierarchically.
[0055]
Based on the first frame of the student signal and the second frame of the teacher signal motion-compensated with the detected motion vector, a plurality of pixels located in the temporal and spatial directions around the target position in the teacher signal The data is selected. Then, coefficient data is obtained using the plurality of pixel data and the pixel data at the position of interest in the teacher signal.
[0056]
As described above, the coefficient data of the estimation formula used when converting the first image signal into the second image signal is generated, but when converting the first image signal into the second image signal, The pixel data of the target position in the second image signal is calculated by the estimation formula using the coefficient data.
[0057]
An image signal processing device according to the present invention converts a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal. A motion vector detecting unit for detecting a motion vector between a plurality of consecutive frames of the first image signal, the motion vector being related to a position of interest in a predetermined frame of the second image signal; Based on a plurality of continuous frames of the first image signal on which motion compensation has been performed based on the plurality of motion vectors detected by the vector detection unit, a spatial direction is determined with respect to a target position in a predetermined frame of the second image signal. Data selecting means for selecting a plurality of first pixel data located in the vicinity in the time direction and a plurality of first pixel data selected by the data selecting means. In which and a pixel data generation means for generating pixel data of the target position in a given frame of the second image signal.
[0058]
Then, a part or all of the frame of the first image signal is divided into a plurality of blocks, each of which has a motion vector with another frame, and the motion vector detection unit outputs the first image signal. Processing for detecting a motion vector between a second frame and a second frame at a predetermined position corresponding to a target position in a predetermined frame of a second image signal existing in a first frame related to a plurality of consecutive frames And a second processing unit for linearly distributing the motion vectors detected by the first processing unit to obtain a motion vector corresponding to a time-direction position of a predetermined frame of the second image signal. .
[0059]
The first processing unit includes a motion vector selection unit that selects motion vectors of a plurality of blocks located around the predetermined position as motion vectors of an upper layer, and a reference block centered on the predetermined position of the first frame. And a position corresponding to the predetermined position of the second frame between the candidate blocks in the plurality of search ranges, each of which is a position moved based on the plurality of motion vectors selected by the motion vector selecting means as a reference position. Correlation detection means for detecting correlation information, and, based on the correlation information corresponding to the plurality of candidate blocks detected by the correlation detection means, the position information of the candidate block having the highest correlation with the reference block, the motion vector of the predetermined position and Motion vector detecting means.
[0060]
Further, the image signal processing method according to the present invention converts the first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal. A first step of detecting a motion vector between a plurality of consecutive frames of the first image signal corresponding to a position of interest in a predetermined frame of the second image signal; Based on a plurality of consecutive frames of the first image signal, the motion of which has been compensated based on the plurality of motion vectors detected in the first step, A second step of selecting a plurality of first pixel data located in the periphery in the spatial direction and the time direction, and using the plurality of first pixel data selected in the second step. , In which a third step of generating pixel data of the target position in a given frame of the second image signal.
[0061]
Then, a part or all of the frame of the first image signal is divided into a plurality of blocks, and each block has a motion vector with respect to another frame. Processing for detecting a motion vector between a second frame at a predetermined position corresponding to a target position in a predetermined frame in a second image signal present in a first frame related to a plurality of consecutive frames And a second processing step of linearly allocating the motion vector detected in the first processing step to obtain a motion vector corresponding to a time-direction position of a predetermined frame of the second image signal. .
[0062]
Then, the first processing step includes a step of selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of an upper layer, and a step of selecting a reference block centered on the predetermined position of the first frame and a second Detecting correlation information between candidate blocks in a plurality of search ranges each having a position corresponding to a predetermined position of the frame based on a plurality of selected motion vectors as a reference position, Setting position information of a candidate block having the highest correlation with the reference block as a motion vector at a predetermined position based on the correlation information corresponding to the plurality of detected candidate blocks.
[0063]
A program according to the present invention is for causing a computer to execute the above-described image signal processing method.
[0064]
According to the present invention, a first image signal composed of a plurality of pixel data is converted into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal. Part or all of the frame of the first image signal is divided into a plurality of blocks, and each block has a motion vector between itself and another frame.
[0065]
A motion vector between a plurality of consecutive frames of the first image signal, which is related to a position of interest in a predetermined frame of the second image signal, is detected.
In this case, first, between the second frame at a predetermined position corresponding to the target position in the predetermined frame of the second image signal existing in the first frame related to a plurality of continuous frames of the first image signal. Are detected. Detection of this motion vector is performed by the above-described motion vector detection device and the like, using not only the motion vector of the block where the predetermined position exists but also the motion vector of a block around this block as a higher-layer motion vector. Detected hierarchically. Then, the detected motion vectors are linearly distributed, and a motion vector corresponding to a time-direction position of a predetermined frame of the second image signal is obtained.
[0066]
Based on a plurality of continuous frames of the first image signal on which motion compensation has been performed based on the detected plurality of motion vectors, a time direction and a spatial direction corresponding to a target position in a predetermined frame of the second image signal A plurality of first pixel data located in the vicinity of are selected. Then, using the plurality of first pixel data, pixel data of a target position in a predetermined frame of the second image signal is generated.
[0067]
For example, the class to which the pixel data of the target position in the second image signal belongs is detected. Then, using the coefficient data generated in accordance with the detected class and the plurality of first pixel data, pixel data of the target position in the predetermined frame of the second image signal is calculated based on the estimation formula. You. As the coefficient data, coefficient data obtained by learning using a student signal corresponding to the first image signal and a teacher signal corresponding to the second image signal is used.
[0068]
Here, based on a plurality of continuous frames of the first image signal motion-compensated by the detected motion vector, a plurality of frames around the time direction and the space direction corresponding to the position of interest in the predetermined frame of the second image signal. A plurality of second pixel data located are selected, and a class to which the pixel data of the target position in the predetermined frame of the second image signal belongs is detected using at least the second pixel data. In this case, the plurality of second pixel data are selected from a plurality of continuous frames of the motion-compensated first image signal, so that the accuracy of class classification can be improved, and thus the second image signal can be improved. Quality can be improved.
[0069]
The detection of the motion vector at the predetermined position is performed hierarchically using not only the motion vector of the block in which the predetermined position exists but also the motion vector of a block around the block as a higher-layer motion vector. In addition, a motion vector between a plurality of continuous frames of the first image signal and a motion vector related to a target position in a predetermined frame of the second image signal is accurately detected.
[0070]
Then, the plurality of first pixel data are selected from a plurality of continuous frames of the first image signal motion-compensated with the detected motion vector. Thereby, a plurality of first pixel data selected from a plurality of continuous frames of the first image signal has a high correlation. Therefore, it is possible to improve the quality of the pixel data of the target position in the predetermined frame of the second image signal generated using the plurality of first pixel data.
[0071]
A coefficient data generating device according to the present invention converts a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal. And generating coefficient data of an estimation formula used when performing the continuation of the student signal corresponding to the first image signal related to the target position in a predetermined frame of the teacher signal corresponding to the second image signal. A motion vector detecting unit that detects a motion vector between a plurality of frames to be processed, and a plurality of continuous frames of a student signal that has been motion-compensated based on the plurality of motion vectors detected by the motion vector detecting unit. Data selection means for selecting a plurality of pixel data located in the spatial and temporal directions around the target position in a predetermined frame of the teacher signal; In which and a calculation means for obtaining the coefficient data by using the pixel data of the target position in the plurality of pixel data and the teacher signal selected by-option means.
[0072]
Then, a part or all of the frame of the student signal is divided into a plurality of blocks, and each block has a motion vector between other frames. A first processing unit for detecting a motion vector between a second frame and a predetermined position corresponding to a target position in a predetermined frame of a teacher signal existing in a related first frame, and the first processing unit And a second processing unit for linearly allocating the motion vector detected in step (a) to obtain a motion vector corresponding to the time-direction position of a predetermined frame of the teacher signal.
[0073]
The first processing unit includes a motion vector selection unit that selects motion vectors of a plurality of blocks located around the predetermined position as motion vectors of a higher hierarchy, and a reference that centers on the predetermined position of the first frame. A block corresponding to a predetermined position of the second frame and a candidate block within a plurality of search ranges each having a position moved based on the plurality of motion vectors selected by the motion vector selection means as a reference position are described. Correlation detection means for detecting correlation information between the plurality of candidate blocks detected by the correlation detection means, and, based on the correlation information corresponding to the plurality of candidate blocks, the position information of the candidate block having the highest correlation with the reference block. And a motion vector detecting means for converting the vector.
[0074]
In the coefficient data generation method according to the present invention, a first image signal including a plurality of pixel data is converted into a second image signal including a plurality of pixel data and having a different number of frames from the first image signal. A method for generating coefficient data of an estimation formula used when performing the method, wherein a sequence of student signals corresponding to the first image signal, related to a target position of a teacher signal corresponding to the second image signal in a predetermined frame. A first step of detecting a motion vector between a plurality of frames to be performed, and a plurality of continuous frames of a student signal which has been motion-compensated based on the plurality of motion vectors detected in the first step. A second step of selecting a plurality of pieces of pixel data located in the periphery in the spatial direction and the temporal direction with respect to the target position in the predetermined frame of the teacher signal, and the second step In which a third step of obtaining the coefficient data by using the pixel data of the target position in the selected plurality of pixel data and the teacher signal.
[0075]
Then, a part or all of the frame of the student signal is divided into a plurality of blocks, and each block has a motion vector with respect to another frame. A first processing step of detecting a motion vector between a second frame and a predetermined position corresponding to a target position in a predetermined frame of a teacher signal existing in a related first frame; and the first processing step And a second processing step of linearly allocating the motion vector detected in step (a) to obtain a motion vector corresponding to the time-direction position of a predetermined frame of the teacher signal.
[0076]
Then, the first processing step includes a step of selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of an upper layer, and a step of selecting a reference block centered on the predetermined position of the first frame and a second Detecting correlation information between candidate blocks in a plurality of search ranges each having a position corresponding to a predetermined position of the frame based on a plurality of selected motion vectors as a reference position, Setting position information of a candidate block having the highest correlation with the reference block as a motion vector at a predetermined position based on the correlation information corresponding to the plurality of detected candidate blocks.
[0077]
Further, a program according to the present invention is for causing a computer to execute the above-described coefficient data generation method.
[0078]
In the present invention, coefficient data of an estimation formula used when converting a first image signal composed of a plurality of pixel data into a second image signal having a different number of frames from the first image signal is calculated. Generate.
[0079]
A motion vector between a plurality of continuous frames of the student signal corresponding to the first image signal, which is related to a target position in a predetermined frame of the teacher signal corresponding to the second image signal, is detected.
[0080]
In this case, some or all frames of the student signal are divided into a plurality of blocks, and each block has a motion vector between itself and another frame. Then, first, a motion vector between the second frame and a predetermined position corresponding to the target position in the predetermined frame of the teacher signal existing in the first frame related to a plurality of continuous frames of the student signal is detected. .
[0081]
Detection of this motion vector is performed by the above-described motion vector detection device and the like, using not only the motion vector of the block where the predetermined position exists but also the motion vector of a block around this block as a higher-layer motion vector. Detected hierarchically. Then, the detected motion vectors are linearly distributed, and a motion vector corresponding to the temporal position of a predetermined frame of the teacher signal is obtained.
[0082]
Based on a plurality of continuous frames of a student signal motion-compensated on the basis of a plurality of detected motion vectors, a plurality of pixel data located in the temporal and spatial directions with respect to a target position in a predetermined frame of the teacher signal. Is selected. Then, coefficient data is obtained using the plurality of pixel data and the pixel data at the position of interest in the teacher signal.
[0083]
As described above, the coefficient data of the estimation formula used when converting the first image signal into the second image signal is generated, but when converting the first image signal into the second image signal, The pixel data of the target position in the predetermined frame of the second image signal is calculated by the estimation formula using the coefficient data.
[0084]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows a configuration of a digital broadcast receiver 100 according to the first embodiment.
[0085]
The digital broadcast receiver 100 includes a microcomputer, and includes a system controller 101 for controlling the operation of the entire system, and a remote control signal receiving circuit 102 for receiving a remote control signal RM. The remote control signal receiving circuit 102 is connected to the system controller 101, receives a remote control signal RM output in response to a user operation from the remote control transmitter 200, and supplies an operation signal corresponding to the signal RM to the system controller 101. It is configured to
[0086]
Further, the digital broadcast receiver 100 is supplied with a receiving antenna 105 and a broadcast signal (RF modulated signal) captured by the receiving antenna 105, performs channel selection processing, demodulation processing, error correction processing, and the like, and executes a predetermined program. And a tuner unit 106 for obtaining an MPEG2 stream as an encoded image signal according to the above.
[0087]
Also, the digital broadcast receiver 100 decodes the MPEG2 stream output from the tuner unit 106 to obtain an image signal Va, and temporarily converts the image signal Va output from the MPEG2 decoder 107 into an image signal Va. And a buffer memory 108 for temporarily storing.
[0088]
In the present embodiment, the picture information PI and the motion compensation vector MI are output from the MPEG2 decoder 107 in addition to the pixel data constituting the image signal Va. The buffer memory 108 also stores information PI and MI in pairs with each pixel data.
[0089]
The picture information PI is information indicating which of the I-picture (Intra-Picture), P-picture (Predictive-Picture), and B-picture (Bidirectionally-predictive-Picture) the output pixel data is. When the output pixel data is related to a P picture or a B picture, the motion compensation vector MI is information indicating how the reference data for decoding the pixel data has been motion compensated. As is well known, the motion compensation vector MI is a motion vector detected by the MPEG2 encoder for each macroblock.
[0090]
FIG. 2 shows the configuration of the MPEG2 decoder 107.
The decoder 107 has an input terminal 71 to which an MPEG2 stream is input, and a stream buffer 72 for temporarily storing the MPEG2 stream input to the input terminal 71.
[0091]
The decoder 107 extracts a DCT (Discrete Cosine Transform) coefficient as a frequency coefficient from the MPEG2 stream stored in the stream buffer 72, and the extraction circuit 73 extracts the DCT coefficient. And a variable-length decoding circuit 74 for performing variable-length decoding on DCT coefficients that have been subjected to variable-length coding, for example, Huffman coding.
[0092]
Further, the decoder 107 includes an extraction circuit 75 for extracting quantization characteristic designation information QI from the MPEG2 stream stored in the stream buffer 72, and a variable length decoding circuit based on the quantization characteristic designation information QI. An inverse quantization circuit 76 that performs inverse quantization on the quantized DCT coefficient output from 74 and an inverse DCT circuit 77 that performs inverse DCT on the DCT coefficient output from the inverse quantization circuit 76 are doing.
[0093]
Also, the decoder 107 stores the pixel data of the I picture and the P picture in a memory (not shown), and uses the pixel data to generate the residual data of the P picture or the B picture from the inverse DCT circuit 77. When output, it has a prediction memory circuit 78 for generating and outputting the corresponding reference data Vref.
[0094]
When the residual data of the P picture or the B picture is output from the inverse DCT circuit 77, the decoder 107 adds the reference data Vref generated by the prediction memory circuit 78 to the residual data. And an output terminal 81 that outputs pixel data of each picture output from the addition circuit 79 as an image signal Va.
[0095]
Note that when the pixel data of the I picture is output from the inverse DCT circuit 77, the reference data Vref is not supplied from the prediction memory circuit 78 to the addition circuit 79. The pixel data of the picture is output as it is.
[0096]
Here, in the encoding of the MPEG system, as is conventionally known, encoding is performed in an order different from the actual order of frames / fields. That is, the image signals of the I picture and the P picture are encoded first, and the image signal of the B picture sandwiched between them is encoded thereafter. The output terminal 81 outputs the image signal Va of each picture in the order of encoding.
[0097]
Further, the decoder 107 includes an extraction circuit 82 that extracts encoding control information, that is, picture information PI and a motion compensation vector MI, from the MPEG2 stream stored in the stream buffer 72, and outputs the picture information PI and the motion compensation vector MI. And an output terminal 83 for outputting the application vector MI.
[0098]
The motion compensation vector MI extracted by the extraction circuit 82 is supplied to the prediction memory circuit 78, and the prediction memory circuit 78 performs motion compensation when generating the reference data Vref using the motion compensation vector MI. The picture information PI extracted by the extraction circuit 82 is also supplied to the prediction memory circuit 78. The prediction memory circuit 78 identifies a picture based on the picture information PI.
[0099]
The operation of the MPEG2 decoder 107 shown in FIG. 2 will be described.
The MPEG2 stream stored in the stream buffer 72 is supplied to an extraction circuit 73, and a DCT coefficient as a frequency coefficient is extracted. This DCT coefficient is subjected to variable length coding, and this DCT coefficient is supplied to a variable length decoding circuit 74 and decoded. Then, the quantized DCT coefficients of each DCT block output from the variable length decoding circuit 74 are supplied to the inverse quantization circuit 76, where the quantization is performed.
[0100]
The DCT coefficient of each DCT block output from the inverse quantization circuit 76 is subjected to inverse DCT by the inverse DCT circuit 77 to obtain data of each picture. The data of each picture is output to the output terminal 81 via the addition circuit 79. In this case, when residual data of a P picture or a B picture is output from the inverse DCT circuit 77, the addition data 79 adds the reference data Vref output from the prediction memory circuit 78.
[0101]
It should be noted that the picture information PI and the motion compensation vector MI are output to the output terminal 83 in pairs with the pixel data constituting the image signal Va output from the output terminal 81.
[0102]
Returning to FIG. 1, the digital broadcast receiver 100 further reduces coding noise (coding distortion) such as block noise (block distortion) and mosquito noise from the image signal Va stored in the buffer memory 108. An image signal processing unit 110 converts the image signal Vb into an image signal Vb, and a display unit 111 that displays an image based on the image signal Vb output from the image signal processing unit 110. The display unit 111 is configured by a display such as a CRT (cathode-ray tube) display or an LCD (liquid crystal display).
[0103]
The operation of the digital broadcast receiver 100 shown in FIG. 1 will be described.
The MPEG2 stream output from the tuner unit 106 is supplied to an MPEG2 decoder 107 and decoded. Then, the image signal Va output from the decoder 107 is supplied to the buffer memory 108 and is temporarily stored. In this case, the decoder 107 also outputs the motion compensation vector MI and the picture information PI in pairs with each pixel data of the image signal Va. The picture information PI and the motion compensation vector MI are also temporarily stored in the buffer memory 108.
[0104]
Thus, the image signal Va stored in the buffer memory 108 is supplied to the image signal processing unit 110, and is converted into an image signal Vb in which coding noise (coding distortion) is reduced. In the image signal processing unit 110, pixel data forming the image signal Vb is obtained from pixel data forming the image signal Va. In the image signal processing unit 110, a conversion process is performed using the picture information PI and the motion compensation vector MI stored in the buffer memory 108.
[0105]
The image signal Vb output from the image signal processing unit 110 is supplied to the display unit 111, and an image based on the image signal Vb is displayed on the screen of the display unit 111.
[0106]
Next, details of the image signal processing unit 110 will be described.
The image signal processing unit 110 includes a first processing unit 121 that performs a conversion process on an I-picture image signal among the image signals Va stored in the buffer memory 108, and an image signal stored in the buffer memory 108. A second processing unit 122 that performs a conversion process on a P picture and a B picture out of Va, and an I picture image signal Vb (I) obtained by the first processing unit 121 and the second processing unit 122 A buffer memory 123 for temporarily storing the obtained picture signal Vb (P) of the P picture. FIG. 3 illustrating the details of the first processing unit 121 and the second processing unit 122 shows the configuration of the first processing unit 121. The first processing unit 121 has a prediction tap selection circuit 131 and a class tap selection circuit 132. The prediction tap selection circuit 131 selectively extracts a plurality of pixel data of a prediction tap used for prediction based on the image signal Va (I) of the I picture stored in the buffer memory 108. The class tap selection circuit 132 selectively extracts a plurality of pixel data of class taps used for class classification based on the image signal Va (I) of the I picture stored in the buffer memory 108.
[0107]
The pixel data of the prediction tap and the class tap are pixel data located in the periphery in the spatial direction (horizontal direction, vertical direction) with respect to the target position in the image signal Vb (I) of the I picture.
[0108]
In addition, the first processing unit 121 includes a class generation circuit 133 that generates a class code CL1 indicating a space class from pixel data of a class tap selectively extracted by the class tap selection circuit 132. The class generation circuit 133 generates a class code CL1 indicating a space class by performing processing such as 1-bit ADRC (Adaptive Dynamic Range Coding) on the pixel data of the class tap.
[0109]
The ADRC determines the maximum value and the minimum value of a plurality of pixel data of a class tap, obtains a dynamic range that is a difference between the maximum value and the minimum value, and requantizes each pixel value according to the dynamic range. . In the case of 1-bit ADRC, the pixel value is converted into 1 bit depending on whether it is larger or smaller than the average value of a plurality of pixel values of the class tap. The ADRC process is a process for making the number of classes representing the level distribution of pixel values relatively small. Therefore, the present invention is not limited to the ADRC, and an encoding method for compressing the number of bits of a pixel value such as VQ (vector quantization) may be used.
[0110]
Further, the first processing unit 121 has a coefficient memory 134. The coefficient memory 134 stores, for each class, coefficient data Wi (i = 1 to n, where n is the number of prediction taps) used in an estimation formula used in an estimation prediction operation circuit 135 described later. .
[0111]
The coefficient data Wi is information for converting the image signal Va (I) into the image signal Vb (I). The coefficient data Wi stored in the coefficient memory 134 is generated in advance by learning between a student signal corresponding to the image signal Va (I) and a teacher signal corresponding to the image signal Vb (I). The class code CL1 generated by the class generation circuit 133 described above is supplied to the coefficient memory 134 as read address information. From the coefficient memory 134, coefficient data Wi of the estimation formula corresponding to the class code CL1 is read and supplied to the estimation prediction operation circuit 135. A method for generating the coefficient data Wi will be described later.
[0112]
Further, the first processing unit 121 obtains the prediction tap pixel data xi selectively extracted by the prediction tap selection circuit 131 and the coefficient data Wi read from the coefficient memory 134 by using the estimation expression of Expression (1). An estimation / prediction calculation circuit 135 is provided for calculating pixel data y at the position of interest in the image signal Vb (I) to be created.
[0113]
(Equation 1)

[0114]
The operation of the first processing unit 121 will be described.
Based on the image signal Va (I) of the I picture stored in the buffer memory 108, the class tap selection circuit 132 uses the class tap selection circuit 132 to set the peripheral position in the spatial direction with respect to the target position in the image signal Vb (I) of the I picture to be created , The pixel data of the class tap located at. The pixel data of the class tap is supplied to the class generation circuit 133. In the class generation circuit 133, a process such as 1-bit ADRC is performed on the pixel data of the class tap, and a class code CL1 indicating a space class is generated.
[0115]
The class code CL1 indicates the detection result of the class to which the pixel data of the target position in the image signal Vb (I) belongs. The class code CL1 is supplied to the coefficient memory 134 as read address information. The coefficient data Wi corresponding to the class code CL1 is read from the coefficient memory 134 and supplied to the estimation / prediction calculation circuit 135.
[0116]
Further, based on the image signal Va (I) of the I picture stored in the buffer memory 108, the prediction tap selection circuit 131 uses the prediction tap selection circuit 131 in the spatial direction with respect to the target position in the image signal Vb (I) of the I picture to be created. , Pixel data of the prediction taps located around is extracted selectively.
[0117]
The estimation prediction operation circuit 135 uses the pixel data xi of the prediction tap and the coefficient data Wi read from the coefficient memory 134 to generate an I-picture image to be created based on the above-described estimation expression (1). The pixel data y at the target position in the signal Vb (I) is obtained.
[0118]
As described above, the first processing unit 121 obtains the I-picture image signal Vb (I) from the I-picture image signal Va (I) using the coefficient data Wi. In this case, a plurality of pixel data (pixel data of the prediction tap) located around the target position in the image signal Vb (I) selected based on the image signal Va (I), and the image signal Vb (I) Is used to generate the pixel data y of the target position in the image signal Vb (I) based on the estimation formula using the coefficient data Wi corresponding to the class CL1 to which the pixel data of the target position belongs.
[0119]
Therefore, as the coefficient data Wi, a student signal corresponding to the image signal Va (I) and including the same coding noise as the image signal Va (I) and a teacher not including the coding noise corresponding to the image signal Vb (I). By using the coefficient data Wi obtained by learning using the signal, it is possible to obtain an image signal Vb (I) in which coding noise is reduced as compared with the image signal Va (I).
[0120]
FIG. 4 shows a configuration of the second processing unit 122. The second processing unit 122 includes a prediction tap selection circuit 141 for selectively extracting a plurality of pixel data of prediction taps used for prediction, and a class for selectively extracting a plurality of pixel data of class taps used for class classification. And a tap selection circuit 142.
[0121]
The tap selection circuits 141 and 142 receive, from the system controller 101, the pixels of the image signal Va (P) / Va (B) corresponding to the target position in the image signal Vb (P) / Vb (B) of the P picture or B picture. The picture information PI paired with the data and the motion vector MV at a predetermined position where the pixel data of the image signal Va (P) / Va (B) detected by the motion vector detection unit 143 described later are located. It is supplied as operation control information. The pixel data of the prediction tap and the class tap are located around the target position in the image signal Vb (P) / Vb (B) in the spatial direction (horizontal direction, vertical direction) and the time direction (frame direction), respectively. This is pixel data.
[0122]
The tap selection circuits 141 and 142 respectively provide a picture signal Va (P) / Va (B) of a P picture or a B picture stored in the buffer memory 108 and an image of the picture stored in the buffer memory 123. I picture used for obtaining signal Va (P) / Va (B), I picture corresponding to image signal Va (I), Va (P) of P picture, already reduced in coding noise, P picture A plurality of pieces of pixel data are selectively extracted based on picture image signals Vb (I) and Vb (P).
[0123]
Here, the tap selection circuits 141 and 142 perform the motion based on the image signal Va (P) / Va (B) with respect to the image signals Vb (I) and Vb (P) based on the motion compensation vector MV. Make compensation.
[0124]
FIG. 5 shows an example of motion compensation. This example is a case where pixel data located around the position of interest in the image signal Vb (P) of the P picture in the spatial and temporal directions is extracted, and the image signal Va (P) of the P picture and the I picture Is a case where the image signal Vb (I) is a pixel data selection target.
[0125]
In this case, motion compensation is performed on the image signal Vb (I) based on the image signal Va (P). Here, as shown in FIG. 5, when the motion vector MV is (−2, +1), the tap selection circuit converts the image signal Va (P) into x ₁ ~ X _Thirteen Is selectively extracted, and x at a position shifted from the image signal Vb (I) by (−2, +1). ₁ '~ X _Thirteen 'Is selectively extracted.
[0126]
As described above, in the prediction tap selection circuit 141, in addition to the image signals Va (P) / Va (B), the image signals Vb (I) and Vb (P) after the motion noise has been reduced and the coding noise has been reduced. Also selectively extracts the pixel data of the prediction tap. Therefore, it is possible to improve the quality of the pixel data of the target position in the image signal Vb (P) / Vb (B) generated as described later using the pixel data of the prediction tap.
[0127]
In addition, the class tap selection circuit 142 uses the class taps not only from the image signals Va (P) / Va (B) but also from the image signals Vb (I) and Vb (P) after the motion compensation has been performed and the coding noise has been reduced. Is selectively extracted. Therefore, as described above, the temporal and spatial classes corresponding to the pixel data of the prediction tap selected by the motion compensation performed by the prediction tap selection circuit 141 can be accurately detected.
[0128]
The second processing unit 122 has a motion vector detection unit 143. The motion vector detection unit 143 uses the motion compensation vector MI supplied from the buffer memory 108 via the system controller 101 to determine the position of interest in the image signal Vb (P) / Vb (B) of the P picture or the B picture. , A motion vector MV at a predetermined position (hereinafter, referred to as “target pixel data position”) where the pixel data of the image signal Va (P) / Va (B) corresponding to.
[0129]
The motion vector detection unit 143 detects a motion vector by a block matching method. In this method, as shown in FIG. 6, a motion vector is obtained by moving a candidate block of a search frame within a predetermined search range and detecting a candidate block that best matches the reference block of the reference frame. is there.
[0130]
In the block matching method, as shown in FIG. 7A, one image, for example, an image of one frame of horizontal H pixels and vertical V lines is subdivided into blocks of P pixels × Q lines as shown in FIG. 7B. . In the example of FIG. 7B, P = 5 and Q = 5. c is the center pixel position of the block.
[0131]
8A to 8C show the positional relationship between a reference block whose center pixel is c and a candidate block whose center is c '. A reference block whose center pixel is c is a certain reference block of interest in the reference frame, and a candidate frame for a search frame that matches the reference block is located at a position of a block centered on c ′ in the search frame. In the block matching method, a motion vector is detected by finding a candidate block that best matches a reference block within a search range.
[0132]
In the case of FIG. 8A, a motion vector of +1 pixel in the horizontal direction and +1 line in the vertical direction, that is, (+1, +1) is detected. In FIG. 8B, a (+3, +3) motion vector MV is detected, and in FIG. 8C, a (+2, -1) motion vector is detected. The motion vector is obtained for each reference block of the reference frame.
[0133]
Assuming that the range for searching for a motion vector is ± S pixels in the horizontal direction and ± T lines in the vertical direction, the reference block is shifted from the center c by ± S horizontally and ± T vertically to the center c ′. Needs to be compared with the candidate block with
[0134]
FIG. 9 shows that when the position of the center c of a reference block having a reference frame is R, comparison with (2S + 1) × (2T + 1) candidate blocks of a search frame to be compared is necessary. That is, all candidate blocks in which c 'exists at the square in FIG. 9 are comparison targets. FIG. 9 is an example in which S = 4 and T = 3.
[0135]
Detecting a minimum value among evaluation values obtained by comparison within the search range (that is, the sum of absolute values of frame differences, the sum of squares of the frame differences, or the sum of n squares of the absolute values of frame differences). Is used to detect a motion vector. The search range in FIG. 9 is an area where the center of the candidate block is located, and the size of the search range including the entire candidate block is (2S + P) × (2T + Q).
[0136]
Further, the motion vector detection unit 143 detects the motion vector MV in two layers, an upper layer and a lower layer. In this case, the motion vector detection unit 143 uses the motion compensation vector MI of a plurality of macroblocks located around the target pixel data position as the motion vector of the upper hierarchy. Here, the plurality of macroblocks located around the target pixel data position include a macroblock including the target pixel data position, and the periphery includes a spatial direction (horizontal and vertical directions) and a temporal direction (frame direction). Either or both.
[0137]
In the present embodiment, for example, the motion vectors MI of four macroblocks located in the periphery in the spatial direction are used as the motion vectors of the upper layer. In FIG. 10, a macroblock including the target pixel data position is defined as MB0, and eight macroblocks adjacent to the macroblock MB0 are defined as MB1 to MB8. As is well known, the size of a macroblock is 16 × 16 pixels, and this macroblock is composed of four 4 × 4 pixel blocks.
[0138]
When the target pixel data position is included in the block a, the motion compensation vector MI of the macroblocks MB0, MB1, MB2, and MB8 is selected as a higher-layer motion vector. Similarly, when the target pixel data position is included in the blocks b, c, and d, [MB0, MB2, MB3, MB4], [MB0, MB4, MB5, MB6], [MB0, MB6, MB7, MB8] Is selected as the motion vector of the upper layer.
[0139]
As described above, the motion compensation vector MI of the three macro blocks adjacent to the macro block MB0 in addition to the macro block MB0 including the target pixel data position is selected as the motion vector of the upper layer by the macro block MB0. In practice, when there are a plurality of motion vectors, depending on the position of the target pixel data, the motion compensation vector of a macroblock adjacent to the macroblock MB0 is more effective than the motion compensation vector of the macroblock MB0. This is because it may be suitable for the data position.
[0140]
For example, as shown in FIG. 11, when the motion areas M1 and M2 are present (the arrows indicate the motion direction), when the target pixel data position in the macroblock MB0 is Q, the motion compensation of the macroblock MB0 is performed. The motion compensation vector MI of the macro block MB1 is more suitable for the target pixel data position than the use vector MI.
[0141]
The motion vector detection unit 143 performs the motion vector detection by the block matching method in the lower hierarchy using the plurality of motion vectors in the upper hierarchy selected as described above. When the reference frame is represented by T and the search frame is represented by T-1, the evaluation function of the block matching is obtained by Expression (2). However, (u _n , V _n ) Is the motion vector in the lower hierarchy n, (u) _{n + 1} , V _{n + 1} ) Indicates a motion vector in an upper layer.
[0142]
(Equation 2)

[0143]
This evaluation function E (Y) _n Gives the minimum of the evaluation value as the correlation information obtained by (u _n , V _n ) Is the lower-layer motion vector V0. The final motion vector MV is obtained by equation (3). Here, V1 is a higher-layer motion vector used when the minimum evaluation value is obtained.
MV = V0 + V1 (3)
[0144]
The flowchart of FIG. 12 shows the procedure of the motion vector MV detection process.
First, in step ST1, the process is started, and in step ST2, N = 1 is set. Then, in step ST3, among the plurality of motion vectors MI selected as the upper-layer motion vectors as described above, the search of the lower layer is started using the N-th motion vector MI as the upper-layer motion vector.
[0145]
In this case, the frame in which the pixel data position of interest of the image signal Va (P) / Va (B) of the P picture or the B picture exists is set as a reference frame, and the image signal Va (P) / Va (B) of the picture is obtained. The frames of the image signals Va (I) and Va (P) of the I picture and P picture used for obtaining are set as search frames.
[0146]
Then, a reference block centering on the pixel data corresponding to the target pixel data position is set in the reference frame (T frame), and the search within the search range of the search frame (T-1 frame) is started. Here, the search frame is motion-compensated by a higher-layer motion vector. That is, the search range of the search frame is a range in which the position corresponding to the target position (target pixel data position) has been moved based on the higher-layer motion vector as the reference position.
[0147]
Next, in step ST4, it is determined whether all searches within the search range have been completed. If the search has not been completed, the process proceeds to step ST5, where the evaluation function E (Y) is calculated using the pixel data based on the equation (2). _n An evaluation value as correlation information is obtained, and the process returns to step ST4. On the other hand, when the search ends, the process proceeds to step ST6.
[0148]
In step ST6, it is determined whether or not N = Nmax. Nmax indicates the number of a plurality of motion vectors selected as the motion vector of the upper layer. In the present embodiment, Nmax = 4. If N is not equal to Nmax, N is incremented in step ST7, and thereafter, the process returns to step ST3, and a search for a lower layer is performed using a motion vector of the next higher layer. On the other hand, when N = Nmax, the process proceeds to step ST8.
[0149]
In step ST8, the motion vector V0 is determined from the evaluation value as the correlation information. That is, the smallest evaluation value is given (u _n , V _n ) Is the motion vector V0. In step ST9, the final motion vector MV is calculated by adding the corresponding upper-layer motion vector V1 to the lower-layer motion vector V0 based on the equation (3). In this case, the position information of the candidate block having the highest correlation with the reference block is determined as the motion vector MV of the target pixel data position. After the process in step ST9, the process ends in step ST10.
[0150]
FIG. 13 shows a configuration of the motion vector detection unit 143.
The motion vector detecting unit 143 selects the motion compensation vector MI of the four macroblocks located around the pixel data position of interest as the motion vector of the upper layer, and selects the motion vector MI by the selection circuit 21. Four motion vectors MI ₁ ~ MI ₄ And a motion vector accumulation unit 22 for accumulating the motion vector. The position information of the target pixel data position is supplied to the motion vector selection circuit 21 as operation control information.
[0151]
In addition, the motion vector detection unit 143 includes a reference block circuit 23 that selectively reads pixel data of a reference block centered on a target pixel data position from a reference frame stored in the buffer memory 108 (see FIG. 8). A search block circuit 24 sequentially reads out image data corresponding to each candidate block within the search range from the search frame stored in the buffer memory 108.
[0152]
Here, the reference frame is a frame in which the target pixel data position of the image signal Va (P) / Va (B) of the P picture or the B picture exists. Further, the search frame is a frame of the image signals Va (I) and Va (P) of the I picture and the P picture used for obtaining the image signal Va (P) / Va (B) of the picture.
[0153]
Here, the search range of the search frame is a range in which the position corresponding to the target pixel data position has been moved based on the higher-layer motion vector as a reference position. The search block circuit 24 outputs the four motion vectors MI stored in the motion vector storage unit 22 as the motion vector of the upper hierarchy. ₁ ~ MI ₄ Will be used sequentially. Then, the search block circuit 24 sequentially reads the pixel data of the plurality of candidate blocks in the search range determined by the motion vector of each upper layer.
[0154]
Further, the motion vector detection unit 143 uses the pixel data of the reference block from the reference block circuit 23 and the image data of each candidate block from the search block circuit 24 to use the evaluation function E ( Y) _n And an evaluation value calculation circuit 25 for obtaining an evaluation value of each candidate block based on the Here, the evaluation value is correlation information indicating the degree of correlation between the reference block and the candidate block, and the evaluation value calculation circuit 25 constitutes a correlation detecting unit.
[0155]
The search block circuit 24 supplies the evaluation value calculation circuit 25 with the pixel data of the candidate block, as well as the motion vector of the upper hierarchy that has determined the search range in which the candidate block exists. The evaluation value calculation circuit 25 stores the calculated evaluation value of each candidate block in the evaluation value memory for each upper-layer motion vector.
[0156]
The motion vector detection section 143 has a motion vector detection circuit 26. The motion vector detection circuit 26 sequentially evaluates the evaluation value of each candidate block calculated by the evaluation value calculation circuit 25, and calculates a motion vector (u) corresponding to the candidate block giving the minimum evaluation value. _n , V _n ) Is the lower-layer motion vector V0. Then, the final motion vector MV is obtained by the above equation (3).
[0157]
That is, the motion vector V1 of the upper layer is added to the motion vector V0 of the lower layer to obtain a motion vector MV. The motion vector V1 is a higher-level motion vector that has determined the search range in which the candidate block having the smallest evaluation value exists. As described above, since the evaluation value of each candidate block is stored in the evaluation value memory for each motion vector of the upper hierarchy, the information of the motion vector V1 can be easily obtained from the storage position information of the minimum evaluation value in the evaluation value memory. Can be.
[0158]
The motion vector detection circuit 26 outputs a lower-layer motion vector V0 and a minimum evaluation value EYmin in addition to the motion vector MV. As described above, the motion vector MV is used for motion compensation when selecting pixel data of a prediction tap and a class tap. The motion vector VO and the evaluation value EYmin are used for class classification as described later. You.
[0159]
FIG. 14 shows the configuration of the evaluation value calculation circuit 25.
The reference block memory 41 stores the pixel data of the reference block selectively read by the reference block circuit 23 (see FIG. 13). The candidate block memory 42 stores the pixel data of the candidate block selectively read out by the search block circuit 24 (see FIG. 13).
[0160]
The contents of the reference block memory 41 and the candidate block memory 42 are read out in the order of the addresses specified by the memory controller 43, and are subtracted by the subtraction circuit 46 through the registers 44 and 45, respectively. The difference data obtained as a result is converted into an absolute value by an absolute value converting circuit 47, and is cumulatively added by an adding circuit 48 and a register 49. The result of the cumulative addition becomes the evaluation value of the candidate block.
[0161]
The evaluation value memory controller 51 also includes a motion vector MI of the upper layer that has determined the search range in which the candidate block exists. ₁ ~ MI ₄ Is also supplied. The evaluation value memory 50 stores the motion vector MI of the upper hierarchy. ₁ ~ MI ₄ The evaluation value of each candidate block within the search range respectively determined by the motion vector (u _n , V _n ) (See equation (2)).
[0162]
In this case, the evaluation value of a certain candidate block is the motion vector (u _n , V _n ) Is the lower address, and the motion vector (u _{n + 1} , V _{n + 1} ) Is stored at the address of the evaluation value memory 50 having the upper address corresponding to the above.
[0163]
FIG. 15 shows the configuration of the motion vector detection circuit 26.
Under the control of the evaluation value memory controller 51, the evaluation values are read from the evaluation value memory 50 in the order of the above-described addresses (consisting of an upper address and a lower address). In this case, the evaluation value memory controller 51 sends a motion vector (u _n , V _n ) And an upper-layer motion vector (u _{n + 1} , V _{n + 1} ) Is output as a motion vector (u, v).
[0164]
The evaluation value read from the evaluation value memory 50 is supplied to the comparator 61 and the register 62. The motion vector (u, v) output from the evaluation value memory controller 51 and the motion vector (u _n , V _n ) Are supplied to the register 63 and the register 64, respectively.
[0165]
The comparator 61 sequentially compares the other input with the evaluation value read from the evaluation value memory 50, and when the evaluation value read from the evaluation value memory 50 is small, updates the contents of the registers 62 to 64. Output a signal. The register 62 finally holds the minimum evaluation value EYmin. Further, the register 63 holds a motion vector (u, v) corresponding to the candidate block giving the minimum evaluation value EYmin as the motion vector MV. Further, the register 64 stores a motion vector (u) of a lower layer corresponding to the candidate block giving the minimum evaluation value EYmin. _n , V _n ) Is held as the motion vector V0.
[0166]
The operation of the motion vector detection unit 143 shown in FIG. 13 will be described.
The motion vector selection circuit 21 selects a motion compensation vector MI of four macroblocks located around the target pixel data position as a higher-layer motion vector based on the position information of the target pixel data position. The four motion vectors MI thus selected ₁ ~ MI ₄ Are supplied to the motion vector storage unit 22 and stored.
[0167]
The reference block circuit 23 selectively reads out the pixel data of the reference block centering on the target pixel data position from the reference frame stored in the buffer memory 108 (see FIG. 8).
[0168]
The search block circuit 24 sequentially reads out image data corresponding to each candidate block within the search range from the search frame stored in the buffer memory 108. In this case, the search range of the search frame is a range in which the position corresponding to the target pixel data position is moved based on the upper-layer motion vector as a reference position. The search block circuit 24 calculates the four motion vectors MI stored in the motion vector storage unit 22 as the motion vector of the upper hierarchy. ₁ ~ MI ₄ Are sequentially used, and pixel data of a plurality of candidate blocks are sequentially read out in the search range determined by the motion vector of each upper layer.
[0169]
The pixel data of the reference block read by the reference block circuit 23 and the image data of each candidate block read by the search block circuit 24 are supplied to the evaluation value calculation circuit 25. The evaluation value calculation circuit 25 calculates the evaluation function E (Y) of the above equation (2). _n , An evaluation value of each candidate block is obtained. The calculated evaluation value of each candidate block is stored in the evaluation value memory 50 (see FIG. 14) for each motion vector of the upper layer.
[0170]
The motion vector detection circuit 26 sequentially evaluates the evaluation value of each candidate block calculated by the evaluation value calculation circuit 25, and selects a candidate block giving the minimum evaluation value, that is, a motion corresponding to the candidate block having the highest correlation with the reference block. Vector (u _n , V _n ) Is the lower-layer motion vector V0. Then, a motion vector (u, v) obtained by adding the motion vector V1 of the upper layer of the candidate block to the motion vector V0 of the lower layer is output as the final motion vector MV. The motion vector detection circuit 26 outputs the motion vector V0 and the evaluation value EYmin in addition to outputting the motion vector MV.
[0171]
As described above, the motion vector detection unit 143 hierarchically detects the motion vector MV at the pixel data position of interest by using the motion compensation vector MI provided for each macroblock as a higher-layer motion vector. The motion vector MV can be quickly detected. In addition, the motion vector detection unit 143 uses not only the motion compensation vector MI of the macroblock in which the pixel data position of interest exists but also the motion compensation vector MI of the macroblock around the macroblock as a higher-layer motion vector. That is, the motion vector MV at the target pixel data position can be accurately detected.
[0172]
Note that the motion vector detection unit 143 shown in FIG. 13 acquires both the reference frame and the search frame from the image signal Va stored in the buffer memory 108, but acquires the reference frame from the image signal Va. Alternatively, a search frame may be obtained from the image signal Vb stored in the buffer memory 123.
[0173]
Further, the motion vector detecting unit 143 shown in FIG. 13 selects a motion compensation vector MI of a macroblock located around a target pixel data position as a higher-layer motion vector. May be generated, and a predetermined number of motion vectors having higher frequencies may be selected as higher-layer motion vectors.
[0174]
FIG. 16 shows a main part of the motion vector detection unit 143 in that case. In this case, the motion vector detection unit 143 generates a frequency distribution of the motion compensation vector MI of each macroblock of the frame including the target pixel data position, and a frequency distribution generating circuit 27 based on the frequency distribution. A motion vector selection circuit 28 for selecting a predetermined number of motion vectors from the top as the motion vector of the upper layer. Then, the predetermined number of motion vectors MI selected by the selection circuit 28 _a ~ MI _n Are supplied to and stored in the motion vector storage unit 22 and are selected by the motion vector selection circuit 21. ₁ ~ MI ₄ Used as well.
[0175]
It is also conceivable that a motion vector suitable for the target pixel data position does not exist in the motion compensation vectors MI of a plurality of macroblocks located around the target pixel data position. In this case, as described above, there may be a case where a motion vector that matches the target pixel data position is included in the motion vectors selected based on the frequency distribution. Accordingly, by adding the motion compensation vector MI selected based on the frequency distribution as described above to the upper-layer motion vector, the detection accuracy of the motion vector MV at the target pixel data position can be further improved.
[0176]
Referring back to FIG. 4, the second processing unit 122 has class generation circuits 144 and 145. The class generation circuit 144 generates a class code CLa indicating a spatio-temporal class from the pixel data of the class tap selectively extracted by the class tap selection circuit 142. The class generation circuit 144 performs processing such as 1-bit ADRC on the pixel data of the class tap, similar to the class generation circuit 133 of the first processing unit 121 (see FIG. 3) described above, so that the space-time class Is generated.
[0177]
The class generation circuit 145 determines the motion vector V0 and the evaluation value EYmin associated with the motion vector MV at the pixel data position of interest, which are output from the motion vector detection unit 143, using threshold values, and generates a class code CLb indicating the motion-related class. Generate.
[0178]
Further, the second processing unit 122 has a class integration circuit 146 that integrates the class codes CLa and CLb generated by the class generation circuits 144 and 145 into one class code CL2. The class code CL2 indicates the class to which the pixel data of the target position in the image signal Vb (P) / Vb (B) of the P picture or the B picture belongs.
[0179]
The second processing unit 122 has a coefficient memory 147. The coefficient memory 147 stores, for each class, coefficient data Wi (i = 1 to n, where n is the number of prediction taps) used in an estimation formula used in an estimation prediction operation circuit 148 described later. .
[0180]
The coefficient memory 147 stores coefficient data Wi for a P picture for converting a picture signal Va (P) of a P picture into a picture signal Vb (P) of a P picture and a picture signal Va (B) of a B picture. B picture coefficient data Wi for conversion into a B picture image signal Vb (B) are stored at separate addresses. The picture information PI is supplied from the system controller 101 to the coefficient memory 147 as read control information.
[0181]
The coefficient data Wi stored in the coefficient memory 147 is between a student signal corresponding to the image signal Va (P) / Va (B) and a teacher signal corresponding to the image signal Vb (P) / Vb (B) in advance. Generated by learning. The class code CL2 obtained by the above-described class integration circuit 146 is supplied to the coefficient memory 147 as read address information. From the coefficient memory 147, coefficient data Wi of the estimation formula corresponding to the class code CL2 is read and supplied to the estimation prediction operation circuit 148. A method for generating the coefficient data Wi will be described later.
[0182]
Further, the second processing unit 122 obtains the above-described estimation expression (1) from the pixel data xi of the prediction tap selectively extracted by the prediction tap selection circuit 141 and the coefficient data Wi read from the coefficient memory 147. Accordingly, an estimation prediction operation circuit 148 for calculating pixel data y at a target position in an image signal Vb (P) / Vb (B) to be created is provided.
[0183]
The operation of the second processing unit 122 will be described.
When obtaining an image signal Va (P) / Va (B) of a P picture or a B picture stored in the buffer memory 108 and the image signal Va (P) / Va (B) stored in the buffer memory 123 Class tap selection circuit based on the I-picture and P-picture image signals Vb (I) and Vb (P) corresponding to the I-picture and P-picture image signals Va (I) and Va (P) used for At 142, the pixel data of the class taps located in the spatial and temporal directions around the target position in the image signal Vb (P) / Vb (B) of the P picture or B picture to be created is selectively extracted. .
[0184]
In this case, based on the motion vector MV at the target pixel data position detected by the motion vector detection unit 143, the image signal Va (P) / Va (B) is converted with respect to the image signals Vb (I) and Vb (P). The motion compensation based on the reference is performed. Note that the target pixel data position is, as described above, a pixel of the image signal Va (P) / Va (B) corresponding to the target position in the image signal Vb (P) / Vb (B) of the P picture or B picture. It refers to a predetermined position where data is located.
[0185]
The pixel data of the class tap extracted by the class tap selection circuit 142 is supplied to the class generation circuit 144. The class generation circuit 144 performs a process such as 1-bit ADRC on the pixel data of the class tap, and generates a class code CLa indicating a space-time class.
[0186]
The motion vector V0 and the evaluation value EYmin related to the motion vector MV output from the motion vector detection unit 143 are supplied to the class generation circuit 145. The class generation circuit 145 determines a threshold value for each of the motion vector V0 and the evaluation value EYmin, and generates a class code CLb indicating a motion-related class.
[0187]
The class codes CLa and CLb generated by the class generation circuits 144 and 145 are integrated by the class integration circuit 146, and the pixel data at the target position in the image signal Vb (P) / Vb (B) of the P picture or the B picture is obtained. One class code CL2 indicating the class to which the user belongs is obtained. This class code CL2 is supplied to the coefficient memory 147 as read address information.
[0188]
The coefficient data Wi corresponding to the class code CL2 is read from the coefficient memory 147 and supplied to the estimation / prediction calculation circuit 148. In this case, when the image signal Vb (P) of the P picture is created based on the picture information PI, the coefficient data Wi for the P picture is read, and the image signal Vb (B) of the B picture is created. , The coefficient data Wi for the B picture is read.
[0189]
Further, the image signal Va (P) / Va (B) of the P picture or the B picture stored in the buffer memory 108 and the image signal Va (P) / Va (B) stored in the buffer memory 123 are used. The prediction tap is performed based on the I-picture and P-picture image signals Vb (I) and Vb (P) corresponding to the I-picture and P-picture image signals Va (I) and Va (P) used at the time of acquisition. The selection circuit 141 selectively selects pixel data of prediction taps located in the spatial and temporal directions around the target position in the image signal Vb (P) / Vb (B) of the P picture or B picture to be created. Taken out. In this case, based on the motion vector MV at the target pixel data position detected by the motion vector detection unit 143, the image signal Va (P) / Va (B) is converted with respect to the image signals Vb (I) and Vb (P). The motion compensation based on the reference is performed.
[0190]
The estimation prediction operation circuit 145 uses the pixel data xi of the prediction tap and the coefficient data Wi read from the coefficient memory 147 to generate a P picture or a B picture to be created based on the above estimation equation (1). The pixel data y at the target position in the picture image signal Vb (P) / Vb (I) is obtained.
[0191]
As described above, the second processing unit 122 uses the coefficient data Wi from the image signal Va (P) / Va (B) of the P picture or B picture to use the image signal Vb (P) / P Vb (B) is obtained. In this case, the image signal Va (P) / Va (B) and Vb (I) are located around the target position in the image signal Vb (P) / Vb (B) selected based on Vb (P). Using a plurality of pixel data (pixel data of the prediction tap) and coefficient data Wi corresponding to the class CL2 to which the pixel data of the target position in the image signal Vb (P) / Vb (B) belongs, based on the estimation formula. This is to generate pixel data y at the target position in the image signal Vb (P) / Vb (B).
[0192]
Therefore, as the coefficient data Wi, the student signal and the image signal Vb (P) corresponding to the image signal Va (P) / Va (B) and including the same coding noise as the image signal Va (P) / Va (B). By using coefficient data Wi obtained by learning using a teacher signal that does not include coding noise corresponding to / Vb (B), an image signal Va (P) is obtained as an image signal Vb (P) / Vb (B). ) / Va (B) can be obtained with reduced coding noise.
[0193]
The pixel data of the prediction tap is not only selected from the image signal Va (P) / Va (B), but is also motion-compensated by the motion vector MV of the target pixel data position detected by the motion vector detection unit 143 with high accuracy. In addition, it is also selected from the image signals Vb (I) and Vb (P) after the coding noise has been reduced, and the quality of the image signals Vb (P) / Vb (B) can be improved. The included coding noise can be favorably reduced.
[0194]
The pixel data of the class tap is not only selected from the image signal Va (P) / Va (B), but also is motion-compensated by the motion vector MV at the target pixel data position detected by the motion vector detection unit 143 with high accuracy. The class code CLa generated by the class generation circuit 144 is also selected from the image signals Vb (I) and Vb (P) after the coding noise has been reduced. The spatio-temporal class corresponding to the pixel data of the prediction tap selected by performing is accurately indicated. Therefore, the accuracy of the classification can be improved, and the quality of the image signal Vb (P) / Vb (B) can be improved.
[0195]
Also, a class code CLb indicating a motion-related class is generated using the lower-layer motion vector V0 related to the motion vector MV and the minimum evaluation value EYmin output from the motion vector detecting unit 143. Then, as the class code CL2 indicating the class to which the pixel data of the target position in the image signal Vb (P) / Vb (B) of the P picture or the B picture belongs, a combination of the class code CLb is used. As described above, since the information related to the motion vector MV for motion-compensating the image signals Vb (I) and Vb (P) after coding noise reduction is used as the class classification information, it is possible to perform accurate class classification. As a result, the quality of the image signal Vb (P) / Vb (B) can be improved.
[0196]
Returning to FIG. 1, the image signal processing unit 110 further includes an image signal Vb (I) of an I picture output from the first processing unit 121 and a P picture and a B picture output from the second processing unit 122. The image signal Vb (P) and Vb (B) are rearranged in the order of actual frames / fields and output as an image signal Vb. The reordering unit 124 performs a reordering process using the buffer memory 125.
[0197]
The operation of the image signal processing unit 110 will be described.
The image signal Va (I) of the I picture among the image signals Va stored in the buffer memory 108 is supplied to the first processing unit 121. The first processing unit 121 generates an I-picture image signal Vb (I) with reduced coding noise from the I-picture image signal Va (I) using the coefficient data Wi.
[0198]
In this case, a plurality of pixel data (pixel data of the prediction tap) located around the target position in the image signal Vb (I) selected based on the image signal Va (I), and the image signal Vb (I) The pixel data y of the target position in the image signal Vb (I) is generated based on the estimation formula using the coefficient data Wi corresponding to the class CL1 to which the pixel data of the target position belongs.
[0199]
The image signal Vb (I) of the I picture generated by the first processing unit 121 is supplied to the rearrangement unit 124 and is also supplied to the buffer memory 123 and temporarily stored. The image signal Vb (I) of the I picture stored in the buffer memory 123 as described above is generated by the second processing unit 122 into the image signal Vb (P) of the P picture and the image signal Vb (B) of the B picture. Used when
[0200]
Further, the image signal Va (P) / Va (I) of the P picture or the B picture among the image signals Va stored in the buffer memory 108 is supplied to the second processing unit 122. The second processing unit 122 converts the image signal Va (P) / Va (B) and the I-picture and P-picture image signals Vb (I) and Vb (P) stored in the buffer memory 123 into coefficient data. Using Wi, an image signal Vb (P) / Vb (B) of a P picture or a B picture with reduced coding noise is generated.
[0201]
In this case, the image signal Va (P) / Va (B) and Vb (I) are located around the target position in the image signal Vb (P) / Vb (B) selected based on Vb (P). Using a plurality of pixel data (pixel data of the prediction tap) and coefficient data Wi corresponding to the class CL2 to which the pixel data of the target position in the image signal Vb (P) / Vb (B) belongs, based on the estimation formula. The pixel data y at the target position in the image signal Vb (P) / Vb (B) is generated.
[0202]
The P-picture and B-picture image signals Vb (P) and Vb (B) generated by the second processing unit 122 are supplied to the rearrangement unit 124. Further, the image signal Vb (P) of the P picture generated by the second processing unit 122 is supplied to the buffer memory 123 and temporarily stored. The image signal Vb (P) of the P picture stored in the buffer memory 123 in this manner is used when the second processing unit 122 generates the image signal Vb (B) of the B picture.
[0203]
The reordering unit 124 outputs the I-picture image signal Vb (I) output from the first processing unit 121 and the P-picture and B-picture image signals Vb (P), Vb ( B) are rearranged from the order of encoding to the order of actual frames / fields, and are sequentially output as image signals Vb.
[0204]
As described above, the image signal processing unit 110 obtains the image signal Vb from the image signal Va using the coefficient data Wi. Therefore, the coefficient data Wi is obtained by learning using a student signal corresponding to the image signal Va and including the same coding noise as the image signal Va and a teacher signal corresponding to the image signal Vb and including no coding noise. By using the obtained coefficient data Wi, it is possible to obtain an image signal Vb in which coding noise is reduced as compared with the image signal Va.
[0205]
When obtaining the image signals Vb (P) and Vb (B) of the P picture and the B picture, the second processing unit 122 converts the pixel data of the prediction tap from the image signal Va (P) / Va (B). In addition to the selection, the selection is also made from the image signals Vb (I) and Vb (P) in which the coding noise has been reduced. Therefore, it is possible to improve the quality of the picture signals Vb (P) and Vb (B) of the P picture and the B picture, and it is possible to reduce the coding noise contained therein.
[0206]
In addition, the second processing unit 122 performs motion compensation on the image signals Vb (I) and Vb (P) based on the image signal Va (P) / Va (B) to convert the pixel data of the prediction tap into pixel data. It is selectively taken out. Therefore, the pixel data of the prediction tap for generating the pixel data of the target position in the image signal Vb (P) / Vb (B) has a high correlation, and the quality of the image signal Vb (P) / Vb (B) is reduced. Can be enhanced.
[0207]
Next, a method of generating the coefficient data Wi stored in the coefficient memory 134 of the first processing unit 121 (see FIG. 3) and the coefficient memory 147 of the second processing unit 122 (see FIG. 4) will be described. The coefficient data Wi is generated in advance by learning.
[0208]
First, this learning method will be described. In the above equation (1), before learning, the coefficient data W ₁ , W ₂ , ‥‥, W _n Is an undetermined coefficient. The learning is performed on a plurality of signal data for each class. When the number of learning data is m, the following equation (4) is set according to the equation (1). n indicates the number of prediction taps.
y _k = W ₁ × x _k1 + W ₂ × x _k2 + ‥‥ + W _n × x _kn ... (4)
(K = 1, 2, ‥‥, m)
[0209]
If m> n, the coefficient data W ₁ , W ₂ , ‥‥, W _n Is not uniquely determined, so the element e of the error vector e _k Is defined by the following equation (5), and e of the equation (6) is ² Find coefficient data that minimizes The coefficient data is uniquely determined by the so-called least square method.
e _k = Y _k − ｛W ₁ × x _k1 + W ₂ × x _k2 + ‥‥ + W _n × x _kn ・ ・ ・… (5)
(K = 1,2, ‥‥ m)
[0210]
[Equation 3]

[0211]
E in equation (6) ² As a practical calculation method for finding coefficient data that minimizes the following equation, first, as shown in equation (7), e ² May be partially differentiated by coefficient data Wi (i = 1, 2,..., N), and the coefficient data Wi may be obtained such that the partial differential value becomes 0 for each value of i.
[0212]
(Equation 4)

[0213]
A specific procedure for obtaining the coefficient data Wi from the equation (7) will be described. When Xji and Yi are defined as in the expressions (8) and (9), the expression (7) can be written in the form of the determinant of the expression (10).
[0214]
(Equation 5)

[0215]
(Equation 6)

[0216]
The expression (10) is generally called a normal equation. By solving this normal equation by a general solution method such as a sweeping-out method (Gauss-Jordan elimination method), coefficient data Wi (i = 1, 2,..., N) can be obtained.
[0219]
FIG. 17 shows a configuration of a coefficient data generation device 150 that generates coefficient data Wi to be stored in the coefficient memory 134 of the first processing unit 121.
The coefficient data generating device 150 includes an input terminal 151 to which a teacher signal ST corresponding to the image signal Vb is input, an MPEG2 encoder 152 that encodes the teacher signal ST to obtain an MPEG2 stream, An MPEG2 decoder 153 for decoding the MPEG2 stream to obtain a student signal SS corresponding to the image signal Va. Here, the MPEG2 decoder 153 corresponds to the MPEG2 decoder 107 and the buffer memory 108 in the digital broadcast receiver 100 shown in FIG.
[0218]
Further, the coefficient data generating device 150 selectively outputs a plurality of pixel data of prediction taps used for prediction based on the student signal SS (I) of the I picture output from the decoder 153. Similarly, a class tap selection circuit 154 for selectively extracting a plurality of pixel data of class taps used for class classification based on the student signal SS (I) of the I picture output from the decoder 153, have. These tap selection circuits 154 and 155 are configured similarly to the tap selection circuits 131 and 132 of the first processing unit 121. The pixel data of the prediction tap and the class tap are pixel data located around the position of interest in the teacher signal ST (I) of the I picture in the spatial direction (horizontal direction, vertical direction).
[0219]
Further, the coefficient data generation device 150 includes a class generation circuit 156 that generates a class code CL1 indicating a space class from pixel data of a class tap selectively extracted by the class tap selection circuit 154. The class generation circuit 156 has the same configuration as the class generation circuit 133 of the first processing unit 121.
[0220]
Further, the coefficient data generation device 150 uses a delay circuit 157 for adjusting the time of the teacher signal ST supplied to the input terminal 151 and an I picture teacher signal ST (I) time-adjusted by the delay circuit 157. The obtained pixel data y of each target position, the pixel data xi of the prediction tap selectively extracted by the prediction tap selection circuit 155 corresponding to the pixel data y of each target position, and the pixel data y of each target position And a class code CL1 generated by the class generation circuit 156 in correspondence with the above, a normal equation (see the above-described equation (10)) for obtaining coefficient data Wi (i = 1 to n) is generated for each class. And a normal equation generating unit 158 that performs the processing.
[0221]
In this case, one piece of learning data is generated by a combination of one piece of pixel data y and corresponding n pieces of pixel data xi of prediction taps, but the teacher signal ST (I) and the student signal SS (I). In between, a lot of learning data is generated for each class. Thus, the normal equation generation unit 158 generates a normal equation for obtaining coefficient data Wi (i = 1 to n) for each class.
[0222]
The coefficient data generation device 150 is supplied with data of the normal equation generated by the normal equation generation unit 158, solves the normal equation, and obtains a coefficient data determination unit 159 for obtaining coefficient data Wi of each class. And a coefficient memory 160 for storing the coefficient data Wi of each class.
[0223]
Next, the operation of the coefficient data generation device 150 shown in FIG. 17 will be described.
The input terminal 151 is supplied with a teacher signal ST corresponding to the image signal Vb, and the MPEG2 encoder 152 encodes the teacher signal ST to generate an MPEG2 stream. This MPEG2 stream is supplied to the MPEG2 decoder 153. The MPEG2 stream is decoded by the MPEG2 decoder 153 to generate a student signal SS corresponding to the image signal Va.
[0224]
Based on the student signal SS (I) of the I picture generated by the decoder 153, the class tap selection circuit 155 locates in the space around the attention position in the teacher signal ST (I) of the I picture. Pixel data of the class tap is selectively extracted. The pixel data of the class tap is supplied to the class generation circuit 156. In the class generation circuit 156, a process such as 1-bit ADRC is performed on the pixel data of the class tap, and a class code CL1 indicating a space class is generated.
[0225]
Further, based on the student signal SS (I) of the I picture generated by the decoder 153, the prediction tap selection circuit 154 places the I-picture teacher signal ST (I) around the target position in the spatial direction with respect to the target position. Pixel data of the located prediction tap is selectively extracted.
[0226]
Then, the pixel data y at each target position obtained from the teacher signal ST (I) time-adjusted by the delay circuit 157 and the prediction tap selection circuit 154 selectively correspond to the pixel data y at each target position. Using the pixel data xi of the extracted prediction tap and the class code CL1 generated by the class generation circuit 156 corresponding to the pixel data y of each target position, the normal equation generation unit 158 generates a coefficient for each class. A normal equation (see equation (10)) for obtaining data Wi (i = 1 to n) is generated. This normal equation is solved by the coefficient data determination unit 159 to obtain coefficient data Wi of each class, and the coefficient data Wi is stored in the coefficient memory 160.
[0227]
As described above, the coefficient data generation device 150 shown in FIG. 17 can generate the coefficient data Wi of each class stored in the coefficient memory 134 of the first processing unit 121.
[0228]
The student signal SS is obtained by encoding the teacher signal ST to generate an MPEG2 stream, and then decoding the MPEG2 stream. Therefore, the student signal SS contains the same coding noise as the image signal Va. Therefore, in the first processing unit 121, the coding noise of the image signal Vb (I) obtained from the image signal Va (I) using the coefficient data Wi is reduced as compared with the image signal Va (I). It will be.
[0229]
FIG. 18 shows a configuration of a coefficient data generation device 170 that generates coefficient data Wi to be stored in the coefficient memory 147 of the second processing unit 122.
The coefficient data generating device 170 has an input terminal 171 to which a teacher signal ST corresponding to the image signal Vb is input, and a buffer memory 172 for temporarily storing the teacher signal ST.
[0230]
Further, the coefficient data generation device 170 encodes the teacher signal ST temporarily stored in the buffer memory 172 to obtain an MPEG2 stream, and decodes the MPEG2 stream. And an MPEG2 decoder 174 for obtaining a student signal SS corresponding to the image signal Va. Here, the MPEG2 decoder 174 corresponds to the MPEG2 decoder 107 and the buffer memory 108 in the digital broadcast receiver 100 shown in FIG.
[0231]
Further, the coefficient data generation device 170 includes a prediction tap selection circuit 175 for selectively extracting a plurality of pixel data of prediction taps used for prediction, and a class for selectively extracting a plurality of pixel data of class taps used for class classification. And a tap selection circuit 176. These tap selection circuits 175 and 176 have the same configuration as the tap selection circuits 141 and 142 of the second processing unit 122 described above.
[0232]
The tap selection circuits 175 and 176 output the student signals SS (P) / SS (B) corresponding to the target position in the teacher signal ST (P) / ST (B) of the P picture or B picture from the decoder 174. A predetermined position where the picture information PI paired with the pixel data and the pixel data of the student signal SS (P) / SS (B) detected by the motion vector detection unit 177 described later (hereinafter, referred to as a “pixel of interest”). The motion vector MV of the “data position”) is supplied as operation control information. The pixel data of the prediction tap and the class tap are spatially (horizontal and vertical) and temporal (frame direction) with respect to the target position in the teacher signal ST (P) / ST (B) of the P picture or B picture, respectively. ) Is the pixel data located around.
[0233]
The tap selection circuits 175 and 176 respectively provide a student signal SS (P) / SS (B) of a P picture or a B picture obtained by the decoder 174 and a student signal of the picture stored in the buffer memory 172. I-picture and P-picture teacher signals ST (I), corresponding to the I-picture and P-picture student signals SS (I) and SS (P) used to obtain SS (P) / SS (B), respectively. A plurality of pixel data is selectively extracted based on ST (P). Here, the tap selection circuits 175 and 176 perform a motion based on the student signal SS (P) / SS (B) with respect to the teacher signals ST (I) and ST (P) based on the above-described motion vector MV. Make compensation.
[0234]
Further, the coefficient data generation device 170 includes a motion vector detection unit 177 that detects the above-described motion vector MV. The student signal SS is supplied from the decoder 174 to the motion vector detection unit 177. The motion vector detection unit 177 has the same configuration as the motion vector detection unit 143 of the second processing unit 122 described above. The motion vector detection unit 177 detects a motion vector MV at a target pixel data position.
[0235]
The motion vector detection unit 177 outputs the lower-layer motion vector V0 and the minimum evaluation value EYmin in addition to the motion vector MV. As described above, the motion vector MV is used for motion compensation when selecting pixel data of a prediction tap and a class tap. The motion vector VO and the evaluation value EYmin are used for class classification as described later. You.
[0236]
The coefficient data generation device 170 has class generation circuits 178 and 179. The class generation circuit 178 generates a class code CL2 indicating a spatio-temporal class from the pixel data of the class tap selectively extracted by the class tap selection circuit 176. The class generation circuit 178 is configured similarly to the class generation circuit 144 of the second processing unit 122.
[0237]
The class generation circuit 179 performs a threshold determination on each of the motion vector V0 and the evaluation value EYmin related to the motion vector MV at the target pixel data position output from the motion vector detection unit 177, and generates a class code CLb indicating a motion-related class. Generate. The class generation circuit 178 has the same configuration as the class generation circuit 145 of the second processing unit 122.
[0238]
Further, the coefficient data generation device 170 has a class integration circuit 180 that integrates the class codes CLa and CLb generated by the class generation circuits 178 and 179 into one class code CL2. The class code CL2 indicates a class to which the pixel data of the target position in the teacher signal ST (P) / ST (B) of the P picture or the B picture belongs.
[0239]
Further, the coefficient data generation device 170 includes a delay circuit 181 for performing time adjustment of the teacher signal ST read from the buffer memory 172, and a P picture or B picture teacher signal ST (P) time-adjusted by the delay circuit 181. ) / ST (B), pixel data y of each target position, and pixel data xi of a prediction tap selectively extracted by the prediction tap selection circuit 175 in correspondence with the pixel data y of each target position. A normal equation for obtaining coefficient data Wi (i = 1 to n) for each class from the class code CL2 obtained by the class integration circuit 180 in correspondence with the pixel data y at each target position (the above-described (( 10) Refer to the equation).
[0240]
In this case, one piece of learning data is generated by a combination of one piece of pixel data y and pixel data xi of the n prediction taps corresponding thereto, but the learning signal ST (P) / ST (B) and the student signal ST A lot of learning data is generated for each class between the signal SS (P) / SS (B). Thus, the normal equation generation unit 182 generates a normal equation for obtaining coefficient data Wi (i = 1 to n) for each class.
[0241]
The coefficient data generation device 170 is supplied with data of the normal equation generated by the normal equation generation unit 182, solves the normal equation, and obtains a coefficient data determination unit 183 for obtaining coefficient data Wi of each class. And a coefficient memory 184 for storing the obtained coefficient data Wi of each class.
[0242]
Next, the operation of the coefficient data generation device 170 shown in FIG. 18 will be described.
A teacher signal ST corresponding to the image signal Vb is supplied to the input terminal 171, and the teacher signal ST is temporarily stored in the buffer memory 172. Then, the teacher signal ST temporarily stored in the buffer memory 172 is encoded by the MPEG2 encoder 173 to generate an MPEG2 stream. This MPEG2 stream is supplied to the MPEG2 decoder 174. The MPEG2 stream is decoded by the MPEG2 decoder 174 to generate a student signal SS corresponding to the image signal Va.
[0243]
The prediction tap selection circuit 175 and the class tap selection circuit 176 are paired with the pixel data of the student signal SS (P) / SS (B) corresponding to the target position of the teacher signal ST (P) / ST (B). The present picture information PI is supplied from the decoder 174, and the motion vector MV at the target pixel data position detected by the motion vector detection unit 177 is supplied.
[0244]
The student signal SS (P) / SS (B) of the P picture or B picture generated by the decoder 174 and the student signal SS (P) / SS (B) of the picture stored in the buffer memory 172 Based on the I-picture and P-picture teacher signals ST (I) and ST (P) corresponding to the I-picture and P-picture student signals SS (I) and SS (P) used to obtain The tap selection circuit 176 selectively extracts pixel data of class taps located around the target position in the teacher signal ST (P) / ST (B) of the P picture or B picture in the space direction and the time direction. . In this case, motion compensation is performed on the teacher signals ST (I) and ST (P) based on the student signal SS (P) / SS (B) based on the motion vector MV.
[0245]
The pixel data of the class tap selectively extracted by the class tap selection circuit 176 is supplied to the class generation circuit 178. In the class generation circuit 178, a process such as 1-bit ADRC is performed on the pixel data of the class tap, and a class code CLa indicating a space-time class is generated.
[0246]
The motion vector V0 and the evaluation value EYmin related to the motion vector MV output from the motion vector detection unit 177 are supplied to the class generation circuit 179. In the class generation circuit 179, a threshold is determined for each of the motion vector V0 and the evaluation value EYmin, and a class code CLb indicating a motion-related class is generated.
[0247]
The class codes CLa and CLb generated by the class generation circuits 178 and 179 are integrated by the class integration circuit 180, and the pixel data at the target position in the teacher signal ST (P) / ST (B) of the P picture or B picture is obtained. One class code CL2 indicating the class to which the user belongs is obtained.
[0248]
Further, the student signal SS (P) / SS (B) of the P picture or B picture generated by the decoder 174 and the student signal SS (P) / SS (S) of the picture stored in the buffer memory 172. B) based on the I-picture, P-picture teacher signals ST (I), ST (P) corresponding to the I-picture, P-picture student signals SS (I), SS (P) used to obtain The prediction tap selection circuit 175 selectively selects pixel data of prediction taps located in the spatial and temporal directions around the target position in the teacher signal ST (P) / ST (B) of the P picture or B picture. Taken out. In this case, motion compensation is performed on the teacher signals ST (I) and ST (P) based on the student signal SS (P) / SS (B) based on the motion vector MV.
[0249]
The pixel data y at each target position obtained from the teacher signal ST (P) / ST (B) time-adjusted by the delay circuit 181 and the prediction tap selection circuit corresponding to the pixel data y at each target position. Using the pixel data xi of the prediction tap selectively extracted at 175 and the class code CL2 obtained by the class integration circuit 180 corresponding to the pixel data y at each target position, the normal equation generation unit 182 generates a class Each time, a normal equation (see equation (10)) for obtaining coefficient data Wi (i = 1 to n) is generated. This normal equation is solved by the coefficient data determination unit 183 to obtain coefficient data Wi of each class, and the coefficient data Wi is stored in the coefficient memory 184.
[0250]
Thus, the coefficient data generation device 170 shown in FIG. 18 can generate the coefficient data Wi of each class stored in the coefficient memory 147 of the second processing unit 122.
[0251]
The student signal SS is obtained by encoding the teacher signal ST to generate an MPEG2 stream, and then decoding the MPEG2 stream. Therefore, the student signal SS contains the same coding noise as the image signal Va. Therefore, in the second processing unit 122, the image signal Vb (P) / Vb (B) obtained from the image signal Va (P) / Va (B) using the coefficient data Wi is converted into the image signal Va (P ) / Va (B) in which coding noise is reduced.
[0252]
Note that the processing in the image signal processing unit 110 in FIG. 1 can be realized by software using, for example, an image signal processing device 300 as shown in FIG.
[0253]
First, the image signal processing device 300 shown in FIG. 8 will be described. The image signal processing device 300 includes a CPU 301 for controlling the operation of the entire device, a ROM (Read Only Memory) 302 in which a control program for the CPU 301, coefficient data, and the like are stored, and a RAM (Random) for configuring a work area of the CPU 301. Access Memory) 303. The CPU 301, the ROM 302, and the RAM 303 are connected to a bus 304, respectively.
[0254]
Further, the image signal processing device 300 has a hard disk drive (HDD) 305 as an external storage device, and a drive (FDD) 307 for driving a floppy (registered trademark) disk 306. These drives 305 and 307 are connected to the bus 304, respectively.
[0255]
In addition, the image signal processing device 300 includes a communication unit 308 that connects to a communication network 400 such as the Internet by wire or wirelessly. The communication unit 308 is connected to the bus 304 via the interface 309.
[0256]
Further, the image signal processing device 300 includes a user interface unit. The user interface unit includes a remote control signal receiving circuit 310 that receives a remote control signal RM from the remote control transmitter 200, and a display 311 including an LCD (Liquid Crystal Display) or the like. The receiving circuit 310 is connected to the bus 304 via the interface 312, and similarly, the display 311 is connected to the bus 304 via the interface 313.
[0257]
Further, the image signal processing device 300 has an input terminal 314 for inputting the image signal Va, and an output terminal 315 for outputting the image signal Vb. The input terminal 314 is connected to the bus 304 via the interface 316, and the output terminal 315 is similarly connected to the bus 304 via the interface 317.
[0258]
Here, instead of previously storing the control program, coefficient data, and the like in the ROM 302 as described above, the control program and the coefficient data are downloaded from the communication network 400 such as the Internet via the communication unit 308 and stored in the hard disk or the RAM 303 for use. You can also. Further, these control programs, coefficient data, and the like may be provided on a floppy (registered trademark) disk 306.
[0259]
Instead of inputting the image signal Va to be processed from the input terminal 314, the image signal Va may be recorded in a hard disk in advance or downloaded from the communication network 400 such as the Internet via the communication unit 308. Alternatively, instead of outputting the processed image signal Vb to the output terminal 315 or in parallel with the output, the image signal Vb is supplied to the display 311 for image display, further stored in a hard disk, the Internet via the communication unit 308, or the like. May be transmitted to the communication network 400.
[0260]
With reference to the flowchart of FIG. 20, a processing procedure for obtaining the image signal Vb from the image signal Va in the image signal processing device 300 shown in FIG. 19 will be described.
[0261]
First, in step ST21, the process is started. In step ST22, an image signal Va for one GOP (Group Of Picture) is input into the apparatus from, for example, the input terminal 314. In this case, a motion compensation vector MI and picture information PI paired with the pixel data of the image signal Va are also input.
[0262]
As described above, the image signal Va and the like input from the input terminal 314 are temporarily stored in the RAM 303. When the image signals Va and the like are recorded in the hard disk drive 305 in the apparatus in advance, the image signals Va and the like are read from the drive 305 and the image signals Va and the like are temporarily stored in the RAM 303.
[0263]
Then, in step ST23, it is determined whether or not processing of all GOPs of the image signal Va has been completed. If the processing has ended, the processing ends in step ST24. On the other hand, if the processing has not been completed, the process proceeds to step ST25.
[0264]
In this step ST25, based on the image signal Va (I) of the I picture input in step ST22, a plurality of pixel data located in the periphery in the spatial direction with respect to the target position in the image signal Vb (I) of the I picture (Pixel data of a class tap) is selectively extracted, and a class code CL1 indicating the class to which the pixel data of the target position in the image signal Vb (I) of the I picture belongs is generated using the pixel data of the class tap.
[0265]
Next, in step ST26, based on the image signal Va (I) of the I picture input in step ST22, a plurality of pixels located in the spatial direction around the target position in the image signal Vb (I) of the I picture. Obtain pixel data (pixel data of a prediction tap). Then, in step ST27, using the coefficient data Wi corresponding to the class code CL1 generated in step ST25 and the pixel data xi of the prediction tap obtained in step ST26, based on the above estimation equation (1). Thus, the pixel data y at the target position in the image signal Vb (I) of the I picture is generated. The image signal Vb (I) of the I picture composed of the image data generated in this way is temporarily stored in the RAM 303.
[0266]
Then, in a step ST28, it is determined whether or not the processing of the I picture is completed. If not, the process returns to step ST25, and proceeds to the process for the next target position. On the other hand, if the processing has been completed, the process proceeds to step ST29 to shift to processing for obtaining image signals Vb (P) and Vb (B) of P and B pictures.
[0267]
The processing for obtaining these image signals Vb (P) and Vb (B) is performed in the same manner as the processing from step ST25 to step ST28 for obtaining the image signal Vb (I) for the I picture described above.
[0268]
However, the pixel data of the prediction tap and the class tap include the generated I-picture and P-picture image signals in addition to the P-picture and B-picture image signals Va (P) and Va (B) input in step ST22. It is also selectively extracted from Vb (I) and Vb (P).
[0269]
Also, the motion compensation vector MI for each macroblock is used as a motion vector of the upper layer, and the motion vector MV at the pixel data position of interest is detected hierarchically. Then, based on the image signals Va (P) and Va (B), motion compensation is performed on the image signals Vb (I) and Vb (P) based on the motion vector MV.
[0270]
When the processing of the predetermined P picture or B picture is completed in step ST29, it is determined in step ST30 whether there is a next picture to be processed. If there is a next picture, the process returns to step ST29 and shifts to a process of obtaining an image signal of the picture. On the other hand, when there is no next picture to be processed, the process proceeds to step ST31.
[0271]
In step ST31, the generated image signals Vb (I), Vb (P), and Vb (B) for one GOP are rearranged from the encoding order to the actual frame / field order and output to the output terminal 315. I do. Then, thereafter, the process returns to step ST22 to shift to the input processing of the image signal Va for the next 1 GOP.
[0272]
In this way, by performing the processing according to the flowchart shown in FIG. 20, the pixel data of the input image signal Va can be processed, and the pixel data of the image signal Vb can be obtained. As described above, the image signal Vb obtained by the above-described processing is output to the output terminal 315, supplied to the display 311 to display an image based thereon, and further supplied to the hard disk drive 305 to be supplied to the hard disk drive 305. Or recorded in
[0273]
Further, although illustration of the processing device is omitted, the processing in the coefficient data generation device 150 in FIG. 17 can also be realized by software.
[0274]
The processing procedure for generating coefficient data will be described with reference to the flowchart in FIG.
First, in step ST31, the process starts, and in step ST32, the teacher signal ST is input for one GOP. Then, in step ST33, it is determined whether or not processing of all GOPs of the teacher signal ST has been completed. If not completed, in step ST34, a student signal SS is generated from the teacher signal ST input in step ST32.
[0275]
Next, in step ST35, based on the student signal SS (I) of the I picture generated in step ST34, a plurality of positions located around the attention position in the teacher signal ST (I) of the I picture in the spatial direction. Pixel data (pixel data of a class tap) is selectively extracted, and a class code CL1 indicating the class to which the pixel data of the target position in the teacher signal ST (I) of the I picture belongs is generated using the pixel data of the class tap. .
[0276]
Next, in step ST36, based on the student signal SS (I) of the I picture generated in step ST34, a plurality of positions located around the target position in the teacher signal ST (I) of the I picture in the spatial direction. Obtain pixel data (pixel data of a prediction tap).
[0277]
In step ST37, the class code CL1 generated in step ST35, the pixel data xi of the prediction tap obtained in step ST36, and the pixel data y of the target position in the teacher signal ST (I) are used for each class. Addition is performed to obtain the normal equation shown in equation (10) (see equations (8) and (9)).
[0278]
Next, in step ST38, it is determined whether or not the learning process has been completed in all regions of the pixel data of the teacher signal ST (I) of the I picture input in step ST32. If the learning process has been completed, the process returns to step ST32, where the teacher signal ST for the next 1 GOP is input, and the same process as described above is repeated. On the other hand, if the learning process has not been completed, the process returns to step ST35 and proceeds to the process for the next target position.
[0279]
When the process is completed in step ST33 described above, in step ST39, the normal equation of each class generated by the addition process in step ST37 is solved by a sweeping method or the like to calculate coefficient data Wi of each class. I do. Then, in step ST40, the coefficient data Wi of each class is stored in the memory, and thereafter, in step ST41, the process ends.
[0280]
In this way, by performing the processing according to the flowchart shown in FIG. 21, it is possible to obtain the coefficient data Wi of each class relating to the I picture by the same method as the coefficient data generating device 150 shown in FIG.
[0281]
Although detailed description is omitted, the processing in the coefficient data generation device 170 in FIG. 18 can be realized by software, similarly to the processing in the coefficient data generation device 150 in FIG. 17 described above.
[0282]
In that case, instead of the I-picture teacher signal ST (I) and the student signal SS (I), the P-picture teacher signal ST (P), the student signal SS (P) or the B-picture teacher signal ST (B), The student signal SS (B) is used. In addition, when obtaining the pixel data of the class tap and the prediction tap, in addition to the student signal SS (P) / SS (B), it is used when obtaining the student signal SS (P) / SS (B). Teacher signals ST (I) and ST (P) corresponding to the student signals SS (I) and SS (P) are also used. Thereby, the coefficient data Wi of each class relating to the P picture or the B picture can be obtained.
[0283]
Also, the motion compensation vector MI for each macroblock is used as a motion vector of the upper layer, and the motion vector MV at the pixel data position of interest is detected hierarchically. Then, motion compensation is performed on the teacher signals ST (I) and ST (P) based on the motion vector MV with reference to the student signals SS (P) and SS (B).
[0284]
Next, a second embodiment of the present invention will be described. FIG. 22 shows a configuration of a digital broadcast receiver 500 according to the second embodiment.
The digital broadcast receiver 500 includes a microcomputer, and includes a system controller 501 for controlling the operation of the entire system, and a remote control signal receiving circuit 502 for receiving a remote control signal RM. The remote control signal receiving circuit 502 is connected to the system controller 501, receives a remote control signal RM output from the remote control transmitter 200 in response to a user operation, and supplies an operation signal corresponding to the signal RM to the system controller 501. It is configured to be.
[0285]
The digital broadcast receiver 500 is supplied with a receiving antenna 505 and a broadcast signal (RF modulated signal) captured by the receiving antenna 505, performs channel selection processing, demodulation processing, error correction processing, and the like, and executes a predetermined program. And a tuner unit 506 for obtaining an MPEG2 stream as an encoded image signal according to the above.
[0286]
The digital broadcast receiver 500 decodes the MPEG2 stream output from the tuner unit 506 to obtain an image signal Va, and temporarily converts the image signal Va output from the MPEG2 decoder 507 into an image signal Va. And a buffer memory 508 for temporarily storing.
[0287]
Note that, in the present embodiment, the MPEG2 decoder 507 outputs picture information PI and a motion compensation vector MI in addition to each pixel data forming the image signal Va. The buffer memory 508 also stores the picture information PI and the motion compensation vector MI in pairs with each pixel data.
[0288]
The picture information PI is information indicating which of the I picture, the P picture, and the B picture the output pixel data relates to. When the output pixel data is related to a P picture or a B picture, the motion compensation vector MI is information indicating how the reference data for decoding the pixel data has been motion compensated. As is well known, the motion compensation vector MI is a motion vector detected by the MPEG2 encoder for each macroblock.
[0289]
The MPEG2 decoder 507 has the same configuration as the MPEG2 decoder 107 of the digital broadcast receiver 100 shown in FIG. 1 described above (see FIG. 2). The decoder 507 outputs image signals Va of each picture in the order of encoding. In the present embodiment, when reading the image signal Va from the buffer memory 508 described above, the image signal of each picture is rearranged from the encoding order to the actual frame / field order.
[0290]
Incidentally, in the encoding of the MPEG system, as is conventionally known, encoding is performed in an order different from an actual frame / field order. That is, the image signals of the I picture and the P picture are encoded first, and the image signal of the B picture sandwiched between them is encoded thereafter.
[0291]
The digital broadcast receiver 500 further includes an image signal processing unit 510 that converts the image signal Va stored in the buffer memory 508 into an image signal Vb having a different number of frames from the image signal Va, A display unit 511 for displaying an image based on the image signal Vb output from the unit 510. The display unit 511 is configured by a display such as a CRT display or an LCD.
[0292]
In the present embodiment, the image signal Vb has five times the number of frames as the image signal Va. For example, when the frame configuration of the image signal Va is as shown in FIG. 23A, the frame configuration of the image signal Vb is as shown in FIG. 23B. In this case, the image signal Vb has a frame f between frames of each picture of the image signal Va. ₁ ~ F ₄ Is inserted. Note that f _I , F _P , F _B Indicates frames of an I picture, a P picture, and a B picture, respectively.
[0293]
The operation of the digital broadcast receiver 500 shown in FIG. 22 will be described.
The MPEG2 stream output from the tuner unit 506 is supplied to an MPEG2 decoder 507 and decoded. The image signal Va output from the decoder 507 is supplied to the buffer memory 508 and is temporarily stored. In this case, the decoder 507 outputs the picture information PI and the motion compensation vector MI in pairs with each pixel data of the image signal Va. These are also temporarily stored in the buffer memory 508.
[0294]
The image signal Va temporarily stored in the buffer memory 508 as described above is supplied to the image signal processing unit 510, and is converted into an image signal Vb having five times the number of frames. In the image signal processing unit 510, the frame f of the image signal Vb is calculated based on the pixel data constituting the image signal Va. ₁ ~ F ₄ Is generated. In the image signal processing unit 510, a conversion process is performed as described later using the picture information PI and the motion compensation vector MI stored in the buffer memory 508.
[0295]
The image signal Vb output from the image signal processing unit 510 is supplied to a display unit 511, and an image based on the image signal Vb is displayed on a screen of the display unit 511.
[0296]
Next, details of the image signal processing unit 510 will be described.
The image signal processing unit 510 outputs the frame f of the image signal Vb. ₁ ~ F ₄ And a frame f of each picture stored in the buffer memory 508. _I , F _P , F _B And the frame f generated by the image signal generation unit 521 ₁ ~ F ₄ And an output selection circuit 522 for selectively taking out the above and obtaining an image signal Vb.
[0297]
The image signal generator 521 will be described. The image signal generation unit 521 generates a frame f based on a plurality of continuous frames of the image signal Va, in this embodiment, three continuous frames. ₁ ~ F ₄ Generate
[0298]
In this case, frame f ₁ , F ₂ Is the frame f ₁ , F ₂ And two frames whose time direction positions are ahead of the frame f ₁ , F ₂ Is generated using one frame that has a time-direction position after it. On the other hand, frame f ₃ , F ₄ Is the frame f ₃ , F ₄ One frame whose time direction position is ahead of ₃ , F ₄ Is generated using the two frames whose time-direction positions are behind.
[0299]
However, the frame f of the I picture _I And frame f of B picture _B Frame f between ₁ , F ₂ With respect to the frame f ₁ , F ₂ One frame whose time direction position is ahead of ₃ , F ₄ Is generated using the two frames whose time-direction positions are behind.
[0300]
Also, in this case, when generating pixel data of a target position in a predetermined frame, motion vectors MV1, MV2, and MV3 between the target position and three frames are detected, and these motion vectors MV1, MV2, and MV3 are detected. Based on the predetermined frame, motion compensation of three frames is performed, and the frame f ₁ ~ F ₄ Improve the quality of
[0301]
For example, the frame f of two B pictures _B , F _B Frame f between ₁ In the case of generating the pixel data of the target position in the I picture, as shown in FIG. _I Between the motion vector MV1 and the frame f of the first B picture _B Between the motion vector MV2 and the frame f of the second B picture _B And the motion vector MV3 between them.
[0302]
Also, for example, the frame f of two B pictures _B , F _B Frame f between ₃ When generating the pixel data at the target position in the frame B of the first B picture as shown in FIG. _B Between the motion vector MV1 and the frame f of the second B picture _B And the motion vector MV2 between P _P And the motion vector MV3 between them.
[0303]
When detecting the motion vectors MV1 to MV3, a motion vector between a second frame and a predetermined position corresponding to a target position existing in the first frame related to the three frames (hereinafter, “ MV ') is detected, and the motion vectors MV1 to MV3 are obtained by linearly distributing the motion vectors MV'.
[0304]
For example, when the motion vectors MV1 to MV3 shown in FIG. 24A are detected, as shown in FIG. _B Is the first frame, and this frame f _B , The frame f of the I picture which is the second frame at the predetermined position _I Between the motion vector MVI and the frame f of the first B picture _B Is the first frame, and this frame f _B , The frame f of the P picture which is the second frame _P Are detected.
[0305]
Then, the motion vectors MVI and MV2 are obtained by multiplying the motion vector MVI by 6/5 and 1/5, respectively. The motion vector MV3 is obtained by multiplying the motion vector MVP by 2/5. When the motion vectors MV1 to MV3 are obtained by linearly allocating the motion vectors MVI and MVP as described above, a portion of one pixel or less is rounded off.
[0306]
Also, for example, when detecting the motion vectors MV1 to MV3 shown in FIG. 24B, as shown in FIG. _B Is the first frame, and this frame f _B , The frame f of the I picture which is the second frame at the predetermined position _I Between the motion vector MVI and the frame f of the second B picture _B Is the first frame, and this frame f _B , The frame f of the P picture which is the second frame _P Are detected.
[0307]
Then, the motion vector MVI is multiplied by 3/10 to obtain the motion vector MV1. Further, the motion vectors MVP and MV3 are obtained by multiplying the motion vector MVP by 2/5 and 7/5, respectively.
[0308]
When detecting the use motion vector MV ′ such as the above-described motion vector MVI, motion vector MVP, etc., as in the above-described motion vector detection unit 143 of the second processing unit 122 (see FIG. 4), Hierarchical detection is performed using the motion compensation vector MI of a macroblock located in the vicinity as a higher-layer motion vector.
[0309]
FIG. 26 illustrates a configuration of the image signal generation unit 521.
The image signal generation unit 521 includes a prediction tap selection circuit 541 for selectively extracting a plurality of pixel data of prediction taps used for prediction, and a class tap for selectively extracting a plurality of pixel data of class taps used for classification. And a selection circuit 542.
[0310]
The tap selection circuits 541 and 542 each determine a predetermined frame (f) of the image signal Vb based on three consecutive frames of the image signal Va. ₁ ~ F ₄ ), A plurality of pieces of pixel data located around the space of interest (horizontal direction, vertical direction) and time direction (frame direction) with respect to the target position are selectively extracted.
[0311]
The tap selection circuits 541 and 542 are provided between the target frame in the predetermined frame of the image signal Vb detected by the motion vector detection unit 543 described later and the three frames described above (hereinafter, referred to as “used frames”). Of motion vectors MV1, MV2, and MV3 are supplied. The tap selection circuits 541 and 542 perform motion compensation on three use frames based on a predetermined frame of the image signal Vb based on the motion vectors MV1, MV2 and MV3.
[0312]
As described above, the prediction tap selection circuit 541 selectively extracts pixel data of a prediction tap from three motion-compensated use frames. Therefore, it is possible to improve the quality of the pixel data of the target position in the predetermined frame of the image signal generated as described later using the pixel data of the prediction tap.
[0313]
The class tap selection circuit 542 selectively extracts pixel data of a prediction tap from three motion-compensated use frames. Therefore, as described above, the spatio-temporal class corresponding to the pixel data of the prediction tap selected by the motion compensation performed by the prediction tap selection circuit 541 can be detected with high accuracy.
[0314]
The image signal generation unit 521 has a motion vector detection unit 543. When detecting the motion vectors MV1 to MV3, the motion vector detection unit 543 detects the use motion vector MV 'and linearly allocates the detected use motion vector MV' to the motion vectors MV (MV1 to MV3). ).
[0315]
The motion vector detection unit 543 detects the use motion vector MV ′ hierarchically. In this case, similarly to the motion vector detection unit 143 of the second processing unit 122 (see FIG. 4), the motion compensation vector MI of a plurality of macroblocks located around the predetermined position is replaced with the motion vector of the upper layer. Used as
[0316]
FIG. 27 illustrates a configuration of the motion vector detection unit 543. 27, parts corresponding to those in FIG. 13 are denoted by the same reference numerals, and detailed description thereof will be omitted as appropriate.
[0317]
The motion vector detection unit 543 includes a motion vector selection circuit 21 that selects the motion compensation vectors MI of four macroblocks located around a predetermined position of the first frame as motion vectors of an upper layer, and the selection circuit The four motion vectors MI selected at 21 ₁ ~ MI ₄ And a motion vector accumulation unit 22 for accumulating the motion vector. The position information of the predetermined position is supplied to the motion vector selection circuit 21 as operation control information. Which of the four macroblocks located around the predetermined position is to be used is determined in the same manner as described with reference to FIG. However, in the description, the “target pixel data position” is replaced with the “predetermined position”.
[0318]
In addition, the motion vector detecting unit 543 selectively reads pixel data of a reference block centered on a predetermined position from a reference frame (first frame) stored in the buffer memory 508 (see FIG. 22). The circuit includes a circuit 23 and a search block circuit 24 that sequentially reads out image data corresponding to each candidate block within the search range from a search frame (second frame) stored in the buffer memory 508.
[0319]
Here, the search range of the search frame is a range in which the position corresponding to the target pixel data position has been moved based on the higher-layer motion vector as a reference position. The search block circuit 24 outputs the four motion vectors MI stored in the motion vector storage unit 22 as the motion vector of the upper hierarchy. ₁ ~ MI ₄ Will be used sequentially. Then, the search block circuit 24 sequentially reads the pixel data of the plurality of candidate blocks in the search range determined by the motion vector of each upper layer.
[0320]
Further, the motion vector detection unit 543 uses the pixel data of the reference block from the reference block circuit 23 and the image data of each candidate block from the search block circuit 24, and uses the evaluation function E ( Y) _n And an evaluation value calculation circuit 25 for obtaining an evaluation value of each candidate block based on the Here, the evaluation value is correlation information indicating the degree of correlation between the reference block and the candidate block, and the evaluation value calculation circuit 25 constitutes a correlation detecting unit.
[0321]
The search block circuit 24 supplies the evaluation value calculation circuit 25 with the pixel data of the candidate block, as well as the motion vector of the upper hierarchy that has determined the search range in which the candidate block exists. The evaluation value calculation circuit 25 stores the calculated evaluation value of each candidate block in an evaluation value memory (not shown) for each higher-layer motion vector.
[0322]
Further, the motion vector detection unit 543 includes the motion vector detection circuit 26. The motion vector detection circuit 26 sequentially evaluates the evaluation value of each candidate block calculated by the evaluation value calculation circuit 25, and calculates a motion vector (u) corresponding to the candidate block giving the minimum evaluation value. _n , V _n ) Is the lower-layer motion vector V0. Then, the final motion vector MV 'is obtained by the above-mentioned equation (3). However, the MV in the expression (3) is MV 'here.
[0323]
In addition, the motion vector detecting unit 543 includes the linear distribution circuit 29 that linearly allocates the use motion vector MV ′ detected by the motion vector detection circuit 26 to obtain the motion vector MV. The motion vector MV is a motion vector corresponding to a time-direction position of a predetermined frame of the image signal Vb. In this case, when the motion vector MV is obtained by linearly allocating the use motion vector MV 'as described above, a portion of one pixel or less is rounded off.
[0324]
The operation of the motion vector detection unit 543 shown in FIG. 27 will be described.
The motion vector selection circuit 21 selects a motion compensation vector MI of four macroblocks located around the predetermined position as a higher-layer motion vector based on the position information of the predetermined position of the first frame. . The four motion vectors MI thus selected ₁ ~ MI ₄ Are supplied to the motion vector storage unit 22 and stored.
[0325]
For example, as shown in FIG. ₁ Three frames f at the position of interest _I , F _B , F _B When detecting the motion vectors MV1, MV2, and MV3 between the motion vectors MV1, MV2, and MV3, it is necessary to detect the motion vector MVI and the motion vector MVP as shown in FIG. 25A as the use motion vector MV '.
[0326]
When detecting the motion vector MVI, the motion vector selecting unit 21 sets the frame f of the first B picture _B Is the first frame, and this frame f _B Frame f of the I picture as the second frame of the four macroblocks located around the predetermined position of _I Between the motion vector MI and the motion vector MI ₁ ~ MI ₄ Select as
[0327]
When detecting the motion vector MVP, the motion vector selecting unit 21 sets the frame f of the first B picture _B Is the first frame, and this frame f _B Of the four macroblocks located around the predetermined position of the P picture as the second frame _P Between the motion vector MI and the motion vector MI ₁ ~ MI ₄ Select as
[0328]
Further, the reference block circuit 23 selectively reads pixel data of a reference block centering on a predetermined position from a reference frame stored in the buffer memory 508 (see FIG. 22). Further, the search block circuit 24 sequentially reads out image data corresponding to each candidate block within the search range from the search frame stored in the buffer memory 508.
[0329]
In this case, the search range of the search frame is a range in which a position corresponding to the predetermined position has been moved based on a higher-layer motion vector as a reference position. The search block circuit 24 calculates the four motion vectors MI stored in the motion vector storage unit 22 as the motion vector of the upper hierarchy. ₁ ~ MI ₄ Are sequentially used, and pixel data of a plurality of candidate blocks are sequentially read out in the search range determined by the motion vector of each upper layer.
[0330]
The pixel data of the reference block read by the reference block circuit 23 and the image data of each candidate block read by the search block circuit 24 are supplied to the evaluation value calculation circuit 25. The evaluation value calculation circuit 25 calculates the evaluation function E (Y) of the above equation (2). _n , An evaluation value of each candidate block is obtained.
[0331]
The motion vector detection circuit 26 sequentially evaluates the evaluation value of each candidate block calculated by the evaluation value calculation circuit 25, and selects a candidate block giving the minimum evaluation value, that is, a motion corresponding to the candidate block having the highest correlation with the reference block. Vector (u _n , V _n ) Is the lower-layer motion vector V0. Then, the motion vector (u, v) obtained by adding the motion vector V1 of the upper layer of the candidate block to the motion vector V0 of the lower layer is output as the final use motion vector MV '.
[0332]
Then, the use motion vector MV ′ obtained by the motion vector detection circuit 26 is supplied to the linear distribution circuit 29. The linear allocation circuit 29 linearly allocates the use motion vector MV 'to obtain and output a motion vector MV (MV1 to MV3) of a target position in a predetermined frame of the image signal Vb.
[0333]
Here, as shown in FIG. ₁ Three frames f at the position of interest _I , F _B , F _B The case where the motion vectors MV1, MV2, MV3 between the two are detected will be described.
[0334]
The frame f of the first B picture in the motion vector selection circuit 21 _B Of the I picture of the four macroblocks located around the predetermined position _I Is selected, and the search block circuit 24 selects the frame f of the I picture. _I Is set as a search frame, and the candidate blocks are read out, so that the motion vector MBI shown in FIG. 25A is obtained from the motion vector detection circuit 26 as the use motion vector MV ′. The linear distribution circuit 29 multiplies the motion vector MBI by 6/5 and 1/5, respectively, to obtain the motion vectors MV1 and MV2.
[0335]
Also, the motion vector selection circuit 21 outputs the frame f of the first B picture. _B Frame P of the P picture of the four macroblocks located around the predetermined position _P Is selected, and the search block circuit 24 selects the frame f of the P picture _P Is used as a search frame, and the candidate block is read, so that the motion vector detection circuit 26 obtains the motion vector MBP shown in FIG. Then, the linear distribution circuit 29 multiplies the motion vector MBI by 2/5 to obtain a motion vector MV3.
[0336]
As described above, the motion vector detection unit 543 hierarchically detects the use motion vector MV ′ using the motion compensation vector MI provided for each macroblock as a higher-layer motion vector. Can be detected quickly. In addition, the motion vector detection unit 543 uses not only the motion compensation vector MI of the macroblock in which the predetermined position exists but also the motion compensation vector MI of the macroblock around the macroblock as the upper-layer motion vector. The use motion vector MV ′ can be detected with high accuracy.
[0337]
Note that the motion vector detection unit 543 shown in FIG. 27 selects the motion compensation vector MI of a macroblock located around a predetermined position as a higher-layer motion vector. A frequency distribution of the motion compensation vector MI of each macroblock may be generated, and a predetermined number of motion vectors having higher frequencies may be selected as higher-layer motion vectors. Thereby, the detection accuracy of the use motion vector MV 'can be further improved.
[0338]
Returning to FIG. 26, the image signal generation unit 521 has a class generation circuit 544. The class generation circuit 544 generates a class code CL indicating a spatio-temporal class from the pixel data of the class tap selectively extracted by the class tap selection circuit 542. The class generation circuit 544 generates a class code CL indicating a spatio-temporal class by performing 1-bit processing such as ADRC on the pixel data of the class tap. The class code CL indicates a spatio-temporal class to which pixel data of a target position in a predetermined frame of the image signal Vb belongs.
[0339]
Further, the image signal generation unit 521 has a coefficient memory 545. The coefficient memory 545 stores, for each class, coefficient data Wi (i = 1 to n, where n is the number of prediction taps) used in an estimation formula used in an estimation prediction operation circuit 546 described later. . The coefficient memory 545 stores coefficient data Wi for converting the image signal Va into the image signal Vb.
[0340]
The coefficient data Wi stored in the coefficient memory 545 is generated in advance by learning between a student signal corresponding to the image signal Va and a teacher signal corresponding to the image signal Vb. The class code CL generated by the above-described class generation circuit 544 is supplied to the coefficient memory 545 as read address information. The coefficient data Wi of the estimation formula corresponding to the class code CL is read from the coefficient memory 545 and supplied to the estimation prediction operation circuit 546. A method for generating the coefficient data Wi will be described later.
[0341]
Further, the image signal generation unit 521 obtains the prediction tap pixel data xi selectively extracted by the prediction tap selection circuit 541 and the coefficient data Wi read from the coefficient memory 545 by using the above-described estimation equation (1). , An estimated prediction calculation circuit 546 that calculates pixel data y of a target position in a predetermined frame of the image signal Vb to be created.
[0342]
The operation of the image signal generator 521 will be described.
The class tap selection circuit 542 determines a predetermined frame (f) of the image signal Vb based on three consecutive use frames of the image signal Va stored in the buffer memory 508. ₁ ~ F ₄ A), a plurality of pieces of pixel data located around the space of interest and in the time direction with respect to the target position are selectively extracted.
[0343]
In this case, based on the motion vectors MV1, MV2, MV3 between the three use frames at the position of interest in the predetermined frame of the image signal Vb detected by the motion vector detection unit 543, the three use frames Is subjected to motion compensation based on a predetermined frame of the image signal Vb.
[0344]
The pixel data of the class tap extracted by the class tap selection circuit 542 is supplied to the class generation circuit 544. The class generation circuit 544 performs a process such as 1-bit ADRC on the pixel data of the class tap, and generates a class code CL indicating a space-time class.
[0345]
The class code CL is supplied to the coefficient memory 545 as read address information. The coefficient data Wi corresponding to the class code CL is read from the coefficient memory 545, and the read coefficient data Wi is supplied to the estimated prediction operation circuit 546.
[0346]
In addition, the prediction tap selection circuit 541 uses the three consecutive use frames of the image signal Va stored in the buffer memory 508 to position the target position in the predetermined frame of the image signal Vb in the spatial direction and the time direction. Are selectively taken out.
[0347]
In this case, based on the motion vectors MV1, MV2, and MV3 between the three use frames at the position of interest in the predetermined frame of the image signal Vb detected by the motion vector detection unit 543, the three use frames Is subjected to motion compensation based on a predetermined frame of the image signal Vb.
[0348]
The estimation prediction operation circuit 546 uses the pixel data xi of the prediction tap and the coefficient data Wi read from the coefficient memory 545 to calculate the image signal Vb to be created based on the estimation expression shown in the above expression (1). The pixel data y at the position of interest in the predetermined frame is obtained.
[0349]
As described above, the image signal generation unit 521 obtains a predetermined frame of the image signal Vb using three consecutive use frames of the image signal Va. In this case, a plurality of pixel data (pixel data of a prediction tap) located around the target position in a predetermined frame of the image signal Vb, which is selected based on three consecutive use frames of the image signal Va, and this image signal Using the coefficient data Wi corresponding to the class CL to which the pixel data of the target position in the predetermined frame of Vb belongs, the pixel data y of the target position in the predetermined frame of the image signal Vb is generated based on the estimation formula.
[0350]
In this case, the plurality of pixel data of the prediction tap is based on three consecutive use frames of the image signal Va, which are motion-compensated based on the motion vectors MV (MV1 to MV3) detected by the motion vector detection unit 543. It is selected and has a high correlation. Therefore, it is possible to improve the quality of the pixel data of the target position in the predetermined frame of the image signal Vb generated using the pixel data of the prediction tap.
[0351]
In addition, the pixel data of the class tap is selected based on three consecutive use frames of the image signal Va, which are motion-compensated based on the motion vectors MV (MV1 to MV3) detected by the motion vector detection unit 543. Things. Therefore, the class code CL generated by the class generation circuit 544 accurately indicates the spatiotemporal class corresponding to the pixel data of the prediction tap selected by the motion compensation performed by the prediction tap selection circuit 541. Therefore, the accuracy of class classification can be improved, and the quality of a predetermined frame in the image signal Vb can be improved.
[0352]
Returning to FIG. 22, the operation of the image signal processing unit 510 will be described.
The frame f of each picture of the image signal Va stored in the buffer memory 508 _I , F _P , F _B Are read out in the order of the actual frames and supplied to the output selection circuit 522.
[0353]
Further, the image signal generation unit 521 uses three consecutive use frames of the image signal Va stored in the buffer memory 508 to generate a frame f between frames of each picture. ₁ ~ F ₄ Generate This frame f ₁ ~ F ₄ Are supplied to the output selection circuit 522.
[0354]
In the output selection circuit 522, the frame f of each picture _I , F _P , F _B And the frame f generated by the image signal generation unit 521 ₁ ~ F ₄ Are selectively taken out. Therefore, from the output selection circuit 522, an image signal Vb in which the number of frames is five times that of the image signal Va is obtained.
[0355]
FIG. 28 illustrates a configuration of a coefficient data generation device 570 that generates coefficient data Wi to be stored in the coefficient memory 545 of the image signal generation unit 521.
The coefficient data generation device 570 includes an input terminal 571 to which a teacher signal ST corresponding to the image signal Vb is input, and an MPEG2 stream obtained by thinning the teacher signal ST to 1/5 and reducing the number of frames to 1/5. And an MPEG2 decoder 573 that decodes the MPEG2 stream to obtain a student signal SS corresponding to the image signal Va. Here, the MPEG2 decoder 573 corresponds to the MPEG2 decoder 507 and the buffer memory 508 in the digital broadcast receiver 500 shown in FIG.
[0356]
Further, the coefficient data generation device 570 includes a prediction tap selection circuit 574 for selectively extracting a plurality of pixel data of prediction taps used for prediction, and a class for selectively extracting a plurality of pixel data of class taps used for class classification. And a tap selection circuit 575. These tap selection circuits 574 and 575 have the same configuration as the tap selection circuits 541 and 542 of the above-described image signal generation unit 521 (see FIG. 26).
[0357]
The tap selection circuits 574 and 575 respectively determine a predetermined frame (f) of the teacher signal ST based on three consecutive frames of the student signal SS. ₁ ~ F ₄ ), A plurality of pieces of pixel data located around the space of interest (horizontal direction, vertical direction) and time direction (frame direction) with respect to the target position are selectively extracted.
[0358]
The tap selection circuits 574 and 575 have a position between the above-mentioned three frames (hereinafter, referred to as “use frames”) of the target position in the predetermined frame of the teacher signal ST detected by the motion vector detection unit 576 described later. The motion vectors MV1, MV2, MV3 are supplied. The tap selection circuits 574 and 575 perform motion compensation on three use frames based on a predetermined frame of the teacher signal ST based on the motion vectors MV1, MV2, and MV3.
[0359]
In addition, the coefficient data generation device 570 includes a motion vector detection unit 576 that detects the above-described motion vectors MV1, MV2, and MV3. The motion vector detection unit 576 has the same configuration as the above-described motion vector detection unit 543 (see FIG. 27) of the image signal generation unit 521. In this case, when detecting the motion vectors MV1 to MV3, the use motion vector MV 'is detected, and the detected use motion vectors MV' are linearly distributed to obtain the motion vectors MV (MV1 to MV3).
[0360]
Further, the coefficient data generation device 570 includes a class generation circuit 577. The class generation circuit 577 generates a class code CL indicating a spatio-temporal class from the pixel data of the class tap selectively extracted by the class tap selection circuit 575. This class generation circuit 577 is configured similarly to the class generation circuit 544 of the image signal generation unit 521 described above. The class code CL indicates the class to which the pixel data of the target position in the predetermined frame of the teacher signal ST belongs.
[0361]
Further, the coefficient data generation device 570 includes a delay circuit 578 for performing time adjustment of the teacher signal ST input to the input terminal 571, and each of the delay circuits 578 obtained from a predetermined frame of the teacher signal ST time-adjusted by the delay circuit 578. It corresponds to the pixel data y of the target position, the pixel data xi of the prediction tap selectively extracted by the prediction tap selection circuit 574 corresponding to the pixel data y of each target position, and the pixel data y of each target position. A normal equation for generating coefficient data Wi (i = 1 to n) (see the above equation (10)) for each class from the class code CL generated by the class generation circuit 577. A generation unit 579 is provided.
[0362]
In this case, one piece of learning data is generated by a combination of one piece of pixel data y and the corresponding n pieces of pixel data xi of prediction taps, but the learning data is generated between a predetermined frame of the teacher signal ST and the student signal SS. Thus, a lot of learning data is generated for each class. Thus, the normal equation generation unit 579 generates a normal equation for obtaining coefficient data Wi (i = 1 to n) for each class.
[0363]
The coefficient data generation device 570 is supplied with data of the normal equation generated by the normal equation generation unit 579, solves the normal equation, and obtains a coefficient data determination unit 580 for obtaining coefficient data Wi of each class. And a coefficient memory 581 for storing the obtained coefficient data Wi of each class.
[0364]
Next, the operation of the coefficient data generation device 570 shown in FIG. 18 will be described.
The input terminal 571 is supplied with a teacher signal ST corresponding to the image signal Vb. The teacher signal ST is subjected to encoding after the number of frames is reduced to 1/5 by the MPEG2 encoder 572 to generate an MPEG2 stream. This MPEG2 stream is supplied to an MPEG2 decoder 573. The MPEG2 stream is decoded by the MPEG2 decoder 573 to generate a student signal SS corresponding to the image signal Va.
[0365]
The prediction tap selection circuit 574 and the class tap selection circuit 575 include a motion vector MV1 between the three use frames of the student signal SS at the target position in the predetermined frame of the teacher signal ST detected by the motion vector detection unit 576. , MV2, MV3.
[0366]
Based on three consecutive use frames of the student signal SS generated by the decoder 573, the class tap selection circuit 575 determines the position of the teacher signal ST in the spatial and temporal directions relative to the target position in the predetermined frame. Are selectively extracted. In this case, based on the motion vectors MV1, MV2, and MV3, motion compensation is performed on the three use frames with reference to a predetermined frame of the teacher signal ST.
[0367]
The pixel data of the class tap selectively extracted by the class tap selection circuit 575 is supplied to the class generation circuit 577. In the class generation circuit 577, a process such as 1-bit ADRC is performed on the pixel data of the class tap, and a class code CL indicating a space-time class is generated.
[0368]
In addition, based on three consecutive use frames of the student signal SS generated by the decoder 573, the prediction tap selection circuit 574 uses the prediction tap selection circuit 574 to position the target position in the predetermined frame of the teacher signal ST in the spatial and temporal directions. Are selectively taken out. In this case, based on the motion vectors MV1, MV2, and MV3, motion compensation is performed on the three use frames with reference to a predetermined frame of the teacher signal ST.
[0369]
The pixel data y at each target position obtained from a predetermined frame of the teacher signal ST time-adjusted by the delay circuit 578 and the prediction tap selection circuit 574 selectively correspond to the pixel data y at each target position. Using the extracted prediction tap pixel data xi and the class code CL generated by the class generation circuit 577 corresponding to the pixel data y of each target position, the normal equation generation unit 579 generates a coefficient for each class. A normal equation (see equation (10)) for obtaining data Wi (i = 1 to n) is generated. This normal equation is solved by the coefficient data determination unit 580 to obtain coefficient data Wi of each class, and the coefficient data Wi is stored in the coefficient memory 581.
[0370]
Thus, the coefficient data generation device 570 shown in FIG. 28 can generate the coefficient data Wi of each class stored in the coefficient memory 545 of the image signal generation unit 521.
[0371]
Note that the processing in the image signal processing unit 510 in FIG. 22 can be realized by software using, for example, an image signal processing device 300 as shown in FIG. With reference to the flowchart of FIG. 29, a processing procedure for obtaining the image signal Vb from the image signal Va in the image signal processing device 300 shown in FIG. 19 will be described.
[0372]
First, in step ST51, the process is started. In step S52, image signals Va for a plurality of frames are input into the apparatus from the input terminal 314, for example. In this case, the motion compensation vector MI paired with the pixel data of the image signal Va is also input.
[0373]
As described above, the image signal Va and the like input from the input terminal 314 are temporarily stored in the RAM 303. When the image signals Va and the like are recorded in the hard disk drive 305 in the apparatus in advance, the image signals Va and the like are read from the drive 305 and the image signals Va and the like are temporarily stored in the RAM 303.
[0374]
Then, in a step ST53, it is determined whether or not the processing of all the frames of the image signal Va has been completed. If the processing has ended, the processing ends in step ST54. On the other hand, if the processing has not been completed, the process proceeds to step ST55.
[0375]
In this step ST55, the frame f of the input image signal Va is _n Is output to the output terminal 315. Thereafter, the process proceeds to step ST56, where the frame f of this image signal Va is _n And the next frame _{n + 1} Frame f to be inserted between ₁ ~ F ₄ Move on to the generation processing of. In step ST56, N = 1 is set.
[0376]
Next, in step ST57, the frame f of the image signal Vb is calculated using the motion compensation vector MI. _N , Motion vectors MV1 to MV3 between the three use frames of the image signal Va at the position of interest in are detected. Frame f _N Is initially f ₁ It is.
[0377]
Then, in step ST58, the frame f of the image signal Vb is determined on the basis of three consecutive use frames of the image signal Va input in step ST52. _N , A plurality of pixel data (pixel data of class taps) located around the time direction and the space direction with respect to the target position in (a) are acquired, and the frame f of the image signal Vb is _N , A class code CL indicating the class to which the pixel data at the target position belongs is generated.
[0378]
Next, in step ST59, the frame f of the image signal Vb is determined based on the three consecutive use frames of the image signal Va input in step ST52. _N A plurality of pixel data (pixel data of the prediction tap) located around the time and space directions with respect to the target position in.
[0379]
In step ST58 and step ST59, three use frames of the image signal Va are divided into frames f based on the motion vectors MV1 to MV3 detected in step ST57. _N , Pixel data is obtained after the motion compensation.
[0380]
Next, in step ST60, using the coefficient data Wi corresponding to the class code CL generated in step ST58 and the pixel data xi of the prediction tap obtained in step ST59, based on the estimation expression of expression (1). , The pixel data y of the target position in the image signal Vb is generated.
[0381]
Then, in step ST61, the frame f _N It is determined whether or not the processing for generating the. Has been completed. If not, the process returns to step ST57, and proceeds to the process for the next target position. On the other hand, if the processing has been completed, the process proceeds to step ST62, where the frame f _N Is output to the output terminal 315.
[0382]
Next, in step ST63, it is determined whether or not N = Nmax. Here, Nmax = 4. If N is not equal to Nmax, N is incremented in step ST64, and thereafter, the flow returns to step ST57 to return to the next frame f _{N + 1} Move on to the generation processing of. On the other hand, when N = Nmax, the frame f ₁ ~ F ₄ Is generated, which means that the output is completed, and the process proceeds to step ST65.
[0383]
In step ST65, it is determined whether or not the processing corresponding to the image signals Va for a plurality of frames input in step ST52 has been completed. If the processing has been completed, the process returns to step ST52, and shifts to input processing of image signals Va for the next plurality of frames. On the other hand, if the processing has not been completed, the process returns to step ST55, where the next frame of the image signal Va is output to the output terminal 315, and the frame f to be inserted between that frame and the next frame is output. ₁ ~ F ₄ Move on to the generation processing of.
[0384]
In this way, by performing the processing according to the flowchart shown in FIG. 29, the input image signal Va is processed, and an image signal Vb having five times the number of frames can be obtained.
[0385]
Although illustration of the processing device is omitted, the processing in the coefficient data generation device 570 in FIG. 28 can also be realized by software.
[0386]
A processing procedure for generating coefficient data will be described with reference to the flowchart in FIG.
First, in step ST71, the process is started, and in step ST72, the teacher signal ST is input for a plurality of frames. Then, in a step ST73, it is determined whether or not the processing of all the frames of the teacher signal ST is completed. If not, in step ST74, a student signal SS is generated from the teacher signal ST input in step ST72. In this case, a motion compensation vector MI paired with the pixel data of the student signal SS is also obtained.
[0387]
In step ST75, a predetermined frame of the teacher signal ST (the frame f of the image signal Vb) is ₁ ~ F ₄ Are detected, and the motion vectors MV1 to MV3 between the three consecutive use frames of the student signal SS at the position of interest in the target position are detected.
[0388]
Then, in step ST76, based on three consecutive use frames of the student signal SS generated in step ST74, a plurality of frames located in the time direction and the space direction around the target position in the predetermined frame of the teacher signal ST. The pixel data (pixel data of the class tap) is acquired, and a class code CL indicating the class to which the pixel data of the target position in the predetermined frame of the teacher signal ST belongs is generated using the pixel data of the class tap.
[0389]
Next, in step ST77, based on three consecutive use frames of the student signal SS generated in step ST74, a plurality of frames located around the time frame and the space direction with respect to the target position in the predetermined frame of the teacher signal ST. (Pixel data of the prediction tap).
[0390]
In steps ST76 and ST77, after three consecutive use frames of the student signal SS are motion-compensated based on the predetermined frame of the teacher signal ST based on the motion vectors MV1 to MV3 detected in step ST75, Acquisition of pixel data is performed.
[0391]
Next, in step ST78, the class code CL generated in step ST76, the pixel data xi of the prediction tap obtained in step ST77, and the pixel data y of the target position in the predetermined frame of the teacher signal ST are used for each class. , (10) are added to obtain the normal equation (see equations (8) and (9)).
[0392]
Next, in step ST79, it is determined whether or not the learning process corresponding to the teacher signals ST for a plurality of frames input in step ST72 has been completed. If the learning process has been completed, the process returns to step ST72 to input the teacher signal ST for the next plurality of frames, and repeats the same process as described above. On the other hand, if the learning process has not been completed, the process returns to step ST75 and proceeds to the process for the next target position.
[0393]
When the processing is completed in step ST73, in step ST80, the normal equation of each class generated by the addition processing in step ST78 is solved by a sweeping method or the like to calculate coefficient data Wi of each class. I do. Then, in step ST81, the coefficient data Wi of each class is stored in the memory, and thereafter, in step ST82, the processing is ended.
[0394]
In this way, by performing the processing according to the flowchart shown in FIG. 30, the coefficient data Wi of each class can be obtained by the same method as the coefficient data generation device 570 shown in FIG.
[0395]
In the above-described embodiment, the case of handling an MPEG2 stream accompanied by DCT has been described. However, the present invention can be similarly applied to a case of handling other encoded digital information signals. Also, instead of DCT, coding involving other orthogonal transform such as wavelet transform and discrete sine transform may be used.
[0396]
In the above-described embodiment, the motion vector detection unit 143 (see FIG. 13) and the motion vector detection unit 543 (see FIG. 27) use the evaluation value as correlation information as the sum of absolute differences. It may be a sum of squares.
[0397]
【The invention's effect】
According to the present invention, when a motion vector at a predetermined position in a first frame that is divided into a plurality of blocks and has a motion vector between each block and a second frame is detected, the motion vector around the predetermined position is detected. A motion vector of a plurality of blocks located is selected as a motion vector of an upper layer, and a plurality of motions in which a reference block centered on a predetermined position of the first frame and a position corresponding to the predetermined position of the second frame are selected Detects correlation information between candidate blocks within a plurality of search ranges, each of which has a position moved based on a vector as a reference position, and calculates position information of a candidate block having the highest correlation with a reference block as a motion vector at a predetermined position. And a motion vector at a predetermined position can be detected quickly and accurately.
[0398]
According to the invention, a first image signal composed of a plurality of pixel data, which is generated by decoding an encoded digital image signal, is converted into a second image signal composed of a plurality of pixel data. When the coding noise is reduced by the conversion, in addition to the first image signal, the second image after the coding noise reduction, which is motion-compensated by the motion vector detected by using the above-described motion vector detecting device or the like. Based on the image signal, a plurality of first pixel data located in the space direction and the time direction around the target position in the second image signal are selected, and a plurality of first pixel data are selected using the plurality of first pixel data. The pixel data of the target position in the second image signal is generated, and the quality of the second image signal can be improved.
[0399]
According to the present invention, when a first image signal composed of a plurality of pixel data is converted into a second image signal composed of a plurality of pixel data, which has a different number of frames from the first image signal. Based on a plurality of continuous frames of the first image signal, motion-compensated by a motion vector detected using the above-described motion vector detection device or the like, a position of interest in a predetermined frame of the second image signal A method of selecting a plurality of first pixel data located around the space direction and the time direction and using the plurality of first pixel data to generate pixel data of a target position in a predetermined frame of a second image signal Thus, the quality of the second image signal can be improved.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of a digital broadcast receiver according to a first embodiment.
FIG. 2 is a block diagram illustrating a configuration of an MPEG2 decoder.
FIG. 3 is a block diagram illustrating a configuration of a first processing unit.
FIG. 4 is a block diagram illustrating a configuration of a second processing unit.
FIG. 5 is a diagram for explaining motion compensation.
FIG. 6 is a diagram for explaining a block matching method.
FIG. 7 is a diagram for explaining a block matching method.
FIG. 8 is a diagram for explaining a block matching method.
FIG. 9 is a diagram for explaining a block matching method.
FIG. 10 is a diagram for explaining selection of a motion vector of an upper layer.
FIG. 11 is a diagram for explaining selection of a motion vector of an upper layer.
FIG. 12 is a flowchart illustrating a motion vector detection process.
FIG. 13 is a block diagram illustrating a configuration of a motion vector detection unit.
FIG. 14 is a block diagram illustrating a configuration of an evaluation value calculation circuit.
FIG. 15 is a block diagram illustrating a configuration of a motion vector detection circuit.
FIG. 16 is a block diagram illustrating a modified example of the motion vector detection unit.
FIG. 17 is a block diagram illustrating a configuration of a coefficient data generation device.
FIG. 18 is a block diagram illustrating a configuration of a coefficient data generation device.
FIG. 19 is a block diagram illustrating a configuration example of an image signal processing device realized by software.
FIG. 20 is a flowchart illustrating image signal processing.
FIG. 21 is a flowchart illustrating a coefficient data generation process.
FIG. 22 is a block diagram illustrating a configuration of a digital broadcast receiver according to a second embodiment.
FIG. 23 is a diagram illustrating a frame configuration of image signals Va and Vb.
FIG. 24 is a diagram for explaining motion vectors MV1 to MV3.
FIG. 25 is a diagram for explaining a use motion vector MV ′ (MVI, MVP).
FIG. 26 is a block diagram illustrating a configuration of an image signal generation unit.
FIG. 27 is a block diagram illustrating a configuration of a motion vector detection unit.
FIG. 28 is a block diagram illustrating a configuration of a coefficient data generation device.
FIG. 29 is a flowchart illustrating image signal processing.
FIG. 30 is a flowchart showing coefficient data generation processing.
[Explanation of symbols]
21, 28: motion vector selection circuit, 22: motion vector storage unit, 23: reference block circuit, 24: search block circuit, 25: evaluation value circuit, 26: motion vector Detection circuit 27 frequency distribution generation circuit 29 linear distribution circuit 100 digital broadcast receiver 101 system controller 102 remote control signal reception circuit 105 reception Antenna, 106: Tuner unit, 107: MPEG2 decoder, 108, 123, 125: Buffer memory, 110: Image signal processing unit, 111: Display unit, 121 ... 1 processing unit, 122 ... second processing unit, 124 ... rearrangement unit, 131, 141 ... prediction tap selection circuit, 132, 142 ... class tap selection , 133, 144, 145 ... class generation circuit, 134, 147 ... coefficient memory, 135, 148 ... estimated prediction calculation circuit, 143 ... motion vector detection unit, 146 ... class integration circuit .., 150, 170, 570... Coefficient data generating device, 300... Image signal processing device, 500... Digital broadcast receiver, 501. .. A receiving antenna, 506 a tuner, 507 an MPEG2 decoder, 508 a buffer memory, 510 an image signal processor, 511 a display, 521 an image signal Generation unit, 522: output selection circuit, 541: prediction tap selection circuit, 542: class tap selection circuit, 543: motion vector Detector, 544 ... class generating circuit, 545 ... coefficient memory, 546 ... estimation predictive calculating circuit

Claims (21)

A motion vector detection device which is divided into a plurality of blocks and detects a motion vector at a predetermined position in a first frame having a motion vector between each block and a second frame,
A motion vector selecting unit that selects a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy,
A reference block centered on the predetermined position of the first frame and a position where a position corresponding to the predetermined position of the second frame is moved based on a plurality of motion vectors selected by the motion vector selecting means. Correlation detection means for detecting correlation information between candidate blocks in a plurality of search ranges each having a reference position,
Based on the correlation information corresponding to the plurality of candidate blocks detected by the correlation detection unit, the motion vector detection unit that sets the position information of the candidate block having the highest correlation with the reference block as the motion vector at the predetermined position. A motion vector detecting device, comprising: A frequency distribution generating unit configured to generate a frequency distribution of a motion vector of each block of the first frame;
The said candidate vector selection means, further based on the frequency distribution produced | generated by the said frequency distribution production | generation means, selects the motion vector of a predetermined number from the higher frequency as a candidate vector of a higher hierarchy, The claim 2. The motion vector detection device according to 1. A motion vector detection method for detecting a motion vector at a predetermined position in a first frame that is divided into a plurality of blocks and has a motion vector between each block and a second frame,
A first step of selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A position where a reference block centered on the predetermined position of the first frame and a position corresponding to the predetermined position of the second frame have been moved based on the plurality of motion vectors selected in the first step. A second step of detecting correlation information between candidate blocks in a plurality of search ranges, each of which is a reference position,
A third step in which, based on the correlation information corresponding to the plurality of candidate blocks detected in the second step, position information of a candidate block having the highest correlation with the reference block is set as a motion vector at the predetermined position; A motion vector detection method comprising: In order to detect a motion vector at a predetermined position in a first frame divided into a plurality of blocks and having a motion vector between each block and a second frame,
A first step of selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A position where a reference block centered on the predetermined position of the first frame and a position corresponding to the predetermined position of the second frame have been moved based on the plurality of motion vectors selected in the first step. A second step of detecting correlation information between candidate blocks in a plurality of search ranges, each of which is a reference position,
A third step in which, based on the correlation information corresponding to the plurality of candidate blocks detected in the second step, position information of a candidate block having the highest correlation with the reference block is set as a motion vector at the predetermined position; For causing a computer to execute a motion vector detection method comprising: An image signal processing device for converting a first image signal composed of a plurality of pixel data, generated by decoding an encoded digital image signal, into a second image signal composed of a plurality of pixel data. hand,
A predetermined position corresponding to a target position in the second image signal of the first frame of the first image signal, which is divided into a plurality of blocks and has a motion vector between each block and a second frame A motion vector detecting unit for detecting a motion vector of
The second frame based on the first frame of the first image signal and the second frame of the second image signal on which motion compensation has been performed by the motion vector detected by the motion vector detection unit. Data selection means for selecting a plurality of first pixel data located in the space direction and the time direction around the position of interest in the image signal of
Pixel data generation means for generating pixel data of a target position in the second image signal using the plurality of first pixel data selected by the data selection means,
The motion vector detection unit includes:
A motion vector selecting unit that selects a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy,
A reference block centered on the predetermined position of the first frame and a position where a position corresponding to the predetermined position of the second frame is moved based on a plurality of motion vectors selected by the motion vector selecting means. Correlation detection means for detecting correlation information between candidate blocks in a plurality of search ranges each having a reference position,
Based on the correlation information corresponding to the plurality of candidate blocks detected by the correlation detection unit, the motion vector detection unit that sets the position information of the candidate block having the highest correlation with the reference block as the motion vector at the predetermined position. An image signal processing device comprising: The pixel data generating means includes:
Class detection means for detecting a class to which the pixel data at the position of interest in the second image signal belongs;
A coefficient data generation unit that generates coefficient data used in the estimation formula, corresponding to the class detected by the class detection unit;
Using the coefficient data generated by the coefficient data generating means and the plurality of first pixel data selected by the data selecting means, the pixel data at the position of interest in the second image signal based on the estimation formula 6. The image signal processing apparatus according to claim 5, further comprising: an arithmetic unit configured to calculate the image signal. The class detecting means includes at least:
Selected based on the first frame of the first image signal and the second frame of the second image signal on which motion compensation has been performed by the motion vector detected by the motion vector detection unit; A class to which the pixel data of the target position in the second image signal belongs is detected by using a plurality of second pixel data located around the position of interest in the second image signal in the spatial and temporal directions. The image signal processing device according to claim 6, wherein The class detecting means includes at least:
Using the position information from the reference position of the search range of the candidate block having the highest correlation with the reference block or the correlation information of the candidate block having the highest correlation, the class to which the pixel data of the target position in the second image signal belongs The image signal processing device according to claim 6, wherein An image signal processing method for converting a first image signal composed of a plurality of pixel data generated by decoding an encoded digital image signal into a second image signal composed of a plurality of pixel data. hand,
A predetermined position corresponding to a target position in the second image signal of the first frame of the first image signal, which is divided into a plurality of blocks and has a motion vector between each block and a second frame A first step of detecting the motion vector of the first image signal, and the second image signal of which the motion compensation has been performed by the first frame of the first image signal and the motion vector detected in the first step. A second step of selecting, based on the second frame, a plurality of pieces of first pixel data located around the spatial position and the temporal direction with respect to the target position in the second image signal;
A third step of generating pixel data of a target position in the second image signal using the plurality of first pixel data selected in the second step,
The first step is
Selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A reference block centered on the predetermined position of the first frame and a position where the position corresponding to the predetermined position of the second frame is moved based on the selected plurality of motion vectors are referred to as reference positions, respectively. Detecting correlation information between candidate blocks in the plurality of search ranges,
Setting the position information of the candidate block having the highest correlation with the reference block on the basis of the correlation information corresponding to the plurality of detected candidate blocks as a motion vector at the predetermined position. Processing method. To convert a first image signal composed of a plurality of pixel data, which is generated by decoding an encoded digital image signal, into a second image signal composed of a plurality of pixel data,
A predetermined position corresponding to a target position in the second image signal of the first frame of the first image signal, which is divided into a plurality of blocks and has a motion vector between each block and a second frame A first step of detecting a motion vector of
The second frame based on the first frame of the first image signal and the second frame of the second image signal which has been motion-compensated by the motion vector detected in the first step. A second step of selecting a plurality of first pixel data located in the space direction and the time direction with respect to the target position in the image signal of
A third step of generating pixel data of a target position in the second image signal using the plurality of first pixel data selected in the second step,
The first step is
Selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A reference block centered on the predetermined position of the first frame and a position where the position corresponding to the predetermined position of the second frame is moved based on the selected plurality of motion vectors are referred to as reference positions, respectively. Detecting correlation information between candidate blocks in the plurality of search ranges,
Based on the correlation information corresponding to the plurality of detected candidate blocks, setting the position information of the candidate block having the highest correlation with the reference block as a motion vector at the predetermined position. The program to be executed. Estimation used in converting a first image signal composed of a plurality of pixel data, which is generated by decoding an encoded digital image signal, into a second image signal composed of a plurality of pixel data. An apparatus for generating coefficient data of an equation,
Decoding means for decoding a digital image signal obtained by encoding the teacher signal corresponding to the second image signal to obtain a student signal corresponding to the first image signal;
A motion vector detecting unit that detects a motion vector at a predetermined position corresponding to a target position in the teacher signal in a first frame that is divided into a plurality of blocks and has a motion vector between each block and a second frame; When,
Based on the first frame of the student signal and the second frame of the teacher signal motion-compensated by the motion vector detected by the motion vector detection unit, Data selection means for selecting a plurality of pixel data located around the space direction and the time direction,
Calculating means for obtaining the coefficient data using a plurality of pixel data selected by the data selecting means and pixel data of a target position in the teacher signal,
The motion vector detection unit includes:
A motion vector selecting unit that selects a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy,
A reference block centered on the predetermined position of the first frame and a position where a position corresponding to the predetermined position of the second frame is moved based on a plurality of motion vectors selected by the motion vector selecting means. Correlation detection means for detecting correlation information between candidate blocks in a plurality of search ranges each having a reference position,
Based on the correlation information corresponding to the plurality of candidate blocks detected by the correlation detection unit, the motion vector detection unit that sets the position information of the candidate block having the highest correlation with the reference block as the motion vector at the predetermined position. A coefficient data generation device characterized by having: Estimation used when converting a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data, which is generated by decoding an encoded digital image signal. A method for generating coefficient data for an equation, comprising:
A first step of decoding a digital image signal obtained by encoding a teacher signal corresponding to the second image signal to obtain a student signal corresponding to the first image signal;
A second step of detecting a motion vector at a predetermined position corresponding to a target position in the teacher signal of a first frame divided into a plurality of blocks and having a motion vector between each block and a second frame; When,
Based on the first frame of the student signal and the second frame of the teacher signal motion-compensated by the motion vector detected in the second step, a position of interest in the teacher signal is A third step of selecting a plurality of pixel data located around the space direction and the time direction;
A fourth step of obtaining the coefficient data using the plurality of pixel data selected in the third step and the pixel data of the target position in the teacher signal.
The second step is
Selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A reference block centered on the predetermined position of the first frame and a position where the position corresponding to the predetermined position of the second frame is moved based on the selected plurality of motion vectors are referred to as reference positions, respectively. Detecting correlation information between candidate blocks in the plurality of search ranges,
Setting the position information of the candidate block having the highest correlation with the reference block as the motion vector at the predetermined position based on the correlation information corresponding to the plurality of detected candidate blocks. Generation method. Estimation used in converting a first image signal composed of a plurality of pixel data, which is generated by decoding an encoded digital image signal, into a second image signal composed of a plurality of pixel data. To generate the coefficient data for the equation,
A first step of decoding a digital image signal obtained by encoding a teacher signal corresponding to the second image signal to obtain a student signal corresponding to the first image signal;
A second step of detecting a motion vector at a predetermined position corresponding to a target position in the teacher signal of a first frame divided into a plurality of blocks and having a motion vector between each block and a second frame; When,
Based on the first frame of the student signal and the second frame of the teacher signal motion-compensated by the motion vector detected in the second step, a position of interest in the teacher signal is A third step of selecting a plurality of pixel data located around the space direction and the time direction;
A fourth step of obtaining the coefficient data using the plurality of pixel data selected in the third step and the pixel data of the target position in the teacher signal.
The second step is
Selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A reference block centered on the predetermined position of the first frame and a position where the position corresponding to the predetermined position of the second frame is moved based on the selected plurality of motion vectors are referred to as reference positions, respectively. Detecting correlation information between candidate blocks in the plurality of search ranges,
The position information of the candidate block having the highest correlation with the reference block as the motion vector at the predetermined position based on the correlation information corresponding to the plurality of detected candidate blocks. The program to be executed. An image signal processing device for converting a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data, having a different number of frames from the first image signal,
A motion vector detecting unit that detects a motion vector between a plurality of consecutive frames of the first image signal, based on a position of interest in a predetermined frame of the second image signal,
Based on a plurality of continuous frames of the first image signal, on which motion compensation has been performed based on a plurality of motion vectors detected by the motion vector detection unit, a position of interest in a predetermined frame of the second image signal is determined. Data selection means for selecting a plurality of first pixel data located around the spatial direction and the temporal direction,
Pixel data generating means for generating pixel data of a target position in a predetermined frame of the second image signal using the plurality of first pixel data selected by the data selecting means;
Part or all of the frame of the first image signal is divided into a plurality of blocks, each block having a motion vector between the other frames,
The motion vector detection unit includes:
A motion vector between a second frame and a predetermined position corresponding to a target position in a predetermined frame of the second image signal which is present in a first frame related to a plurality of continuous frames of the first image signal; A first processing unit for detecting
A second processing unit that linearly distributes the motion vector detected by the first processing unit to obtain a motion vector corresponding to a time-direction position of a predetermined frame of the second image signal;
The first processing unit includes:
A motion vector selecting unit that selects a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy,
A reference block centered on the predetermined position of the first frame and a position where a position corresponding to the predetermined position of the second frame is moved based on a plurality of motion vectors selected by the motion vector selecting means. Correlation detection means for detecting correlation information between candidate blocks in a plurality of search ranges each having a reference position,
Based on the correlation information corresponding to the plurality of candidate blocks detected by the correlation detection unit, the motion vector detection unit that sets the position information of the candidate block having the highest correlation with the reference block as the motion vector at the predetermined position. An image signal processing device comprising: The pixel data generating means includes:
Class detection means for detecting a class to which the pixel data at the position of interest in the second image signal belongs;
A coefficient data generation unit that generates coefficient data used in the estimation formula, corresponding to the class detected by the class detection unit;
Using the coefficient data generated by the coefficient data generating means and the plurality of first pixel data selected by the data selecting means, the position of interest of the second image signal in a predetermined frame based on the estimation formula 15. The image signal processing device according to claim 14, further comprising: an arithmetic unit that calculates and calculates the pixel data. The class detecting means includes at least:
In a predetermined frame of the second image signal selected based on a plurality of continuous frames of the first image signal on which motion compensation has been performed based on the plurality of motion vectors detected by the motion vector detection unit. A class to which the pixel data of the target position in a predetermined frame of the second image signal belongs is detected by using a plurality of second pixel data located around the target position in the spatial direction and the time direction. The image signal processing device according to claim 15, wherein An image signal processing method for converting a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal,
A first step of detecting a motion vector between a plurality of consecutive frames of the first image signal corresponding to a position of interest in a predetermined frame of the second image signal;
Based on a plurality of continuous frames of the first image signal, the motion compensation of which has been performed based on the plurality of motion vectors detected in the first step, a position of interest in a predetermined frame of the second image signal A second step of selecting a plurality of first pixel data located around the spatial direction and the temporal direction,
A third step of using the plurality of first pixel data selected in the second step to generate pixel data of a target position in a predetermined frame of the second image signal,
Part or all of the frame of the first image signal is divided into a plurality of blocks, each block having a motion vector between the other frames,
The first step is
A motion vector between the second frame and a second frame at a predetermined position corresponding to a target position in the predetermined frame, in the second image signal present in a first frame related to a plurality of continuous frames of the first image signal; A first processing step of detecting
A second processing step of linearly allocating the motion vector detected in the first processing step to obtain a motion vector corresponding to a time-direction position of a predetermined frame of the second image signal;
The first processing step includes:
Selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A reference block centered on the predetermined position of the first frame and a position where the position corresponding to the predetermined position of the second frame is moved based on the selected plurality of motion vectors are referred to as reference positions, respectively. Detecting correlation information between candidate blocks in the plurality of search ranges,
Setting the position information of the candidate block having the highest correlation with the reference block on the basis of the correlation information corresponding to the plurality of detected candidate blocks as a motion vector at the predetermined position. Processing method. In order to convert a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal,
A first step of detecting a motion vector between a plurality of consecutive frames of the first image signal corresponding to a position of interest in a predetermined frame of the second image signal;
Based on a plurality of continuous frames of the first image signal, the motion compensation of which has been performed based on the plurality of motion vectors detected in the first step, a position of interest in a predetermined frame of the second image signal A second step of selecting a plurality of first pixel data located around the spatial direction and the temporal direction,
A third step of using the plurality of first pixel data selected in the second step to generate pixel data of a target position in a predetermined frame of the second image signal,
Part or all of the frame of the first image signal is divided into a plurality of blocks, each block having a motion vector between the other frames,
The first step is
A motion vector between the second frame and a second frame at a predetermined position corresponding to a target position in the predetermined frame, in the second image signal present in a first frame related to a plurality of continuous frames of the first image signal; A first processing step of detecting
A second processing step of linearly allocating the motion vector detected in the first processing step to obtain a motion vector corresponding to a time-direction position of a predetermined frame of the second image signal;
The first processing step includes:
Selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A reference block centered on the predetermined position of the first frame and a position where the position corresponding to the predetermined position of the second frame is moved based on the selected plurality of motion vectors are referred to as reference positions, respectively. Detecting correlation information between candidate blocks in the plurality of search ranges,
Based on the correlation information corresponding to the plurality of detected candidate blocks, setting the position information of the candidate block having the highest correlation with the reference block as a motion vector at the predetermined position. The program to be executed. Coefficient of an estimation formula used when converting a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal An apparatus for generating data,
A motion vector detecting unit that detects a motion vector between a plurality of continuous frames of the student signal corresponding to the first image signal, which is related to a target position in a predetermined frame of the teacher signal corresponding to the second image signal; ,
Based on a plurality of continuous frames of the student signal, on which motion compensation has been performed based on the plurality of motion vectors detected by the motion vector detection unit, a spatial direction and a time with respect to a position of interest in a predetermined frame of the teacher signal. Data selection means for selecting a plurality of pixel data located around the direction,
Calculating means for obtaining the coefficient data using a plurality of pixel data selected by the data selecting means and pixel data of a target position in the teacher signal,
Some or all frames of the student signal are divided into a plurality of blocks, each block having a motion vector between other frames,
The motion vector detection unit includes:
A first step of detecting a motion vector between a second frame and a predetermined position corresponding to a target position in a predetermined frame of the teacher signal, which is present in a first frame related to a plurality of continuous frames of the student signal; A processing unit;
A second processing unit for linearly allocating the motion vector detected by the first processing unit to obtain a motion vector corresponding to a time-direction position of a predetermined frame of the teacher signal,
The first processing unit includes:
A motion vector selecting unit that selects a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy,
A reference block centered on the predetermined position of the first frame and a position where a position corresponding to the predetermined position of the second frame is moved based on a plurality of motion vectors selected by the motion vector selecting means. Correlation detection means for detecting correlation information between candidate blocks in a plurality of search ranges each having a reference position,
Based on the correlation information corresponding to the plurality of candidate blocks detected by the correlation detection unit, the motion vector detection unit that sets the position information of the candidate block having the highest correlation with the reference block as the motion vector at the predetermined position. A coefficient data generation device characterized by having: Coefficient of an estimation formula used when converting a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal A method of generating data,
A first step of detecting a motion vector between a plurality of continuous frames of the student signal corresponding to the first image signal, which is related to a target position in a predetermined frame of the teacher signal corresponding to the second image signal; ,
Based on a plurality of continuous frames of the student signal, the motion of which has been compensated based on the plurality of motion vectors detected in the first step, a spatial direction and a time with respect to a target position in a predetermined frame of the teacher signal. A second step of selecting a plurality of pixel data located in the periphery of the direction;
A third step of obtaining the coefficient data using the plurality of pixel data selected in the second step and the pixel data of the target position in the teacher signal.
Some or all frames of the student signal are divided into a plurality of blocks, each block having a motion vector between other frames,
The first step is
A first step of detecting a motion vector between a second frame and a predetermined position corresponding to a target position in a predetermined frame of the teacher signal, which is present in a first frame related to a plurality of continuous frames of the student signal; Processing steps;
A second processing step of linearly allocating the motion vector detected in the first processing step to obtain a motion vector corresponding to a temporal position of a predetermined frame of the teacher signal,
The first processing step includes:
Selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A reference block centered on the predetermined position of the first frame and a position where the position corresponding to the predetermined position of the second frame is moved based on the selected plurality of motion vectors are referred to as reference positions, respectively. Detecting correlation information between candidate blocks in the plurality of search ranges,
Setting the position information of the candidate block having the highest correlation with the reference block as the motion vector at the predetermined position based on the detected correlation information corresponding to the plurality of candidate blocks. Data generation method. Coefficient of an estimation formula used when converting a first image signal composed of a plurality of pixel data into a second image signal composed of a plurality of pixel data having a different number of frames from the first image signal To generate data,
A first step of detecting a motion vector between a plurality of continuous frames of the student signal corresponding to the first image signal, which is related to a target position in a predetermined frame of the teacher signal corresponding to the second image signal; ,
Based on a plurality of continuous frames of the student signal, the motion of which has been compensated based on the plurality of motion vectors detected in the first step, a spatial direction and a time with respect to a target position in a predetermined frame of the teacher signal. A second step of selecting a plurality of pixel data located in the periphery of the direction;
A third step of obtaining the coefficient data using the plurality of pixel data selected in the second step and the pixel data of the target position in the teacher signal.
Some or all frames of the student signal are divided into a plurality of blocks, each block having a motion vector between other frames,
The first step is
A first step of detecting a motion vector between a second frame and a predetermined position corresponding to a target position in a predetermined frame of the teacher signal, which is present in a first frame related to a plurality of continuous frames of the student signal; Processing steps;
A second processing step of linearly allocating the motion vector detected in the first processing step to obtain a motion vector corresponding to a temporal position of a predetermined frame of the teacher signal,
The first processing step includes:
Selecting a motion vector of a plurality of blocks located around the predetermined position as a motion vector of a higher hierarchy;
A reference block centered on the predetermined position of the first frame and a position where the position corresponding to the predetermined position of the second frame is moved based on the selected plurality of motion vectors are referred to as reference positions, respectively. Detecting correlation information between candidate blocks in the plurality of search ranges,
Using the detected correlation information corresponding to the plurality of candidate blocks as the motion vector at the predetermined position based on the position information of the candidate block having the highest correlation with the reference block. Program to be executed.

Images