JP4447671B2 - Image signal converting apparatus and method, and neural network coupling coefficient generating apparatus and generating method used therefor - Google Patents

Description

【００２８】
そして、動きクラス決定回路１０５では、上述したように算出された平均値ＡＶが１個または複数個のしきい値と比較されてクラス情報ＭＶが得られる。例えば、３個のしきい値ｔｈ₁，ｔｈ₂，ｔｈ₃（ｔｈ₁＜ｔｈ₂＜ｔｈ₃）が用意され、４つの動きクラスを決定する場合、ＡＶ≦ｔｈ₁のときはＭＶ＝０、ｔｈ₁＜ＡＶ≦ｔｈ₂のときはＭＶ＝１、ｔｈ₂＜ＡＶ≦ｔｈ₃のときはＭＶ＝２、ｔｈ₃＜ＡＶのときはＭＶ＝３とされる。
【００２９】
また、画像信号変換装置１００は、ＡＤＲＣ回路１０３より出力される空間クラスのクラス情報としての再量子化コードｑiと、動きクラス決定回路１０５より出力される動きクラスの情報ＭＶに基づき、予測しようとするＨＤ画素データが属するクラスを示すクラスコードＣＬを得るためのクラスコード発生回路１０６を有している。クラスコード発生回路１０６では、（３）式によって、クラスコードＣＬの演算が行われる。なお、（３）式において、Ｎａは領域切り出し回路１０２で切り出されるＳＤ画素データの個数、ｐはＡＤＲＣ回路１０３における再量子化ビット数を示している。
【００３０】
【数２】

【００３１】
また、画像信号変換装置１００は、後述ニューラルネットワーク予測回路で使用されるニューラルネットワークの結合係数が各クラス毎に記憶されている記憶手段としてのＲＯＭテーブル１０７を有している。ＲＯＭテーブル１０７にはクラスコード発生回路１０６より出力されるクラスコードＣＬが読み出しアドレス情報として供給され、このＲＯＭテーブル１０７からはクラスコードＣＬに対応した結合係数Ｗiが読み出される。
【００３２】
また、画像信号変換装置１００は、入力端子１０１に供給されるＳＤ画素データより、予測しようとする所定のＨＤ画素データに対応した領域のＳＤ画素データを切り出す領域切り出し回路１０８と、この領域切り出し回路１０８で切り出されたＳＤ画素データと、上述したＲＯＭテーブル１０７より読み出される結合係数Ｗiとから、ニューラルネットワークを使用して、予測しようとするＨＤ画素データを求めるニューラルネット予測回路１０９と、このニューラルネット予測回路１０９で求められたＨＤ画素データを導出する出力端子１１０とを有している。
【００３３】
領域切り出し回路１０８では、例えば図８に示すように、ＨＤ画素データｙを予測しようとする場合、このＨＤ画素データｙの近傍に位置する２５個のＳＤ画素データｘ₁〜ｘ₂₅が切り出される。ここで、領域切り出し回路１０８は、上記した領域切り出し回路１０２と共通であってもよい。
【００３４】
図２は、ニューラルネット予測回路１０９の要部構成例を示しており、バックプロバゲーションニューラルネットワークを使用した例である。このニューラルネット予測回路１０９は、ニューラルネットワーク部１２１と、このニューラルネットワーク部１２１の出力層のユニットより得られる結合の度合いを示す値を画素信号値に変換して出力する正規化部１２２とを有して構成されている。
【００３５】
ニューラルネットワーク部１２１は、３階層で構成され、入力層にｎ個のユニット、隠れ層に４個のユニット、出力層に１個のユニットが配されている。この場合、入力層の各ユニットには、上述した領域切り出し回路１０８で切り出されるｎ個のＳＤ画素データｘ₁〜ｘ_nがそれぞれ供給される。そして、この入力層の各ユニットは、それぞれ供給されたＳＤ画素データｘ₁〜ｘ_nを何等変更せずに、隠れ層の各ユニットに分配する。
【００３６】
また、隠れ層の各ユニットは、入力層の各ユニットと結合係数で結合されている。そして、隠れ層の各ユニットより出力される結合の度合いを示す値は出力層のユニットに供給される。さらに、出力層のユニットは、隠れ層の各ユニットと結合係数で結合されている。そして、出力層のユニットより出力される結合の度合いを示す値はニューラルネットワーク部１２１の出力値として正規化部１２２に供給される。
【００３７】
図３は、隠れ層および出力層のユニットの構成例を示している。前層のユニットの個数がｍ個であるとして、前層の各ユニットの出力値をＡ₁〜Ａ_mとし、また前層の各ユニットとの結合における結合係数をＷ₁〜Ｗ_mとするとき、ユニットの出力値Ｕは、（４）式で表される。そして、この（４）式において、ｆ( )は非線形要素としての伝達関数であり、例えば（５）式のような関数がよく使用される。この（５）式で、ａは適当な実数である。
【００３８】
【数３】

【００３９】
なお、上述したようにＲＯＭテーブル１０７よりクラスコードＣＬに対応して読み出される結合係数Ｗiが、隠れ層および出力層のユニットにおける結合係数Ｗ₁〜Ｗ_mとして使用されることになる。
【００４０】
また、ニューラルネットワーク部１２１の出力層のユニットより得られる結合の度合いを示す値は０より大きく、かつ１より小さな値となる。そのため、正規化部１２２では、例えば画素信号値が８ビットで表される場合には、結合の度合いを示す値に「２５６」をかけ算して、予測対象画素値としてのＨＤ画素データｙを得るようにされる。
【００４１】
図２に示すニューラルネット予測回路１０９は一例であって、これに限定されるものではない。例えば、ニューラルネットワーク部１２１を、隠れ層を増やすことで４層以上の構成とすることもできる。また、隠れ層および出力層のユニットの個数も任意に設定できる。
【００４２】
図１に示す画像信号変換装置１００の動作を説明する。予測しようとする所定のＨＤ画素データｙに対応して、入力端子１０１に供給されるＳＤ画素データより領域切り出し回路１０２で所定領域のＳＤ画素データｋiが切り出され、この切り出された各ＳＤ画素データｋiに対してＡＤＲＣ回路１０３でＡＤＲＣ処理が施されて空間クラス（主に空間内の波形表現のためのクラス分類）のクラス情報としての再量子化コードｑiが得られる。
【００４３】
また、上述した予測しようとするＨＤ画素データｙに対応して、入力端子１０１に供給されるＳＤ画素データより領域切り出し回路１０４で所定領域のＳＤ画素データｍi，ｎiが切り出され、この切り出された各ＳＤ画素データｍi，ｎiより動きクラス決定回路１０５で動きクラス（主に動きの程度を表すためのクラス分類）を示すクラス情報ＭＶが得られる。
【００４４】
この動きクラス情報ＭＶと上述したＡＤＲＣ回路１０３で得られる再量子化コードｑiとから、クラスコード発生回路１０６で予測しようとするＨＤ画素データｙが属するクラスを示すクラス情報としてのクラスコードＣＬが得られる。そして、このクラスコードＣＬがＲＯＭテーブル１０７に読み出しアドレス情報として供給され、このＲＯＭテーブル１０７より予測しようとするＨＤ画素データｙが属するクラスに対応したニューラルネットワークの結合係数Ｗiが読み出される。
【００４５】
また、上述した予測しようとするＨＤ画素データｙに対応して、入力端子１０１に供給されるＳＤ画素データより領域切り出し回路１０８で所定領域のＳＤ画素データｘiが切り出される。そして、ニューラルネット予測回路１０９では、その切り出されたＳＤ画素データｘiと、上述したようにＲＯＭテーブル１０７より読み出された結合係数Ｗiとから、ニューラルネットワークを使用して、予測しようとするＨＤ画素データｙが求められる。そして、このニューラルネット予測回路１０９より順次出力されるＨＤ画素データｙが出力端子１１０に導出される。
【００４６】
図１に示す画像信号変換装置１００においては、ＨＤ画素データｙをニューラルネットワークを使用して予測するものであり、非線形な予測（線形な予測を含む）が行われるため、従来の線形１次式によって予測するものと比べて、常に最適なＨＤ画素データｙの予測を行うことが可能となる。
【００４７】
ところで、画像信号変換装置１００のＲＯＭテーブル１０７には、上述したように各クラスに対応したニューラルネットワークの結合係数が記憶されている。この係数データは、予め学習によって生成されたものである。図９は、学習によって、各クラス毎にニューラルネットワークの結合係数を生成する結合係数生成装置２００の構成例を示している。
【００４８】
この結合係数生成装置２００は、教師信号としてのハイビジョンのビデオ信号を構成するＨＤ画素データが供給される入力端子２０１と、このＨＤ画素データに対して水平および垂直の間引きフィルタ処理を行って、入力信号としてのＮＴＳＣ方式のビデオ信号を構成するＳＤ画素データを得る信号処理部２０２とを有している。信号処理部２０２では、ＨＤ画素データに対して、垂直間引きフィルタによってフィールド内の垂直方向のライン数が１／２となるように間引き処理されると共に、さらに水平間引きフィルタによって水平方向の画素数が１／２となるように間引き処理される。したがって、ＳＤ画素とＨＤ画素の位置関係は、図４および図５に示すようになる。
【００４９】
また、結合係数生成装置２００は、入力端子２０１に供給されるＨＤ画素データより得られる予測対象画素値としての複数個のＨＤ画素データにそれぞれ対応して、信号処理部２０２より出力されるＳＤ画素データより所定領域のＳＤ画素データを順次切り出す領域切り出し回路２０３と、この領域切り出し回路２０３で順次切り出されたＳＤ画素データに対してＡＤＲＣ処理を適用して、主に空間内の波形を表すクラス（空間クラス）を決定してクラス情報を出力するＡＤＲＣ回路２０４とを有している。
【００５０】
領域切り出し回路２０３は、上述した画像信号変換装置１００の領域切り出し回路１０２と同様に構成される。この領域切り出し回路２０３からは、例えば図６に示すように、予測対象画素値としてのＨＤ画素データｙに対応して、このＨＤ画素データｙの近傍に位置するＳＤ画素データｋ₁〜ｋ₅が切り出される。また、ＡＤＲＣ回路２０４も、上述した画像信号変換装置１００のＡＤＲＣ回路１０３と同様に構成される。このＡＤＲＣ回路２０４からは、予測対象値としての各ＨＤ画素データにそれぞれ対応して切り出された所定領域のＳＤ画素データ毎に再量子化コードｑiが空間クラスを示すクラス情報として出力される。
【００５１】
また、結合係数生成装置２００は、上述した予測対象画素値としての各ＨＤ画素データにそれぞれ対応して、信号処理部２０２より出力されるＳＤ画素データより所定領域のＳＤ画素データを順次切り出す領域切り出し回路２０５と、この領域切り出し回路２０５で切り出されたＳＤ画素データより、主に動きの程度を表すためのクラス（動きクラス）を決定してクラス情報を出力する動きクラス決定回路２０６とを有している。
【００５２】
領域切り出し回路２０５は、上述した画像信号変換装置１００の領域切り出し回路１０４と同様に構成される。この領域切り出し回路２０５からは、例えば図７に示すように、予測対象画素値としてのＨＤ画素データｙに対応して、このＨＤ画素データｙの近傍に位置する１０個のＳＤ画素データｍ₁〜ｍ₅，ｎ₁〜ｎ₅が切り出される。また、動きクラス決定回路２０６も、上述した画像信号変換装置１００の動きクラス決定回路１０５と同様に構成される。この動きクラス決定回路２０６からは、予測対象画素値としての各ＨＤ画素データにそれぞれ対応して切り出された所定領域のＳＤ画素データ毎に動きの指標である動きクラスのクラス情報ＭＶが出力される。
【００５３】
また、結合係数生成装置２００は、ＡＤＲＣ回路２０４より出力される空間クラスのクラス情報としての再量子化コードｑiと、動きクラス決定回路２０６より出力される動きクラスのクラス情報ＭＶに基づいてクラスコードＣＬを得るためのクラスコード発生回路２０７を有している。このクラスコード発生回路２０７は、上述した画像信号変換装置１００のクラスコード発生回路１０６と同様に構成される。このクラスコード発生回路２０７からは、予測対象画素値としての各ＨＤ画素データにそれぞれ対応して、そのＨＤ画素データが属するクラスを示すクラスコードＣＬが出力される。
【００５４】
また、結合係数生成装置２００は、上述した予測対象画素値としての各ＨＤ画素データにそれぞれ対応して、信号処理部２０２より出力されるＳＤ画素データより所定領域のＳＤ画素データを順次切り出す領域切り出し回路２０８を有している。領域切り出し回路２０８は、上述した画像信号変換装置１００の領域切り出し回路１０８と同様に構成される。この領域切り出し回路２０８からは、例えば図８に示すように、予測対象画素値としてのＨＤ画素データｙに対応して、このＨＤ画素データｙの近傍に位置する２５個のＳＤ画素データｘ₁〜ｘ₂₅が切り出される。
【００５５】
また、結合係数生成装置２００は、入力端子２０１に供給されるＨＤ画素データより得られる予測対象画素値としての各ＨＤ画素データｙと、予測対象画素値としての各ＨＤ画素データｙにそれぞれ対応して領域切り出し回路２０８で順次切り出されたＳＤ画素データｘiと、予測対象画素値としての各ＨＤ画素データｙにそれぞれ対応してクラスコード発生回路２０７より出力されるクラスコードＣＬとから、ニューラルネットワークによる学習によって、各クラス毎に結合係数を得るニューラルネット学習回路２０９と、このニューラルネット学習回路２０９で得られた各クラス毎の結合係数Ｗiを記憶するためのメモリ２１０とを有している。
【００５６】
図１０は、ニューラルネット学習回路２０９の要部構成例を示しており、図２示すニューラルネット予測回路１０９と同様に、バックプロバゲーションニューラルネットワークを使用した例である。このニューラルネット学習回路２０９は、ニューラルネットワーク部２２１と、このニューラルネットワーク部２２１の出力層のユニットより出力される結合の度合いを示す値を画素信号値に変換して出力する正規化部２２２と、この正規化部２２２より出力される画素信号値ｙoと予測対象画素値としてのＨＤ画素データｙとを比較して誤差εを検出する誤差検出部２２３とを有して構成されている。ニューラルネットワーク部２２１および正規化部２２２は、それぞれは図２に示すニューラルネット予測回路１０９のニューラルネットワーク部１２１および正規化部１２２と同様に構成されている。
【００５７】
このニューラルネット学習回路２０９では、各クラス毎に、複数個の学習データを用いて結合係数が生成される。ここで、１個の学習データは、予測対象画素値としての１個のＨＤ画素データｙとそれに対応するｎ個のＳＤ画素データｘ₁〜ｘ_nとの組み合わせで構成されている。
【００５８】
ここで、あるクラスにおける結合係数の生成は、以下のように行われる。まず、ニューラルネットワーク部２２１の隠れ層および出力層のユニットにおける結合係数が初期値の状態で、あるクラスの複数個の学習データのうち１番目の学習データを構成するｎ個のＳＤ画素データｘ₁〜ｘ_nをニューラルネットワーク部２２１の入力層のユニットに供給する。この状態で、出力層のユニットより得られる結合の度合いを示す値を正規化部２２２で画素信号値ｙoに変換する。そして、誤差検出部２２３で、この画素信号値ｙoと上述の１番目の学習データを構成する１個のＨＤ画素データｙとを比較して誤差εを検出する。この誤差εが充分小さな一定範囲内にあるときは、この状態での隠れ層および出力層のユニットにおける結合係数をあるクラスの結合係数Ｗiとしてメモリ２１０に供給して記憶させ、あるクラスにおける結合係数の生成動作を終了する。
【００５９】
一方、誤差εが一定範囲内にないときは、この誤差εが一定範囲内に入る方向にニューラルネットワーク部２２１の出力層、さらには隠れ層のユニットにおける結合係数を変更する。つまり、ニューラルネットワーク部２２１の出力層での誤差を入力層に向かって伝搬させて学習を行うものである。そして、このように出力層および隠れ層のユニットにおける結合係数を変更した後に、あるクラスの複数個の学習データのうち２番目の学習データを構成するｎ個のＳＤ画素データｘ₁〜ｘ_nをニューラルネットワーク部２２１の入力層のユニットに供給する。
【００６０】
この状態で、出力層のユニットより得られる結合の度合いを示す値を正規化部２２２で画素信号値ｙoに変換する。そして、誤差検出部２２３で、この画素信号値ｙoと上述の２番目の学習データを構成する１個のＨＤ画素データｙとを比較して誤差εを検出する。この誤差εが充分小さな一定範囲内にあるときは、この状態での隠れ層および出力層のユニットにおける結合係数をあるクラスの結合係数Ｗiとしてメモリ２１０に供給して記憶させ、あるクラスにおける結合係数の生成動作を終了する。一方、誤差εが一定範囲内にないときは、この誤差εが一定範囲内に入る方向にニューラルネットワーク部２２１の出力層、さらには隠れ層のユニットにおける結合係数を変更する。
【００６１】
以下、同様にして、誤差εが一定範囲内に入るまで、順次次の学習データを使用し、ニューラルネットワーク部２２１の出力層、隠れ層のユニットにおける結合係数を変更していく。ただし、あるクラスの複数個の学習データを全て使用しても、誤差εが一定範囲内に入らないときは、最終的に出力層、隠れ層のユニットで使用された結合係数をあるクラスの結合係数Ｗiとしてメモリ２１０に供給して記憶させ、あるクラスにおける結合係数の生成動作を終了する。
【００６２】
なお、あるクラスの複数個の学習データを全て使用しても、誤差εが一定範囲内に入らないときは、出力層、隠れ層のユニットにおける結合係数の初期値や（５）式の伝達関数における実数ａを変更して、上述した生成動作を最初からやり直すようにしてもよい。
【００６３】
図９に示す結合係数生成装置２００の動作を説明する。入力端子２０１には教師信号としてのハイビジョンのビデオ信号を構成するＨＤ画素データが供給され、そしてこのＨＤ画素データに対して信号処理部２０２で水平および垂直の間引き処理等が行われて入力信号としてのＮＴＳＣ方式のビデオ信号を構成するＳＤ画素データが得られる。
【００６４】
また、入力端子２０１に供給されるＨＤ画素データより得られる予測対象画素値としての各ＨＤ画素データｙにそれぞれ対応して、信号処理部２０２より出力されるＳＤ画素データから領域切り出し回路２０３で所定領域のＳＤ画素データｋiが順次切り出され、この切り出された各ＳＤ画素データｋiに対してＡＤＲＣ回路２０４でＡＤＲＣ処理が施されて空間クラス（主に空間内の波形表現のためのクラス分類）のクラス情報としての再量子化コードｑiが得られる。
【００６５】
また、予測対象画素値としての各ＨＤ画素データｙにそれぞれ対応して、信号処理部２０２より出力されるＳＤ画素データから領域切り出し回路２０５で所定領域のＳＤ画素データｍi，ｎiが順次切り出され、この切り出された各ＳＤ画素データｍi，ｎiより動きクラス決定回路２０６で動きクラス（主に動きの程度を表すためのクラス分類）を示すクラス情報ＭＶが得られる。そして、このクラス情報ＭＶと上述したＡＤＲＣ回路２０４で得られる再量子化コードｑiとからクラスコード発生回路２０７で、予測対象画素値としての各ＨＤ画素データｙが属するクラスを示すクラス情報としてのクラスコードＣＬが得られる。
【００６６】
また、予測対象画素値としての各ＨＤ画素データｙにそれぞれ対応して、信号処理部２０２より出力されるＳＤ画素データから領域切り出し回路２０８で所定領域のＳＤ画素データｘiが順次切り出される。そして、入力端子２０１に供給されるＨＤ画素データより得られる予測対象画素値としての各ＨＤ画素データｙと、予測対象画素値としての各ＨＤ画素データｙにそれぞれ対応して領域切り出し回路２０８で順次切り出されたＳＤ画素データｘiと、予測対象画素値としての各ＨＤ画素データｙにそれぞれ対応してクラスコード発生回路２０７より出力されるクラスコードＣＬとから、ニューラルネットワークによる学習によって、各クラス毎に結合係数Ｗiが生成される。そして、この結合係数Ｗiは、クラス別にアドレス分割されたメモリ２１０に記憶される。
【００６７】
図１１は、ニューラルネット学習回路２０９における結合係数の学習フローを示している。まず、ステップＳＴ１で、入力端子２０１に供給されるＨＤ画素データより得られる予測対象画素値としての１個のＨＤ画素データｙと、それに対応して領域切り出し回路２０８で切り出されるｎ個のＳＤ画素データｘ₁〜ｘ_nとの組み合わせで、１個の学習データを生成する。次に、ステップＳＴ２で、クラスコード発生回路２０７より供給されるクラスコードＣＬによって、ステップＳＴ１で生成された学習データのクラス分類をする。
【００６８】
次に、ステップＳＴ３で、入力端子２０１に供給されるＨＤ画素データより得られる予測対象画素値としての各ＨＤ画素データｙの全てに対応した学習データの生成が終了したか否かを判定し、学習データの生成が終了していないときは、ステップＳＴ１に戻って、学習データの生成を続ける。一方、学習データの生成が終了したときは、ステップＳＴ４で、Ｎ＝１とし、ステップＳＴ５で、Ｎクラスに分類された複数個の学習データを使用して、このＮクラスの結合係数Ｗiを生成する処理をする。
【００６９】
次に、ステップＳＴ６で、全てのクラスの結合係数Ｗiの生成が終了したか否かを判定する。終了していないときは、ステップＳＴ７で、Ｎを１だけ増し、その後にステップＳＴ５に戻って、上述したようにＮクラスの結合係数Ｗiを生成する処理をする。一方、終了したときは、結合係数の学習フローを終了する。
【００７０】
また、図１２は、Ｎクラスの結合係数生成処理フローを示している。まず、ステップＳＴ１１で、Ｍ＝１とする。次に、ステップＳＴ１２で、ＮクラスのＭ番目の学習データを構成するｎ個のＳＤ画素データをニューラルネットワーク部２２１の入力層のユニットに供給し、この状態で出力層のユニットより結合の度合いを示す値を得る。
【００７１】
次に、ステップＳＴ１３で、ニューラルネットワーク部２２１の出力層のユニットとより得られる結合の度合いを示す値を正規化部２２２に供給して正規化することで画素信号値ｙoを得る。そして、ステップＳＴ１４で、ＮクラスのＭ番目の学習データを構成する１個のＨＤ画素データｙと、ステップＳＴ１３で得られた画素信号値ｙoとを誤差検出部２２３で比較して誤差εを検出する。
【００７２】
次に、ステップＳＴ１５で、ステップＳＴ１４で検出される誤差εが充分小さな一定範囲内に入っているか否かを判定する。誤差εが一定範囲内に入っているときは、ステップＳＴ１６で、その状態での隠れ層および出力層のユニットにおける結合係数をＮクラスの結合係数Ｗiとしてメモリ２１０に供給して記憶させ、その後にＮクラスにおける結合係数の生成動作を終了する。
【００７３】
一方、誤差εが一定範囲内にないときは、ステップＳＴ１７で、この誤差εが一定範囲内に入る方向にニューラルネットワーク部２２１の出力層および隠れ層のユニットにおける結合係数を変更する。そして、ステップＳＴ１８で、Ｍを１だけ増やし、その後にステップＳＴ１９で、ＭがＮクラスに分類された学習データの個数Ｍnより大きいか否かを判定する。
【００７４】
Ｍ＞Ｍnであるときは、Ｎクラスに分類された学習データの全てを使用したことになるので、ステップＳＴ１６で、その状態での隠れ層および出力層のユニットにおける結合係数をＮクラスの結合係数Ｗiとしてメモリ２１０に供給して記憶させ、その後にＮクラスにおける結合係数の生成動作を終了する。一方、Ｍ＞Ｍnでないときは、ステップＳＴ１２に戻って、上述した動作を繰り返し行う。
【００７５】
なお、図１２には図示していないが、学習データを全て使用しても誤差εが一定範囲内に入らないとき、出力層、隠れ層のユニットにおける結合係数の初期値や（５）式の伝達関数における実数ａを変更して、上述した生成動作を最初からやり直す場合には、ステップＳＴ１９でＭ＞Ｍnであるとき、結合係数の初期値や伝達関数の実数ａを変更して、ステップＳＴ１１に戻るようにすればよい。
【００７６】
なお、上述実施の形態においては、空間波形を少ないビット数でパターン化する情報圧縮手段として、ＡＤＲＣ回路１０３（図１）、２０４（図９）を設けることにしたが、これはほんの一例であり、信号波形のパターンの少ないクラスで表現できるような情報圧縮手段であれば何を設けるかは自由であり、例えばＤＰＣＭ（Differential Pulse Code Modulation）やＶＱ（Vector Quantization ）等の圧縮手段を用いてもよい。
【００７７】
また、上述実施の形態では、ＳＤ画素データよりＨＤ画素データを得るものを示したが、例えばあるＳＤ画素データより解像度を増した他のＳＤ画素データを得る場合、あるいはあるＨＤ画素データより解像度を増した他のＨＤ画素データを得る場合にも同様に適用できる。この場合には、変換前と変換後の画素数は同じとなる。
【００７８】
また、上述実施の形態においては、ＮＴＳＣ方式のビデオ信号をハイビジョンのビデオ信号に変換する例を示したが、この発明はそれに限定されるものでなく、第１の画像信号をこの第１の画像信号と同じあるいはそれより多い画素数の第２の画像信号に変換する場合に同様に適用できることは勿論である。
【００７９】
【発明の効果】
この発明によれば、第１の画像信号をこの第１の画像信号と同じあるいはそれより多い画素数の第２の画像信号に変換する際、ニューラルネットワークを使用して第２の画像信号を構成する画素信号を予測するものであり、非線形な予測（線形な予測を含む）が行われるため、従来の線形１次式によって予測するものと比べて、常に最適な画素信号の予測を行うことができる。
【図面の簡単な説明】
【図１】実施の形態としての画像信号変換装置の構成を示すブロック図である。
【図２】画像信号変換装置のニューラルネット予測回路の要部構成例を示す図である。
【図３】ニューラルネットワークを構成する隠れ層、出力層のユニットの構成例を示す図である。
【図４】ＳＤ画素とＨＤ画素の位置関係を説明するための略線図である。
【図５】ＳＤ画素とＨＤ画素の位置関係を説明するための略線図である。
【図６】空間クラス分類に使用するＳＤ画素データを説明するための略線図である。
【図７】動きクラス分類に使用するＳＤ画素データを説明するための略線図である。
【図８】ニューラルネットワークを使用したＨＤ画素データの予測に用いるＳＤ画素データを説明するための略線図である。
【図９】ニューラルネット予測回路で使用される結合係数を学習によって生成する結合係数生成装置の構成例を示すブロック図である。
【図１０】結合係数生成装置のニューラルネット学習回路の要部構成例を示す図である。
【図１１】結合係数の学習フローを示すフローチャートである。
【図１２】学習フローにおけるＮクラスの結合係数生成処理を説明するためのフローチャートである。
【符号の説明】
１００・・・画像信号変換装置、１０１，２０１・・・入力端子、１０２，１０４，１０８，２０３，２０５，２０８・・・領域切り出し回路、１０３，２０４・・・ＡＤＲＣ回路、１０５，２０６・・・動きクラス決定回路、１０６，２０７・・・クラスコード発生回路、１０７・・・ＲＯＭテーブル、１０９・・・ニューラルネット予測回路、１１０・・・出力端子、１２１，２２１・・・ニューラルネットワーク部、１２２，２２２・・・正規化部、２０９・・・ニューラルネット学習回路、２１０・・・メモリ、２２３・・・誤差検出部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image signal conversion apparatus and conversion method for converting, for example, an NTSC video signal into a high-definition video signal, and a generation apparatus and generation method for a coupling coefficient of a neural network used therefor. More specifically, the present invention relates to an image signal conversion apparatus that can perform non-linear prediction by using a neural network when predicting a pixel signal, and can always predict an optimal pixel signal. .
[0002]
[Prior art]
In recent years, the development of television receivers capable of obtaining higher-resolution images has been desired due to the increase in audio / visual orientation, and so-called high-vision has been developed in response to this demand. The number of high-definition scanning lines is 1125, which is more than twice that of 525 scanning lines in the NTSC system. The aspect ratio of the high-definition television is 9:16, whereas the aspect ratio of the NTSC system is 3: 4. For this reason, high-definition images can be displayed with a higher resolution and presence than in the NTSC system.
[0003]
Although HDTV has such excellent characteristics, even if an NTSC video signal is supplied as it is, image display using the HDTV system cannot be performed. This is because the standards differ between the NTSC system and the high vision as described above.
[0004]
Therefore, in order to display an image corresponding to the NTSC video signal in the high-definition system, the present applicant has previously proposed a conversion device for converting an NTSC video signal into a high-definition video signal (Japanese Patent Application). No. 6-205934). In this conversion device, the signal of each pixel constituting the high-definition video signal is predicted from the signal of the pixel in a predetermined area of the NTSC system and coefficient data (prediction coefficient value) using a linear linear expression. ing.
[0005]
[Problems to be solved by the invention]
In the above-described conversion device, the signal of each pixel constituting the high-definition video signal is predicted by a linear linear expression, and an optimal pixel signal may not be obtained.
[0006]
Therefore, an object of the present invention is to provide an image signal conversion device and the like that can always predict an optimal pixel signal.
[0007]
[Means for Solving the Problems]
The image signal conversion device according to the present invention is the image signal conversion device configured to convert the first image signal into a second image signal having the same or more number of pixels as the first image signal. A first pixel cutout unit that cuts out a signal of a pixel in a first region located in the vicinity of a predetermined pixel that constitutes the second image signal from the first image signal; and the first pixel cutout unit The level distribution pattern of the signal of the pixel in the first area is detected, and the class information is determined by determining the class to which the signal of the predetermined pixel constituting the second image signal to be predicted belongs based on this pattern. Classifying means for outputting, and at least an input layer and an output layer are provided. The units of each layer are combined with each other by a predetermined coupling coefficient, and the input values to the units of the combined layer are combined. Sum of products of output values of multiple units in the previous layer and the above coupling coefficient Including non-linear operations A neural network prediction means for predicting a signal of a predetermined pixel constituting the second image signal by a neural network, and the coupling coefficient of the neural network prediction means corresponding to each class indicated by the class information; Coupling coefficient storage means to be placed, coupling coefficient reading means for reading a coupling coefficient from the coupling coefficient storage means corresponding to the class information output from the class determining means, and the second image from the first image signal Second pixel clipping means for cutting out a signal of a pixel in a second region located in the vicinity of a predetermined pixel constituting the signal, and the neural network prediction means is a combination read out from the coupling coefficient storage means The second region cut out by the second pixel cut-out means is set as a coupling coefficient of the neural network. The pixel signal value is expressed as an input value to the input layer unit of the neural network, and the output value of the output unit of the neural network obtained by performing the operation of the neural network is expressed as a pixel signal value. Is converted into a pixel signal value corresponding to the number of bits to be output, and is output as a predicted value of a predetermined pixel signal constituting the second image signal.
[0008]
Here, the first and second pixel cutout means may be common, and in this case, the first and second regions are the same. Further, the neural network prediction means is configured using, for example, a back propagation network. Furthermore, the neural network predicting means may have a normalizing means for converting a value indicating the degree of coupling obtained from the unit of the output layer of the neural network into a pixel signal value at its output unit.
[0009]
The image signal conversion method according to the present invention is an image signal conversion method in which the first image signal is converted into a second image signal having the same or greater number of pixels as the first image signal. A first step of cutting out a signal of a pixel in a first region located in the vicinity of a predetermined pixel constituting the second image signal from the first image signal, and the above step cut out in the first step The level distribution pattern of the signal of the pixel in the first region is detected, and the class to which the signal of the predetermined pixel constituting the second image signal to be predicted belongs is determined based on this pattern, and class information is output. A second step, comprising at least an input layer and an output layer, wherein the units of each layer are combined with each other by a predetermined coupling coefficient, and the input value to the unit of the combined layer is a plurality of layers before the combination; Product sum of the output value of the unit and the above coupling coefficient Including non-linear operations A third step of predicting a signal of a predetermined pixel constituting the second image signal by a neural network; and the coupling coefficient corresponding to each class indicated by the class information output in the second step. A fourth step of reading out by the storage means, and a fifth step of cutting out the signal of the pixel in the second region located in the vicinity of the predetermined pixel constituting the second image signal from the first image signal. And in the third step, the coupling coefficient read out from the storage means is used as the coupling coefficient of the neural network, and the signal of the pixel in the second region cut out by the second pixel cutting-out means is obtained. As the input value to the unit of the input layer of the neural network, the neural network obtained by performing the operation of the neural network The output value of the unit of the output layer of the network is converted into a pixel signal value corresponding to the number of bits for expressing the pixel signal value, and output as a predicted value of a predetermined pixel signal constituting the second image signal Is.
[0010]
A signal of a pixel in the first region is cut out from the first image signal corresponding to a predetermined pixel constituting the second image signal to be predicted, and the above-described second image signal is based on the level detection pattern. The class to which the signal of the predetermined pixel that constitutes belongs is determined and class information is output. Corresponding to this class information, the coupling coefficient is read from the coupling coefficient storage means. Further, the signal of the pixel in the second region is cut out from the first image signal corresponding to the predetermined pixel constituting the second image signal described above. Then, a signal of a predetermined pixel constituting the above-described second image signal is obtained from the signal of the pixel in the second area and the coupling coefficient read from the coupling coefficient storage means by using a neural network. . Thus, by predicting a pixel signal using a neural network, non-linear prediction is performed, and it is possible to always predict an optimal pixel signal as compared with that predicted by a linear linear expression. It becomes possible.
[0011]
The coupling coefficient generation device according to the present invention generates a coupling coefficient of a neural network used when converting the first image signal into a second image signal having a larger number of pixels than the first image signal. The signal processing means for obtaining the input signal corresponding to the first image signal by reducing the number of pixels of the teacher signal corresponding to the second image signal, A first pixel cutout unit that sequentially cuts out signals of pixels in a first region located in the vicinity of a plurality of pixels constituting the teacher signal; and a first pixel cutout unit that is cut out sequentially by the first pixel cutout unit. A class of detecting a level distribution pattern of pixel signals, determining a class to which each of the plurality of pixel signals constituting the teacher signal belongs based on the pattern, and outputting class information. A second pixel cutout means for sequentially cutting out signals of pixels in a second region located in the vicinity of a plurality of pixels constituting the teacher signal from the input signal, and at least an input layer and an output layer Each layer unit is combined with a predetermined coupling coefficient between the layers, and the input value to the combined layer unit is the product sum of the output values of the plurality of layer units before combining and the above coupling coefficient Including non-linear operations The pixel values of the second region sequentially cut out by the second pixel cut-out means are inputted as input values to the unit of the input layer of the neural network, and the plurality of pixel values constituting the teacher signal are inputted as the input values. The combination of the neural network for each class to which the signals of the plurality of pixels constituting the teacher signal output from the class determination means belong by learning to be a true value based on the output value of the unit of the output layer And a neural network learning means for obtaining a coefficient.
[0012]
Here, the first and second pixel cutout means may be common, and in this case, the first and second regions are the same. Further, for example, the neural network learning means is configured using a back propagation network.
[0013]
The neural network learning means is composed of units of an input layer, a hidden layer, and an output layer, for example, and the signals of the pixels in the second area sequentially cut out by the second pixel cut-out means are supplied to the input layer. Neural network unit and output layer unit of the above neural network unit The output value from the output signal according to the number of bits used to represent the pixel signal value Normalization means for converting to pixel signal values; pixel signal values output from the normalization means; - Error detection means for detecting an error by comparing a signal of a predetermined pixel constituting the teacher signal corresponding to a signal of a pixel in the second region supplied to the input layer of the network unit, and the class information Learning data consisting of a combination of a signal of the pixel in the second area sequentially cut out by the second pixel cut-out means and a signal of a predetermined pixel constituting the teacher signal corresponding to each class indicated by On the other hand, coupling coefficient changing means for changing the coupling coefficient in the output layer and hidden layer units of the neural network unit in a direction in which the error detected by the error detection means falls within a sufficiently small fixed range, Coupling coefficient determination using the coupling coefficient in the output layer and hidden layer units of the above neural network unit when it falls within a certain range as the learning result And a means.
[0014]
Further, the coupling coefficient generation method according to the present invention generates a coupling coefficient of a neural network used when converting the first image signal into a second image signal having a larger number of pixels than the first image signal. In the method, a first step of obtaining an input signal corresponding to the first image signal by reducing the number of pixels of the teacher signal corresponding to the second image signal, and from the input signal, A second step of sequentially cutting out signals of pixels in a first region located in the vicinity of a plurality of pixels constituting the teacher signal, and a signal of pixels in the first region sequentially cut out in the second step A third step of detecting a level distribution pattern, determining a class to which the signals of the plurality of pixels constituting the teacher signal belong based on the pattern, and outputting class information; A fourth step of sequentially cutting out signals of pixels in the second region located in the vicinity of the plurality of pixels constituting the teacher signal from the input signal; and at least an input layer and an output layer, each layer unit comprising: Combined with a predetermined coupling coefficient between layers, the input value to the unit of the layer after the combination is the product sum of the output value of the multiple units of the layer before the combination and the above coupling coefficient Including non-linear operations The pixel values of the second region sequentially extracted in the fourth step are input as input values to the input layer unit of the neural network, and the plurality of pixel values constituting the teacher signal are input to the output layer. The above-mentioned coupling coefficient of the neural network for each class to which the signals of the plurality of pixels constituting the teacher signal output in the third step belong by learning to be a true value based on the output value of the unit of And a fifth step of obtaining
[0015]
A teacher signal corresponding to the second image signal, for example, a high-definition video signal is processed to obtain an input signal corresponding to the first image signal, for example, the NTSC video signal. Corresponding to the signals of the plurality of pixels constituting the teacher signal, the signals of the pixels in the first region are sequentially cut out from the input signal, and the teacher signal is based on the level distribution pattern of the signals of the pixels in the first region. The class to which the signals of the plurality of pixels constituting each belong is determined and class information is output.
[0016]
Further, the signals of the pixels in the second region are sequentially cut out from the input signal corresponding to the signals of the plurality of pixels constituting the teacher signal. Then, learning by a neural network is performed from the signals of the pixels in the second region, the class information indicating the classes to which the signals of the plurality of pixels constituting the teacher signal respectively belong, and the signals of the plurality of pixels constituting the teacher signal. Thus, a coupling coefficient is obtained for each class.
[0017]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows a configuration of an image signal conversion apparatus 100 as an embodiment. The image signal conversion apparatus 100 obtains pixel data (hereinafter referred to as “HD pixel data”) constituting a high-definition video signal from pixel data (hereinafter referred to as “SD pixel data”) constituting an NTSC video signal. Is for.
[0018]
The image signal converter 100 includes an input terminal 101 to which SD pixel data is supplied, and SD pixel data in an area corresponding to predetermined HD pixel data to be predicted based on the SD pixel data supplied to the input terminal 101. A region cutout circuit 102 serving as a pixel cutout means for cutting out a pixel, and a class that mainly represents a waveform in space by applying ADRC (Adaptive Dynamic Range Coding) processing to the SD pixel data cut out by the region cutout circuit 102 An ADRC circuit 103 that determines (space class) and outputs class information.
[0019]
4 and 5 show the positional relationship between the SD pixel and the HD pixel. For example, as shown in FIG. 6, in the region cutout circuit 102, when the HD pixel data y is to be predicted, the SD pixel data k positioned in the vicinity of the HD pixel data y is used. ₁ ~ K _Five Is cut out.
[0020]
In the ADRC circuit 103, for the purpose of patterning the level distribution of the SD pixel data cut out by the area cutout circuit 102, an operation is performed to compress each SD pixel data from, for example, 8-bit data to 2-bit data. The ADRC circuit 103 outputs compressed data (requantized code) qi corresponding to each SD pixel data as class information of the space class.
[0021]
ADRC is originally an adaptive requantization method developed for high-performance coding for VTR (Video Tape Recorder), but it can express local patterns of signal level efficiently with a short word length. In the embodiment, it is used for patterning the level distribution of the SD pixel data cut out by the area cutout circuit 102.
[0022]
In the ADRC circuit 103, assuming that the maximum value of SD pixel data in the region is MAX, the minimum value is MIN, the dynamic range in the region is DR (= MAX−MIN + 1), and the number of requantization bits is p, For each SD pixel data ki, a requantization code qi is obtained by the calculation of the equation (1). However, in the expression (1), [] means a truncation process. When Na pieces of SD pixel data are cut out by the area cutout circuit 102, i = 1 to Na.
qi = [(ki-MIN + 0.5) .2 ^p / DR] (1)
[0023]
Further, the image signal conversion apparatus 100 includes an area extraction circuit 104 that extracts SD pixel data of an area corresponding to predetermined HD pixel data to be predicted from the SD pixel data supplied to the input terminal 101, and the area extraction circuit. A motion class determining circuit 105 that determines a class (motion class) mainly representing the degree of motion from the SD pixel data cut out at 104 and outputs class information.
[0024]
For example, as shown in FIG. 7, in the region cutout circuit 104, when the HD pixel data y is to be predicted, ten SD pixel data m positioned in the vicinity of the HD pixel data y are displayed. ₁ ~ M _Five , N ₁ ~ N _Five Is cut out.
[0025]
In the motion class determination circuit 105, an inter-frame difference is calculated from the SD pixel data mi, ni extracted by the region extraction circuit 104, and a threshold process is performed on the average value of the absolute value of the difference to perform motion. Class information MV of a motion class that is an index of
[0026]
That is, in the motion class determination circuit 105, the average value AV of the absolute value of the difference is calculated by the equation (2). In the area cutout circuit 104, for example, as described above, ten SD pixel data m ₁ ~ M _Five , N ₁ ~ N _Five Is cut out, Nb in the equation (2) is 5.
[0027]
[Expression 1]

[0028]
In the motion class determination circuit 105, the average value AV calculated as described above is compared with one or a plurality of threshold values to obtain class information MV. For example, three thresholds th ₁ , Th ₂ , Th _Three (Th ₁ <Th ₂ <Th _Three ) And four motion classes are determined, AV ≦ th ₁ When MV = 0, th ₁ <AV ≦ th ₂ When MV = 1, th ₂ <AV ≦ th _Three When MV = 2, th _Three When AV, MV = 3.
[0029]
Also, the image signal conversion apparatus 100 tries to perform prediction based on the requantization code q i as the class information of the spatial class output from the ADRC circuit 103 and the motion class information MV output from the motion class determination circuit 105. A class code generation circuit 106 for obtaining a class code CL indicating a class to which the HD pixel data belongs. In the class code generation circuit 106, the calculation of the class code CL is performed by the equation (3). In Equation (3), Na represents the number of SD pixel data cut out by the region cutout circuit 102, and p represents the number of requantization bits in the ADRC circuit 103.
[0030]
[Expression 2]

[0031]
Further, the image signal conversion apparatus 100 includes a ROM table 107 as a storage unit in which a coupling coefficient of a neural network used in a neural network prediction circuit described later is stored for each class. The class code CL output from the class code generation circuit 106 is supplied to the ROM table 107 as read address information, and the coupling coefficient Wi corresponding to the class code CL is read from the ROM table 107.
[0032]
Further, the image signal conversion apparatus 100 includes an area extraction circuit 108 that extracts SD pixel data of an area corresponding to predetermined HD pixel data to be predicted from the SD pixel data supplied to the input terminal 101, and the area extraction circuit. A neural network prediction circuit 109 for obtaining HD pixel data to be predicted using a neural network from the SD pixel data cut out at 108 and the coupling coefficient Wi read from the ROM table 107, and the neural network And an output terminal 110 for deriving HD pixel data obtained by the prediction circuit 109.
[0033]
For example, as shown in FIG. 8, in the region cutout circuit 108, when HD pixel data y is to be predicted, 25 SD pixel data x located in the vicinity of the HD pixel data y ₁ ~ X _{twenty five} Is cut out. Here, the region cutout circuit 108 may be common to the region cutout circuit 102 described above.
[0034]
FIG. 2 shows a configuration example of a main part of the neural network prediction circuit 109, which is an example using a back-propagation neural network. The neural network prediction circuit 109 includes a neural network unit 121 and a normalization unit 122 that converts a value indicating the degree of coupling obtained from the unit of the output layer of the neural network unit 121 into a pixel signal value and outputs the pixel signal value. Configured.
[0035]
The neural network unit 121 is configured in three layers, with n units in the input layer, four units in the hidden layer, and one unit in the output layer. In this case, each unit of the input layer includes n pieces of SD pixel data x cut out by the region cutout circuit 108 described above. ₁ ~ X _n Are supplied respectively. Each unit of the input layer is supplied with each supplied SD pixel data x ₁ ~ X _n Is distributed to each unit in the hidden layer without any change.
[0036]
Each unit in the hidden layer is coupled to each unit in the input layer with a coupling coefficient. A value indicating the degree of coupling output from each unit in the hidden layer is supplied to the unit in the output layer. Furthermore, the unit of the output layer is coupled to each unit of the hidden layer with a coupling coefficient. A value indicating the degree of coupling output from the output layer unit is supplied to the normalization unit 122 as an output value of the neural network unit 121.
[0037]
FIG. 3 shows a configuration example of the units of the hidden layer and the output layer. Assuming that the number of units in the previous layer is m, the output value of each unit in the previous layer is A ₁ ~ A _m And the coupling coefficient for coupling with each unit in the previous layer is W ₁ ~ W _m , The output value U of the unit is expressed by equation (4). In the equation (4), f () is a transfer function as a nonlinear element, and for example, a function like the equation (5) is often used. In this equation (5), a is an appropriate real number.
[0038]
[Equation 3]

[0039]
As described above, the coupling coefficient Wi read from the ROM table 107 corresponding to the class code CL is the coupling coefficient W in the unit of the hidden layer and the output layer. ₁ ~ W _m Will be used as.
[0040]
The value indicating the degree of coupling obtained from the unit of the output layer of the neural network unit 121 is greater than 0 and smaller than 1. Therefore, in the normalization unit 122, for example, when the pixel signal value is represented by 8 bits, the value indicating the degree of combination is multiplied by “256” to obtain the HD pixel data y as the prediction target pixel value. To be done.
[0041]
The neural network prediction circuit 109 shown in FIG. 2 is an example, and the present invention is not limited to this. For example, the neural network unit 121 can be configured to have four or more layers by increasing the number of hidden layers. The number of hidden layer and output layer units can also be set arbitrarily.
[0042]
The operation of the image signal conversion apparatus 100 shown in FIG. 1 will be described. Corresponding to the predetermined HD pixel data y to be predicted, the SD pixel data ki of the predetermined area is cut out by the area cutout circuit 102 from the SD pixel data supplied to the input terminal 101, and each of the cut out SD pixel data The ADRC circuit 103 applies ADRC processing to k i to obtain a requantized code q i as class information of a space class (mainly class classification for waveform expression in space).
[0043]
Corresponding to the HD pixel data y to be predicted, the SD pixel data mi, ni in a predetermined area is extracted from the SD pixel data supplied to the input terminal 101 by the area extraction circuit 104. The class information MV indicating the motion class (mainly class classification for representing the degree of motion) is obtained by the motion class determination circuit 105 from each SD pixel data mi, ni.
[0044]
From this motion class information MV and the requantized code qi obtained by the above-mentioned ADRC circuit 103, a class code CL as class information indicating a class to which the HD pixel data y to be predicted by the class code generation circuit 106 belongs is obtained. It is done. The class code CL is supplied to the ROM table 107 as read address information, and the coupling coefficient Wi of the neural network corresponding to the class to which the HD pixel data y to be predicted belongs is read from the ROM table 107.
[0045]
Corresponding to the HD pixel data y to be predicted, the SD pixel data xi in a predetermined area is cut out by the area cutout circuit 108 from the SD pixel data supplied to the input terminal 101. Then, the neural network prediction circuit 109 uses the neural network to predict the HD pixel to be predicted from the extracted SD pixel data xi and the coupling coefficient Wi read from the ROM table 107 as described above. Data y is determined. Then, the HD pixel data y sequentially output from the neural network prediction circuit 109 is derived to the output terminal 110.
[0046]
In the image signal conversion apparatus 100 shown in FIG. 1, HD pixel data y is predicted using a neural network, and nonlinear prediction (including linear prediction) is performed. Therefore, it is possible to always predict the optimal HD pixel data y as compared with the case of predicting by.
[0047]
By the way, the ROM table 107 of the image signal conversion apparatus 100 stores the coupling coefficient of the neural network corresponding to each class as described above. This coefficient data is generated in advance by learning. FIG. 9 shows a configuration example of a coupling coefficient generation apparatus 200 that generates a coupling coefficient of a neural network for each class by learning.
[0048]
This coupling coefficient generation apparatus 200 performs an input terminal 201 to which HD pixel data constituting a high-definition video signal as a teacher signal is supplied, and performs horizontal and vertical thinning filter processing on the HD pixel data to obtain an input. And a signal processing unit 202 for obtaining SD pixel data constituting an NTSC video signal as a signal. In the signal processing unit 202, the HD pixel data is thinned by the vertical thinning filter so that the number of lines in the vertical direction in the field is halved, and the horizontal thinning filter further reduces the number of pixels in the horizontal direction. Thinning-out processing is performed so as to be ½. Therefore, the positional relationship between the SD pixel and the HD pixel is as shown in FIGS.
[0049]
Further, the coupling coefficient generation device 200 corresponds to each of a plurality of HD pixel data as prediction target pixel values obtained from the HD pixel data supplied to the input terminal 201, and outputs SD pixels output from the signal processing unit 202. An area extraction circuit 203 that sequentially extracts SD pixel data of a predetermined area from data, and a class (mainly representing a waveform in space by applying ADRC processing to the SD pixel data sequentially extracted by the area extraction circuit 203 And an ADRC circuit 204 that determines the (space class) and outputs class information.
[0050]
The region cutout circuit 203 is configured in the same manner as the region cutout circuit 102 of the image signal conversion apparatus 100 described above. For example, as shown in FIG. 6, the region cutout circuit 203 corresponds to the HD pixel data y as the prediction target pixel value, and the SD pixel data k located in the vicinity of the HD pixel data y. ₁ ~ K _Five Is cut out. Further, the ADRC circuit 204 is configured in the same manner as the ADRC circuit 103 of the image signal conversion apparatus 100 described above. From this ADRC circuit 204, a re-quantization code qi is output as class information indicating a spatial class for each SD pixel data of a predetermined area cut out corresponding to each HD pixel data as a prediction target value.
[0051]
In addition, the coupling coefficient generation device 200 performs region segmentation in which SD pixel data in a predetermined region is sequentially segmented from the SD pixel data output from the signal processing unit 202, corresponding to each HD pixel data serving as the above-described prediction target pixel value. A circuit 205, and a motion class determination circuit 206 that determines a class (motion class) mainly representing the degree of motion from the SD pixel data cut out by the region cutout circuit 205 and outputs class information. ing.
[0052]
The area cutout circuit 205 is configured in the same manner as the area cutout circuit 104 of the image signal conversion apparatus 100 described above. For example, as shown in FIG. 7, the area cutout circuit 205 corresponds to the HD pixel data y as the prediction target pixel value, and the 10 SD pixel data m positioned in the vicinity of the HD pixel data y. ₁ ~ M _Five , N ₁ ~ N _Five Is cut out. The motion class determination circuit 206 is also configured in the same manner as the motion class determination circuit 105 of the image signal conversion apparatus 100 described above. The motion class determination circuit 206 outputs motion class class information MV that is a motion index for each SD pixel data of a predetermined area cut out corresponding to each HD pixel data as a prediction target pixel value. .
[0053]
In addition, the coupling coefficient generation apparatus 200 classifies the class code based on the requantization code q i as the class information of the space class output from the ADRC circuit 204 and the class information MV of the motion class output from the motion class determination circuit 206. A class code generation circuit 207 for obtaining CL is provided. The class code generation circuit 207 is configured in the same manner as the class code generation circuit 106 of the image signal conversion apparatus 100 described above. The class code generation circuit 207 outputs a class code CL indicating the class to which the HD pixel data belongs, corresponding to each HD pixel data as the prediction target pixel value.
[0054]
In addition, the coupling coefficient generation device 200 performs region segmentation in which SD pixel data in a predetermined region is sequentially segmented from the SD pixel data output from the signal processing unit 202, corresponding to each HD pixel data serving as the above-described prediction target pixel value. A circuit 208 is included. The area cutout circuit 208 is configured in the same manner as the area cutout circuit 108 of the image signal conversion apparatus 100 described above. For example, as shown in FIG. 8, the region cutout circuit 208 corresponds to the HD pixel data y as the prediction target pixel value and corresponds to 25 SD pixel data x located in the vicinity of the HD pixel data y. ₁ ~ X _{twenty five} Is cut out.
[0055]
Further, the coupling coefficient generation device 200 corresponds to each HD pixel data y as a prediction target pixel value obtained from the HD pixel data supplied to the input terminal 201 and each HD pixel data y as a prediction target pixel value. From the SD pixel data x i sequentially cut out by the region cut-out circuit 208 and the class code CL output from the class code generation circuit 207 corresponding to each HD pixel data y as the prediction target pixel value, a neural network is used. A neural network learning circuit 209 that obtains a coupling coefficient for each class by learning and a memory 210 for storing the coupling coefficient Wi for each class obtained by the neural network learning circuit 209 are provided.
[0056]
FIG. 10 shows a configuration example of a main part of the neural network learning circuit 209, which is an example using a back-propagation neural network, like the neural network prediction circuit 109 shown in FIG. The neural network learning circuit 209 includes a neural network unit 221, a normalization unit 222 that converts a value indicating a degree of coupling output from a unit of an output layer of the neural network unit 221 into a pixel signal value, and outputs the pixel signal value; The pixel signal value yo output from the normalization unit 222 and the HD pixel data y as the prediction target pixel value are compared to detect an error ε. The neural network unit 221 and the normalizing unit 222 are configured in the same manner as the neural network unit 121 and the normalizing unit 122 of the neural network prediction circuit 109 shown in FIG.
[0057]
In this neural network learning circuit 209, a coupling coefficient is generated for each class using a plurality of learning data. Here, one learning data includes one HD pixel data y as a prediction target pixel value and n SD pixel data x corresponding thereto. ₁ ~ X _n It is composed of a combination.
[0058]
Here, the generation of the coupling coefficient in a certain class is performed as follows. First, n SD pixel data x constituting the first learning data among a plurality of learning data of a certain class in the state where the coupling coefficient in the unit of the hidden layer and the output layer of the neural network unit 221 is an initial value. ₁ ~ X _n Is supplied to the unit of the input layer of the neural network unit 221. In this state, the normalization unit 222 converts the value indicating the degree of coupling obtained from the output layer unit into the pixel signal value yo. Then, the error detection unit 223 compares this pixel signal value yo with one HD pixel data y constituting the first learning data described above, and detects an error ε. When the error ε is within a sufficiently small fixed range, the coupling coefficient in the unit of the hidden layer and the output layer in this state is supplied to and stored in the memory 210 as the coupling coefficient Wi of a certain class. The generation operation of is terminated.
[0059]
On the other hand, when the error ε is not within a certain range, the coupling coefficient in the output layer of the neural network unit 221 and further the unit of the hidden layer is changed in a direction in which the error ε falls within the certain range. That is, learning is performed by propagating an error in the output layer of the neural network unit 221 toward the input layer. Then, after changing the coupling coefficient in the units of the output layer and the hidden layer in this way, the n pieces of SD pixel data x constituting the second learning data among the plurality of learning data of a certain class ₁ ~ X _n Is supplied to the unit of the input layer of the neural network unit 221.
[0060]
In this state, the normalization unit 222 converts the value indicating the degree of coupling obtained from the output layer unit into the pixel signal value yo. Then, the error detection unit 223 compares this pixel signal value yo with one HD pixel data y constituting the second learning data described above, and detects an error ε. When the error ε is within a sufficiently small fixed range, the coupling coefficient in the unit of the hidden layer and the output layer in this state is supplied to and stored in the memory 210 as the coupling coefficient Wi of a certain class. The generation operation of is terminated. On the other hand, when the error ε is not within a certain range, the coupling coefficient in the output layer of the neural network unit 221 and further the unit of the hidden layer is changed in a direction in which the error ε falls within the certain range.
[0061]
In the same manner, the next learning data is used sequentially until the error ε falls within a certain range, and the coupling coefficients in the output layer and hidden layer units of the neural network unit 221 are changed. However, if the error ε does not fall within a certain range even if all the learning data of a certain class is used, the coupling coefficient used in the unit of the output layer and the hidden layer is finally combined. The coefficient Wi is supplied to and stored in the memory 210, and the generation operation of the coupling coefficient in a certain class is finished.
[0062]
If the error ε does not fall within a certain range even if all the learning data of a certain class is used, the initial value of the coupling coefficient in the unit of the output layer and the hidden layer, and the transfer function of equation (5) The real number a may be changed, and the above-described generation operation may be performed again from the beginning.
[0063]
The operation of the coupling coefficient generation device 200 shown in FIG. 9 will be described. HD pixel data constituting a high-definition video signal as a teacher signal is supplied to the input terminal 201, and the HD pixel data is subjected to horizontal and vertical thinning processing and the like by the signal processing unit 202 as an input signal. SD pixel data constituting the NTSC video signal is obtained.
[0064]
In addition, in accordance with each HD pixel data y as a prediction target pixel value obtained from the HD pixel data supplied to the input terminal 201, the region extraction circuit 203 performs predetermined processing from the SD pixel data output from the signal processing unit 202. The SD pixel data ki of the area is sequentially cut out, and the ADRC circuit 204 performs ADRC processing on the cut out SD pixel data ki, and the space class (mainly class classification for waveform representation in the space) is obtained. A requantization code q i as class information is obtained.
[0065]
Corresponding to each HD pixel data y as a prediction target pixel value, SD pixel data mi, ni of a predetermined region are sequentially cut out from the SD pixel data output from the signal processing unit 202 by the region cutout circuit 205, The class information MV indicating the motion class (mainly class classification for representing the degree of motion) is obtained by the motion class determination circuit 206 from the extracted SD pixel data mi, ni. Then, a class code as a class information indicating a class to which each HD pixel data y as a prediction target pixel value belongs in the class code generation circuit 207 from the class information MV and the requantization code qi obtained by the ADRC circuit 204 described above. A code CL is obtained.
[0066]
Corresponding to each HD pixel data y as a prediction target pixel value, SD pixel data x i of a predetermined region is sequentially cut out from the SD pixel data output from the signal processing unit 202 by the region cut-out circuit 208. Then, the region extraction circuit 208 sequentially corresponds to each HD pixel data y as a prediction target pixel value obtained from the HD pixel data supplied to the input terminal 201 and each HD pixel data y as a prediction target pixel value. From the extracted SD pixel data xi and the class code CL output from the class code generation circuit 207 corresponding to each HD pixel data y as a prediction target pixel value, learning is performed for each class by neural network learning. A coupling coefficient Wi is generated. The coupling coefficient Wi is stored in the memory 210 that is divided into addresses by class.
[0067]
FIG. 11 shows a learning flow of coupling coefficients in the neural network learning circuit 209. First, in step ST1, one HD pixel data y as a prediction target pixel value obtained from the HD pixel data supplied to the input terminal 201, and n SD pixels cut out by the region cut-out circuit 208 corresponding thereto. Data x ₁ ~ X _n One learning data is generated in combination. Next, in step ST2, the learning data generated in step ST1 is classified by the class code CL supplied from the class code generation circuit 207.
[0068]
Next, in step ST3, it is determined whether or not the generation of learning data corresponding to all of the HD pixel data y as prediction target pixel values obtained from the HD pixel data supplied to the input terminal 201 is completed, If the generation of learning data has not ended, the process returns to step ST1 and continues to generate learning data. On the other hand, when the generation of learning data is completed, N = 1 is set in step ST4, and a plurality of learning data classified into N classes is used in step ST5 to generate this N class coupling coefficient Wi. To process.
[0069]
Next, in step ST6, it is determined whether or not the generation of the coupling coefficient Wi for all classes has been completed. If not completed, N is incremented by 1 in step ST7, and then the process returns to step ST5 to generate the N-class coupling coefficient Wi as described above. On the other hand, when finished, the learning flow of the coupling coefficient is finished.
[0070]
FIG. 12 shows an N-class coupling coefficient generation processing flow. First, in step ST11, M = 1 is set. Next, in step ST12, n SD pixel data constituting the M-th learning data of the N class are supplied to the input layer unit of the neural network unit 221, and in this state, the degree of coupling is set by the output layer unit. Get the value shown.
[0071]
Next, in step ST13, a value indicating the degree of coupling obtained from the unit of the output layer of the neural network unit 221 is supplied to the normalizing unit 222 and normalized to obtain a pixel signal value yo. In step ST14, the error detection unit 223 compares the single HD pixel data y constituting the M-th learning data of the N class with the pixel signal value yo obtained in step ST13 to detect the error ε. To do.
[0072]
Next, in step ST15, it is determined whether or not the error ε detected in step ST14 is within a sufficiently small fixed range. When the error ε is within a certain range, in step ST16, the coupling coefficient in the unit of the hidden layer and the output layer in that state is supplied to the memory 210 as the N-class coupling coefficient Wi and stored. The generation operation of the coupling coefficient in the N class is finished.
[0073]
On the other hand, when the error ε is not within the certain range, the coupling coefficient in the unit of the output layer and the hidden layer of the neural network unit 221 is changed in a direction in which the error ε falls within the certain range in step ST17. In step ST18, M is increased by 1, and then in step ST19, it is determined whether M is larger than the number Mn of learning data classified into the N class.
[0074]
When M> Mn, all of the learning data classified into the N class is used. Therefore, in step ST16, the coupling coefficient in the unit of the hidden layer and the output layer in the state is changed to the coupling coefficient of the N class. Wi is supplied to the memory 210 and stored therein, and then the coupling coefficient generation operation in the N class is terminated. On the other hand, if not M> Mn, the process returns to step ST12 and the above-described operation is repeated.
[0075]
Although not shown in FIG. 12, when the error ε does not fall within a certain range even when all of the learning data is used, the initial value of the coupling coefficient in the units of the output layer and the hidden layer and the equation (5) When the real number a in the transfer function is changed and the above-described generation operation is started again from the beginning, when M> Mn in step ST19, the initial value of the coupling coefficient and the real number a of the transfer function are changed, and step ST11 is performed. Return to.
[0076]
In the above embodiment, the ADRC circuits 103 (FIG. 1) and 204 (FIG. 9) are provided as information compression means for patterning the spatial waveform with a small number of bits, but this is only an example. Any information compression means that can be expressed in a class with few signal waveform patterns can be freely provided. For example, compression means such as DPCM (Differential Pulse Code Modulation) and VQ (Vector Quantization) can be used. Good.
[0077]
In the above-described embodiment, the HD pixel data is obtained from the SD pixel data. However, for example, when obtaining other SD pixel data whose resolution is higher than that of certain SD pixel data, or the resolution is higher than that of certain HD pixel data. The present invention can be applied in the same way to obtain other increased HD pixel data. In this case, the number of pixels before conversion and after conversion is the same.
[0078]
In the above-described embodiment, an example in which an NTSC video signal is converted into a high-definition video signal has been described. However, the present invention is not limited to this, and the first image signal is converted into the first image signal. Of course, the present invention can be similarly applied to the case of converting to a second image signal having the same or larger number of pixels as the signal.
[0079]
【The invention's effect】
According to the present invention, when the first image signal is converted into the second image signal having the same or larger number of pixels as the first image signal, the second image signal is configured using the neural network. Since the prediction of the pixel signal to be performed is performed and nonlinear prediction (including linear prediction) is performed, it is possible to always perform prediction of the optimal pixel signal as compared with the prediction based on the conventional linear linear expression. it can.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an image signal conversion apparatus according to an embodiment.
FIG. 2 is a diagram illustrating a configuration example of a main part of a neural network prediction circuit of an image signal conversion apparatus.
FIG. 3 is a diagram illustrating a configuration example of a hidden layer and an output layer unit constituting a neural network.
FIG. 4 is a schematic diagram for explaining a positional relationship between SD pixels and HD pixels.
FIG. 5 is a schematic diagram for explaining the positional relationship between SD pixels and HD pixels.
FIG. 6 is a schematic diagram for explaining SD pixel data used for space class classification.
FIG. 7 is a schematic diagram for explaining SD pixel data used for motion class classification;
FIG. 8 is a schematic diagram for explaining SD pixel data used for prediction of HD pixel data using a neural network.
FIG. 9 is a block diagram illustrating a configuration example of a coupling coefficient generation device that generates a coupling coefficient used in a neural network prediction circuit by learning;
FIG. 10 is a diagram illustrating a configuration example of a main part of a neural network learning circuit of a coupling coefficient generation device.
FIG. 11 is a flowchart showing a learning flow of coupling coefficients.
FIG. 12 is a flowchart for explaining N-class coupling coefficient generation processing in a learning flow;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 100 ... Image signal converter 101, 201 ... Input terminal, 102, 104, 108, 203, 205, 208 ... Area extraction circuit, 103, 204 ... ADRC circuit, 105, 206 ... Motion class determination circuit, 106, 207 ... class code generation circuit, 107 ... ROM table, 109 ... neural network prediction circuit, 110 ... output terminal, 121, 221 ... neural network unit, 122, 222 ... normalization unit, 209 ... neural network learning circuit, 210 ... memory, 223 ... error detection unit

Claims (7)

In the image signal conversion apparatus configured to convert the first image signal into a second image signal having the same or more pixels as the first image signal,
First pixel cutout means for cutting out a signal of a pixel in a first region located in the vicinity of a predetermined pixel constituting the second image signal from the first image signal;
A signal of a predetermined pixel constituting the second image signal to be detected based on the level distribution pattern of the signal of the pixel in the first region cut out by the first pixel cutout unit is detected. A class determining means for determining a class to which the class belongs and outputting class information;
At least an input layer and an output layer are provided, the units of each layer are combined with each other by a predetermined coupling coefficient, and the input value to the unit of the layer after the combination is combined with the output value of the plurality of units of the layer before the combination A neural network prediction means for predicting a signal of a predetermined pixel constituting the second image signal by a neural network including a non-linear operation of a product sum with a coefficient;
Coupling coefficient storage means for storing the coupling coefficient of the neural network prediction means corresponding to each class indicated by the class information;
A coupling coefficient reading means for reading out a coupling coefficient from the coupling coefficient storage means corresponding to the class information output from the class determining means;
Second pixel clipping means for clipping a signal of a pixel in a second region located in the vicinity of a predetermined pixel constituting the second image signal from the first image signal,
The neural network prediction means is
The coupling coefficient read out from the coupling coefficient storage means is used as the coupling coefficient of the neural network, and the pixel signal of the second region cut out by the second pixel cutting-out means is input to the input layer of the neural network. As an input value to the unit, the output value of the unit of the output layer of the neural network obtained by performing the operation of the neural network is converted into a pixel signal value corresponding to the number of bits for expressing the pixel signal value. And outputting as a predicted value of a predetermined pixel signal constituting the second image signal.   2. The image signal conversion apparatus according to claim 1, wherein the neural network prediction means is configured using a back-propagation network. In the image signal conversion method in which the first image signal is converted into a second image signal having the same or larger number of pixels as the first image signal,
A first step of cutting out a signal of a pixel in a first region located in the vicinity of a predetermined pixel constituting the second image signal from the first image signal;
The level distribution pattern of the signal of the pixel in the first region cut out in the first step is detected, and the signal of the predetermined pixel constituting the second image signal to be predicted based on this pattern belongs A second step of determining a class and outputting class information;
At least an input layer and an output layer are provided, the units of each layer are combined with each other by a predetermined coupling coefficient, and the input value to the unit of the layer after the combination is combined with the output value of the plurality of units of the layer before the combination A third step of predicting a signal of a predetermined pixel constituting the second image signal by a neural network including a nonlinear operation of a product sum with a coefficient;
A fourth step of reading out the coupling coefficient corresponding to each class indicated by the class information output in the second step by a storage means;
A fifth step of cutting out a signal of a pixel in a second region located in the vicinity of a predetermined pixel constituting the second image signal from the first image signal;
Comprising
In the third step,
The coupling coefficient read from the storage means is used as the coupling coefficient of the neural network, and the pixel signal of the second area cut out by the second pixel cutting-out means is sent to the unit of the input layer of the neural network. As an input value, the output value of the output layer unit of the neural network obtained by performing the operation of the neural network is converted into a pixel signal value corresponding to the number of bits for expressing the pixel signal value, An image signal conversion method comprising: outputting as a predicted value of a predetermined pixel signal constituting the second image signal. In an apparatus for generating a coupling coefficient of a neural network used when converting a first image signal into a second image signal having a larger number of pixels than the first image signal,
Signal processing means for obtaining an input signal corresponding to the first image signal by reducing the number of pixels of the teacher signal corresponding to the second image signal;
First pixel clipping means for sequentially cutting out signals of pixels in a first region located in the vicinity of a plurality of pixels constituting the teacher signal from the input signal;
A level distribution pattern of the signals of the pixels in the first region sequentially cut out by the first pixel cut-out means is detected, and the signals of the plurality of pixels constituting the teacher signal belong to each of the patterns based on the pattern. A class determining means for determining a class and outputting class information;
A second pixel cutout means for sequentially cutting out signals of pixels in a second region located in the vicinity of a plurality of pixels constituting the teacher signal from the input signal;
At least an input layer and an output layer are provided, the units of each layer are combined with each other by a predetermined coupling coefficient, and the input value to the unit of the layer after the combination is combined with the output value of the plurality of units of the layer before the combination Input the pixel values of the second region sequentially cut out by the second pixel cut-out means as input values to the unit of the input layer of the neural network including the non-linear calculation of the product sum with the coefficient, and the teacher signal The plurality of pixel values constituting the teacher signal output from the class determining means are learned by making the plurality of pixel values constituting the true value of the value based on the output value of the unit of the output layer, respectively. And a neural network learning means for obtaining the coupling coefficient of the neural network for each class to which the class belongs.   5. The coupling coefficient generation apparatus according to claim 4, wherein the neural network learning means is configured using a back-propagation network. The neural network learning means
A neural network unit composed of units of an input layer, a hidden layer, and an output layer, and a signal of a pixel in the second region sequentially cut out by the second pixel cutout unit is supplied to the input layer;
Normalization means for converting the output value from the unit of the output layer of the neural network unit into a pixel signal value corresponding to the number of bits for expressing the pixel signal value;
The pixel signal value output from the normalizing means is compared with the signal of a predetermined pixel constituting the teacher signal corresponding to the pixel signal of the second region supplied to the input layer of the neural network unit. And error detection means for detecting the error,
For each class indicated by the class information, a combination of a signal of the pixel in the second area sequentially cut out by the second pixel cut-out unit and a signal of a predetermined pixel constituting the teacher signal corresponding thereto. Coupling coefficient changing means for changing the coupling coefficient in the unit of the output layer and hidden layer of the neural network unit in a direction in which the error detected by the error detection means falls within a sufficiently small fixed range,
6. Coupling coefficient deciding means that uses a coupling coefficient in a unit of an output layer and a hidden layer of the neural network unit when the error falls within the certain range as a learning result. Coupling coefficient generator. In a method for generating a coupling coefficient of a neural network used when converting a first image signal into a second image signal having a larger number of pixels than the first image signal,
A first step of obtaining an input signal corresponding to the first image signal by processing the teacher signal corresponding to the second image signal to reduce the number of pixels;
A second step of sequentially cutting out signals of pixels in a first region located in the vicinity of a plurality of pixels constituting the teacher signal from the input signal;
The level distribution pattern of the signal of the pixels in the first region sequentially cut out in the second step is detected, and the class to which the signals of the plurality of pixels constituting the teacher signal belong respectively is detected based on the pattern. A third step of determining and outputting class information;
A fourth step of sequentially cutting out the signals of the pixels in the second region located in the vicinity of the plurality of pixels constituting the teacher signal from the input signal;
At least an input layer and an output layer are provided, the units of each layer are combined with each other by a predetermined coupling coefficient, and the input value to the unit of the layer after the combination is combined with the output value of the plurality of units of the layer before the combination A pixel value of the second region sequentially cut out in the fourth step is input as an input value to the unit of the input layer of the neural network including a nonlinear product-sum operation with a coefficient, and the teacher signal is configured The signals of the plurality of pixels constituting the teacher signal output in the third step belong to each of the plurality of pixel values to be learned by learning the true value of the value based on the output value of the output layer unit. And a fifth step of obtaining the coupling coefficient of the neural network for each class.

Images