JP3796844B2 - Image processing apparatus, image processing method, parameter generation apparatus, and parameter generation method - Google Patents

Description

【００６７】
そこで、一般化するために、予測係数ｗの集合でなる行列Ｗ、学習データの集合でなる行列Ｘ、および予測値Ｅ［ｙ］の集合でなる行列Ｙ’を、
【数１】

で定義すると、次のような観測方程式が成立する。
【００６８】

【００６９】
そして、この観測方程式に最小自乗法を適用して、ＨＤ画素の画素値ｙに近い予測値Ｅ［ｙ］を求めることを考える。この場合、ＨＤ画素の画素値（以下、適宜、教師データという）ｙの集合でなる行列Ｙ、およびＨＤ画素の画素値ｙに対する予測値Ｅ［ｙ］の残差ｅの集合でなる行列Ｅを、
【数２】

で定義すると、式（２）から、次のような残差方程式が成立する。
【００７０】

【００７１】
この場合、ＨＤ画素の画素値ｙに近い予測値Ｅ［ｙ］を求めるための予測係数ｗ_iは、自乗誤差
【数３】

を最小にすることで求めることができる。
【００７２】
従って、上述の自乗誤差を予測係数ｗ_iで微分したものが０になる場合、即ち、次式を満たす予測係数ｗ_iが、ＨＤ画素の画素値ｙに近い予測値Ｅ［ｙ］を求めるため最適値ということになる。
【００７３】
【数４】

【００７４】
そこで、まず、式（３）を、予測係数ｗ_iで微分することにより、次式が成立する。
【００７５】
【数５】

【００７６】
式（４）および（５）より、式（６）が得られる。
【００７７】
【数６】

【００７８】
さらに、式（３）の残差方程式における学習データｘ、予測係数ｗ、教師データｙ、および残差ｅの関係を考慮すると、式（６）から、次のような正規方程式を得ることができる。
【００７９】
【数７】

【００８０】
式（７）の正規方程式は、求めるべき予測係数ｗの数と同じ数だけたてることができ、従って、式（７）を解くことで（但し、式（７）を解くには、式（７）において、予測係数ｗにかかる係数で構成される行列が正則である必要がある）、最適な予測係数ｗを求めることができる。なお、式（７）を解くにあたっては、例えば、掃き出し法（Gauss-Jordanの消去法）などを適用することが可能である。
【００８１】
以上のようにして、最適な予測係数ｗを求めておき、さらに、その予測係数ｗを用い、式（１）により、ＨＤ画素の画素値ｙに近い予測値Ｅ［ｙ］を求めるのが適応処理であり、この適応処理が、積和演算器６２において行われるようになされている。
【００８２】
なお、適応処理は、ＳＤ画像には含まれていない、ＨＤ画像に含まれる成分が再現される点で、補間処理とは異なる。即ち、適応処理では、式（１）だけを見る限りは、いわゆる補間フィルタを用いての補間処理と同一であるが、その補間フィルタのタップ係数に相当する予測係数ｗが、教師データｙを用いての、いわば学習により求められるため、ＨＤ画像に含まれる成分を再現することができる。即ち、容易に、高解像度の画像を得ることができる。このことから、適応処理は、いわば画像の創造作用がある処理ということができる。
【００８３】
以上説明した適応処理は、ＨＤ画素の画素値ｙを教師データとして学習を行うことにより得られる予測係数ｗ₁，ｗ₂，・・・を用いて行うものであり、このような適応処理（以下、適宜、第１の適応処理という）は、本件出願人が先に提案している。
【００８４】
ところで、第１の適応処理によれば、予測係数ｗ₁，ｗ₂，・・・を求めるのに際し、即ち、学習に際し、ＨＤ画像が教師データとして必要となる。即ち、例えば、図２において、ＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄から、ＨＤ画素Ａ₄₃，Ａ₄₄，Ａ₄₅，Ａ₅₃，Ａ₅₄，Ａ₅₅，Ａ₆₃，Ａ₆₄，Ａ₆₅の予測値を第１の適応処理により求める場合、そのための予測係数ｗ₁，ｗ₂，・・・の算出には、ＨＤ画素Ａ₄₃，Ａ₄₄，Ａ₄₅，Ａ₅₃，Ａ₅₄，Ａ₅₅，Ａ₆₃，Ａ₆₄，Ａ₆₅の画素値が必要となる。
【００８５】
しかしながら、ＨＤ画像がなくても、予測係数ｗ₁，ｗ₂，・・・が求められれば便利である。
【００８６】
そこで、画像の自己相似性を利用し、例えば、次のように、ＳＤ画素のみから予測係数ｗ₁，ｗ₂，・・・を求め、予測回路６では、これを用いて適応処理（このような適応処理を、以下、適宜、第２の適応処理という）を行うようにすることが可能である。
【００８７】
即ち、図２において、予測値計算用ブロックを構成するＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄と、ＨＤ画素Ａ₄₃，Ａ₄₄，Ａ₄₅，Ａ₅₃，Ａ₅₄，Ａ₅₅，Ａ₆₃，Ａ₆₄，Ａ₆₅との間の位置関係に注目した場合、その位置関係は、間引きＳＤ画素Ｃ₁₁，Ｃ₁₂，Ｃ₁₃，Ｃ₂₁，Ｃ₂₂，Ｃ₂₃，Ｃ₃₁，Ｃ₃₂，Ｃ₃₃と、ＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄との間の位置関係と、いわば相似である。
【００８８】
従って、画像の自己相似性から、間引きＳＤ画素Ｃ₁₁，Ｃ₁₂，Ｃ₁₃，Ｃ₂₁，Ｃ₂₂，Ｃ₂₃，Ｃ₃₁，Ｃ₃₂，Ｃ₃₃を学習データとするとともに、ＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄を教師データとして学習を行い、これにより、予測係数ｗ₁，ｗ₂，・・・を求め、その予測係数ｗ₁，ｗ₂，・・・を、予測値計算用ブロックを構成するＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄から、ＨＤ画素Ａ₄₃，Ａ₄₄，Ａ₄₅，Ａ₅₃，Ａ₅₄，Ａ₅₅，Ａ₆₃，Ａ₆₄，Ａ₆₅の予測値を予測するのに用いることができる。
【００８９】
図９は、第２の適応処理を行うための予測係数を求める学習処理を行う画像処理装置の構成例を示している。
【００９０】
クラス分類用ブロック化回路７１、学習用ブロック化回路７５、および教師用ブロック化回路７６には、ＳＤ画像が供給される。
【００９１】
クラス分類用ブロック化回路７１、間引き回路７２、縮小回路７３、またはクラス分類回路７４では、図１のクラス分類用ブロック化回路１、間引き回路２、縮小回路３、またはクラス分類回路４における場合とそれぞれ同様の処理が行われ、これにより、後述する学習用ブロック化回路７５が出力する学習用ブロックのクラスに対応するインデックスが、学習回路７７に供給される。
【００９２】
一方、学習用ブロック化回路７５は、ＳＤ画像から、上述したように、学習データとするＳＤ画素（間引きＳＤ画素）Ｃ₁₁，Ｃ₁₂，Ｃ₁₃，Ｃ₂₁，Ｃ₂₂，Ｃ₂₃，Ｃ₃₁，Ｃ₃₂，Ｃ₃₃で構成される、予測値計算用ブロックより大きいブロックを構成し、これを学習用ブロックとして、学習回路７７に出力する。また、同時に、教師用ブロック化回路７６では、ＳＤ画像から、教師データとするＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄で構成されるブロックが構成され、これが教師用ブロックとして、やはり、学習回路７７に出力される。
【００９３】
学習回路７７では、学習用ブロックを構成するＳＤ画素を学習データとするとともに、教師用ブロックを構成するＳＤ画素を教師データとして、例えば、最小自乗法により、誤差を最小とする予測係数が算出される。
【００９４】
即ち、例えば、いま、学習用ブロックを構成するＳＤ画素（間引きＳＤ画素）の画素値を、ｘ₁，ｘ₂，ｘ₃，・・・とし、求めるべき予測係数をｗ₁，ｗ₂，ｗ₃，・・・とするとき、これらの線形１次結合により、教師用ブロックを構成する、あるＳＤ画素の画素値ｙを求めるには、予測係数ｗ₁，ｗ₂，ｗ₃，・・・は、次式を満たす必要がある。
【００９５】
ｙ＝ｗ₁ｘ₁＋ｗ₂ｘ₂＋ｗ₃ｘ₃＋・・・
【００９６】
そこで、学習回路７７では、学習用ブロックと教師用ブロックとから、真値ｙに対する、予測値ｗ₁ｘ₁＋ｗ₂ｘ₂＋ｗ₃ｘ₃＋・・・の自乗誤差を最小とする予測係数ｗ₁，ｗ₂，ｗ₃，・・・が、上述した式（７）に示す正規方程式をたてて解くことにより求められる。
【００９７】
学習回路７７において求められた予測係数は、クラス分類回路７４からのインデックスに対応するクラスの予測係数として出力される。
【００９８】
即ち、本実施の形態においては、学習ブロックを構成するＳＤ画素（間引きＳＤ画素）Ｃ₁₁，Ｃ₁₂，Ｃ₁₃，Ｃ₂₁，Ｃ₂₂，Ｃ₂₃，Ｃ₃₁，Ｃ₃₂，Ｃ₃₃から、教師用ブロックを構成する９個のＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄を求めるための予測係数を算出する必要がある。
【００９９】
このため、学習回路７７では、クラス分類回路７４が出力するインデックスに対応するクラスＣＬについて、ＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄それぞれを教師データとするとともに、ＳＤ画素（間引きＳＤ画素）Ｃ₁₁，Ｃ₁₂，Ｃ₁₃，Ｃ₂₁，Ｃ₂₂，Ｃ₂₃，Ｃ₃₁，Ｃ₃₂，Ｃ₃₃を学習データとして、式（７）に示した正規方程式がたてられる。
【０１００】
さらに、学習回路７７では、クラスＣＬにクラス分類される、他の学習用ブロックについても同様にして、正規方程式がたてられ、ＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄それぞれの予測値Ｅ［Ｂ₂₂］，Ｅ［Ｂ₂₃］，Ｅ［Ｂ₂₄］，Ｅ［Ｂ₃₂］，Ｅ［Ｂ₃₃］，Ｅ［Ｂ₃₄］，Ｅ［Ｂ₄₂］，Ｅ［Ｂ₄₃］，Ｅ［Ｂ₄₄］を求めるための予測係数ｗ₁（Ｂ₂₂）乃至ｗ₉（Ｂ₂₂），ｗ₁（Ｂ₂₃）乃至ｗ₉（Ｂ₂₃），ｗ₁（Ｂ₂₄）乃至ｗ₉（Ｂ₂₄），ｗ₁（Ｂ₃₂）乃至ｗ₉（Ｂ₃₂），ｗ₁（Ｂ₃₃）乃至ｗ₉（Ｂ₃₃），ｗ₁（Ｂ₃₄）乃至ｗ₉（Ｂ₃₄），ｗ₁（Ｂ₄₂）乃至₉（Ｂ₄₂），ｗ₁（Ｂ₄₃）乃至ｗ₉（Ｂ₄₃），ｗ₁（Ｂ₄₄）乃至ｗ₉（Ｂ₄₄）（本実施の形態では、１つの予測値を求めるのに学習データが９個用いられるので、それに対応して、予測係数ｗも９個必要となる）を算出することができるだけの数の正規方程式が得られると（従って、そのような数の正規方程式が得られるまでは、学習回路７７では、正規方程式が繰り返したてられる）、その正規方程式を解くことで、クラスＣＬについて、ＳＤ画素Ｂ_3+m,3+nの予測値Ｅ［Ｂ_3+m,3+n］を求めるのに最適な予測係数ｗ₁（Ｂ_3+m,3+n）乃至ｗ₉（Ｂ_3+m,3+n）が算出される（但し、ここでは、ｍ＝−１，０，＋１、ｎ＝−１，０，＋１）。
【０１０１】
図１の予測回路６を構成する係数ＲＯＭ６１（図８）には、以上のようにして学習回路７７から出力される予測係数を記憶させておくことができ、この場合、積和演算器６２では、式（１）に対応する次式にしたがって、予測値計算用ブロック内におけるＨＤ画素Ａ₄₃，Ａ₄₄，Ａ₄₅，Ａ₅₃，Ａ₅₄，Ａ₅₅，Ａ₆₃，Ａ₆₄，Ａ₆₅それぞれの予測値Ｅ［Ａ₄₃］，Ｅ［Ａ₄₄］，Ｅ［Ａ₄₅］，Ｅ［Ａ₅₃］，Ｅ［Ａ₅₄］，Ｅ［Ａ₅₅］，Ｅ［Ａ₆₃］，Ｅ［Ａ₆₄］，Ｅ［Ａ₆₅］が求められる。
【０１０２】

【０１０３】
以上のように、ＳＤ画素Ｂ₂₂，Ｂ₂₃，Ｂ₂₄，Ｂ₃₂，Ｂ₃₃，Ｂ₃₄，Ｂ₄₂，Ｂ₄₃，Ｂ₄₄それぞれの画素値（予測値）を、間引きＳＤ画素Ｃ₁₁，Ｃ₁₂，Ｃ₁₃，Ｃ₂₁，Ｃ₂₂，Ｃ₂₃，Ｃ₃₁，Ｃ₃₂，Ｃ₃₃で構成される所定の学習用ブロック（所定のブロック）から算出することができるように、学習を行うことで予測係数を求め、その予測係数を用いて、学習用ブロックより小さい（注目画素Ｂ₃₃を中心として学習ブロックを、いわば縮小した）予測値計算用ブロック（小ブロック）に対して適応処理を施した場合によれば、画像の自己相似性によって、ＨＤ画素Ａ₄₃，Ａ₄₄，Ａ₄₅，Ａ₅₃，Ａ₅₄，Ａ₅₅，Ａ₆₃，Ａ₆₄，Ａ₆₅についての適正な予測値を得ることができる。
【０１０４】
次に、図１０は、図９の学習回路７７の構成例を示している。
【０１０５】
乗算回路８１には、学習用ブロックを構成する学習データｘ₁，ｘ₂，・・・，ｘ_mと、教師用ブロックを構成する教師データｙとが入力されるようになされており、そこでは、式（７）の正規方程式におけるサメーション（Σ）の対象となる学習データｘ₁，ｘ₂，・・・，ｘ_mどうしの積、および学習データｘ₁，ｘ₂，・・・，ｘ_mそれぞれと教師データｙとの積が求められ、加算回路８２に供給される。
【０１０６】
加算回路８２には、乗算回路８１の出力の他、デコーダ８３の出力も供給されるようになされている。デコーダ８３には、クラス分類回路７４（図９）からインデックスが供給されるようになされており、デコーダ８３は、そのインデックスに基づいて、学習データｘ₁，ｘ₂，・・・，ｘ_mのクラス（学習用ブロックのクラス）を認識し、その認識結果を、加算回路８２に出力する。
【０１０７】
加算回路８２は、乗算回路８１の出力を用いて、式（７）の正規方程式におけるサメーションに相当する演算を、デコーダ８３からのクラスごとに独立して行い、その演算結果を、演算回路８４に供給する。演算回路８４では、加算回路８２の出力を用い、掃き出し法による演算が行われ、これにより、予測係数が算出されて出力される。
【０１０８】
次に、図１１は、図１０の乗算回路８１の構成例を示している。
【０１０９】
乗算回路８１は、同図に示すように、乗算器アレイで構成されている（乗算器が所定形状に配列されて構成されている）。即ち、乗算回路８１は、式（７）の正規方程式の左辺における予測係数ｗの係数（サメーションの部分）、およびその右辺における各項に対応する乗算器から構成されている。
【０１１０】
なお、式（７）の正規方程式の左辺における予測係数ｗにかかる係数で構成される行列（以下、適宜、係数行列という）と、その転置行列とは等しいため、図１１の実施の形態では、乗算器は、係数行列の対角成分を含む右上部分の成分と、式（７）の右辺の項に対応する乗算器だけが設けられている。
【０１１１】
以上のように構成される乗算回路８１では、各乗算器において、上述したように、式（７）の正規方程式におけるサメーション（Σ）の対象となる学習データｘ₁，ｘ₂，・・・，ｘ_mどうしの積、および学習データｘ₁，ｘ₂，・・・，ｘ_mそれぞれと教師データｙとの積が求められ、加算回路８２に供給される。
【０１１２】
次に、図１２は、図１０の加算回路８２の構成例を示している。
【０１１３】
加算回路８２は、図１１の乗算回路８１を構成する乗算器と同様に加算器またはメモリセルがそれぞれ配置された加算器アレイまたはメモリアレイ（レジスタアレイ）から構成されている。なお、加算器アレイは、乗算器アレイと同様に１つだけ設けられているが、メモリアレイは、クラスに対応する数だけ設けられている。
【０１１４】
以上のように構成される加算回路８２では、加算器アレイを構成する各加算器に、乗算器アレイの、対応する乗算器の出力が供給される。さらに、各加算器には、デコーダ８３からのクラスに対応するメモリアレイを構成する、対応するメモリセルの記憶値が供給される。各加算器では、乗算器とメモリセルの出力どうしが加算され、その加算結果が、元のメモリセルに供給される。そして、各メモリセルでは、加算器から供給される加算結果が記憶され、加算回路８２は、以下、同様の処理を繰り返すことで、式（７）の正規方程式におけるサメーションに相当する演算を行う。
【０１１５】
これにより、各メモリアレイには、対応するクラスについての、正規方程式の各項の係数が記憶されることになる。
【０１１６】
そして、図１０の演算回路８４では、メモリアレイの各メモリセルの記憶値を用いて、掃き出し法により、各クラスごとの予測係数が求められる。
【０１１７】
ところで、例えば、ＮＴＳＣ方式などに準拠したＳＤ画像を受信する受信装置などに、図１の画像変換装置を内蔵させ、その予測回路６において第１の適応処理によりＨＤ画素の予測値を求めるようにした場合においては、上述したように、学習に際し、教師データとしてＨＤ画素が必要となるため、受信装置において、予測係数の更新をすることは困難である。
【０１１８】
その一方、第２の適応処理によりＨＤ画素の予測値を求めるようにした場合においては、教師データとしてＨＤ画素は必要でなく、ＳＤ画素（間引きＳＤ画素を含む）だけで学習を行うことができ、従って、受信装置において、予測係数を更新することができる。
【０１１９】
そこで、図１３は、予測係数を更新しながら、ＳＤ画像をＨＤ画像に変換する画像変換装置の一実施の形態の構成を示している。なお、図中、図１における場合と対応する部分については、同一の符号を付してあり、以下では、その説明は、適宜省略する。即ち、この画像変換装置は、フレームメモリ９１、学習用ブロック化回路９２、教師用ブロック化回路９３、学習回路９４、および係数ＲＡＭ（例えば、ＳＲＡＭ（Static Read Only Memory）など）９５が新たに設けられているとともに、予測回路６に代えて予測回路９６が設けられている他は、図１の画像変換装置と同様に構成されている。
【０１２０】
フレームメモリ９１には、伝送路を介して伝送され、または、記録媒体から再生されたＳＤ画像が、例えば、フレーム（またはフィールド）単位で記憶される。フレームメモリ９１に記憶されたＳＤ画像は、クラス分類用ブロック化回路１に供給され、以下、図１における場合と同様にして、クラス分類回路４からは、後述する学習用ブロック９２が出力する学習用ブロックのクラスに対応するインデックスが出力される。
【０１２１】
また、フレームメモリ９１に記憶されたＳＤ画像は、同時に、学習用ブロック化回路９２および教師用ブロック化回路９３にも供給される。
【０１２２】
学習用ブロック化回路９２、教師用ブロック化回路９３、または学習回路９４は、図９における学習用ブロック化回路７５、教師用ブロック化回路７６、または学習回路７７と同様に構成されており、また、学習回路９４には、クラス分類回路７４（図９）に対応するクラス分類回路４からインデックスが供給されるようになされている。従って、学習回路９４では、図９で説明したような学習が行われ、その結果得られる予測係数が、係数ＲＡＭ９５に供給される。
【０１２３】
係数ＲＡＭ９５には、学習回路９４から予測係数が供給される他、クラス分類回路４からインデックスが供給されるようになされており、係数ＲＡＭ９５では、そのインデックスに対応するアドレスに、学習回路９４からの予測係数が記憶（上書き）される。
【０１２４】
以上の学習処理が、フレームメモリ９１に記憶されたＳＤ画像を構成する、例えば、すべての画素を注目画素として行われると、予測値計算用ブロック化回路５は、フレームメモリ９１に記憶されたＳＤ画像から予測値計算用ブロックを順次構成し、予測回路９６に出力する。
【０１２５】
また、このとき、フレームメモリ９１に記憶されたＳＤ画像は、クラス分類用ブロック化回路１にも、再び供給され、以下、上述した場合と同様にして、予測値計算用ブロック化回路５が構成する予測値計算用ブロックのクラスに対応するインデックスが、クラス分類回路４から出力される。
【０１２６】
このインデックスは、係数ＲＡＭ９５にアドレスとして与えられ、そのアドレスに記憶された予測係数が、係数ＲＡＭ９５から読み出されて予測回路９６に供給される。
【０１２７】
予測回路９６は、図８に示した予測回路６を構成するブロックのうちの、係数ＲＯＭ６１を除く、積和演算器６２およびリミッタ６３で構成され、そこでは、係数ＲＡＭ９５からの予測係数を用い、式（８）で説明したようにしてＨＤ画素の予測値が求められる。
【０１２８】
以上のように、ＨＤ画像に変換しようとするＳＤ画像から予測係数を求め、その予測係数を用いて、ＳＤ画像をＨＤ画像に変換する場合によれば、より精度の高いＨＤ画像を得ることが可能となる。
【０１２９】
以上、本発明を、ＳＤ画像をＨＤ画像に変換する画像変換装置に適用した場合について説明したが、本発明は、その他、例えば、画像の拡大処理などを行う場合にも適用可能である。
【０１３０】
なお、本実施の形態においては、係数ＲＯＭ６１（図８）には（図１３における係数ＲＡＭ９５についても同様）、各クラスに対応するアドレスに、予測係数を記憶させるようにしたが、係数ＲＯＭ６１には、その他、例えば、教師用ブロックを構成する画素値の平均値などを記憶させるようにすることが可能である。この場合、クラスについてのインデックスが与えられると、そのクラスに対応する画素値が出力されることになり、予測値計算用ブロック化回路５および予測回路６（または予測回路９６）を設けずに済むようになる。
【０１３１】
さらに、本実施の形態においては、第１クラス情報および第２クラス情報の両方から、最終的なクラスを決定するようにしたが、第１クラス情報または第２クラス情報のうちのいずれか一方から最終的なクラスを決定するようにすることも可能である。
【０１３２】
また、本実施の形態では、間引き回路２において、画素を単純に間引くことにより、間引きブロックを構成するようにしたが、間引きブロックは、その他、例えば、幾つかの画素の平均値などを１の画素に割り当てることなどによって生成するようにすることなども可能である。
【０１３３】
さらに、本実施の形態では、各ブロックの形状を正方形としたが、ブロックの形状は正方形に限定されるものではない。即ち、本明細書中におけるブロックとは、幾つかの画素の集合を意味し、その形状は、正方形の他、例えば、長方形や、十字形、円形その他の任意の形状とすることができる。
【０１３４】
また、本実施の形態においては、画像の自己相似性を考慮してクラス分類を行うようにしたが、クラス分類は、画像の自己相似性を考慮せず、単純に、画素値のパターンにのみ基づいて行うようにすることも可能である。即ち、例えば、いま、図１４（Ａ）に示すように、ある注目画素と、それに隣接する３つの画素により、２×２画素でなるブロックを構成し、また、各画素は、１ビットで表現される（０または１のうちのいずれかのレベルをとる）ものとする。この場合、２×２の４画素のブロックは、各画素のレベル分布により、図１４（Ｂ）に示すように、１６（＝（２¹）⁴）パターンに分類することができる。クラス分類は、このようなパターン分けのみによって行うようにすることも可能である。
【０１３５】
さらに、本発明は、ハードウェアおよびソフトウェアのいずれによっても実現可能である。
【０１３６】
【発明の効果】
請求項１に記載の画像処理装置および請求項６に記載の画像処理方法によれば、予測パラメータは、第１の画素のうちの予測パラメータの算出において注目している画素である第２の注目画素と第２の注目画素の周辺の第１の画素とからなる所定の大きさの第２のブロックを構成するように、第１の画像信号をブロック化し、第２の注目画素が属する第２のブロックであって、第２の注目画素の位置を含む所定の大きさの第２のブロックを構成する第１の画素に基づき、クラスを表す第２のクラス情報を生成し、予測される第２の画素の位置に対応する位置であって、第１の画素と同じ位置かまたはその間の位置から、その第２の画素の予測のために積和演算される第１の画素のそれぞれの位置までの方向およびその間の第１の距離に対し、同じ方向であって第１の距離に比例する第２の距離の位置である相似の位置関係にある第１の画素を用いて、画素値の予測値の、真値に対する誤差を最小にする学習を行うことにより、第２のクラス情報で示されるクラスごとに算出されたものであり、その予測パラメータを用いて適応処理を施すことにより、第２の画像の画素値が求められる。従って、第２の画像として、高解像度の画像を、容易に得ることが可能となる。
請求項５に記載の画像処理装置によれば、第１の画素のうちの注目している画素である第１の注目画素と第１の注目画素の周辺の第１の画素とからなる所定の大きさの第１のブロックを構成するように、第１の画像信号がブロック化され、第１の注目画素が属する第１のブロックであって、第１の注目画素の位置を含む所定の大きさの第１のブロックを構成する第１の画素に基づき、クラスを表す第１のクラス情報が生成され、第１の画像信号を用いて第２の画像信号を予測するための予測パラメータが記憶され、第１のクラス情報に応じて予測パラメータ記憶手段より読み出される予測パラメータと第１の注目画素の周辺に位置する第１の画素との積和演算により、第２の画像信号の第２の画素であって、第１の注目画素の位置と対応する位置に対して、同じかまたは対応する位置の周辺の第２の画素が予測され、予測される第２の画素の位置に対応する位置であって、第１の画素と同じ位置かまたはその間の位置から、その第２の画素の予測のために積和演算される第１の画素のそれぞれの位置までの方向およびその間の第１の距離に対し、同じ方向であって第１の距離に比例する第２の距離の位置である相似の位置関係にある第１の画素を用いて、画素値の予測値の、真値に対する誤差を最小にする学習を行うことにより、クラスごとの予測パラメータが算出されるようにしたので、第２の画像として、高解像度の画像を、容易に得ることが可能となる。
また、請求項７に記載のパラメータ生成装置および請求項１３に記載のパラメータ生成方法によれば、予測パラメータは、第１の画素のうちの注目している画素である注目画素と注目画素の周辺の第１の画素とからなる所定の大きさのブロックを構成するように、第１の画像信号がブロック化され、注目画素が属するブロックであって、注目画素の位置を含む所定の大きさのブロックを構成する第１の画素に基づき、クラスを表すクラス情報が生成される。そして、予測される第２の画素の位置に対応する位置であって、第１の画素と同じ位置かまたはその間の位置から、その第２の画素の予測のために積和演算される第１の画素のそれぞれの位置までの方向およびその間の第１の距離に対し、同じ方向であって第１の距離に比例する第２の距離の位置である相似の位置関係にある第１の画素を用いて、画素値の予測値の、真値に対する誤差を最小にする学習を行うことにより、クラスごとに算出されたものであり、即ち、第１の画像信号の画素のみを用いて、画素値の予測値の、真値に対する誤差を最小にする学習を行うことにより、予測パラメータを得ることが可能となる。
また、請求項１４に記載の画像処理装置および請求項１５に記載の画像処理方法によれば、予測パラメータは、積和演算に用いられる第１の画像信号の画素と、第１の画像信号を低解像度化させた第３の画像信号の画素とを用いて、画素値の予測値の、真値に対する誤差を最小にするように生成されたものであり、その予測パラメータ、即ち、第１の画像信号の画素のみを用いて生成された予測パラメータを用いて適応処理を施すことにより、第２の画像の画素値が求められる。従って、第２の画像として、高解像度の画像を、容易に得ることが可能となる。
【図面の簡単な説明】
【図１】本発明を適用した画像変換装置の第１の実施の形態の構成を示すブロック図である。
【図２】図１の画像変換装置の処理を説明するための図である。
【図３】図１のクラス分類回路４の構成例を示すブロック図である。
【図４】図３の第１クラス生成回路１１の構成例を示すブロック図である。
【図５】図３の第２クラス生成回路１２の構成例を示すブロック図である。
【図６】ＡＤＲＣ処理を説明するための図である。
【図７】図３の最終クラス決定回路１３の構成例を示すブロック図である。
【図８】図１の予測回路６の構成例を示すブロック図である。
【図９】予測係数を求める学習処理を行う画像処理装置の構成例を示すブロック図である。
【図１０】図９の学習回路７７の構成例を示すブロック図である。
【図１１】図１０の乗算回路８１の構成例を示すブロック図である。
【図１２】図１０の加算回路８２の構成例を示すブロック図である。
【図１３】本発明を適用した画像変換装置の第２の実施の形態の構成を示すブロック図である。
【図１４】画素値のレベルのパターンにのみ基づいて行うクラス分類を説明するための図である。
【符号の説明】
１ クラス分類用ブロック化回路， ２ 間引き回路， ３ 縮小回路， ４クラス分類回路， ５ 予測値計算用ブロック化回路， ６ 予測回路， ７モニタ， １１ 第１クラス生成回路， １２ 第２クラス生成回路， １３最終クラス決定回路， ２１₁乃至２１₄ 相似性算出部， ２２ 最大相似性方向判定部， ２６Ｂ，２６Ｃ 画素抽出部， ２７ ノルム計算部， ３１ 最大相似性方向画素抽出部， ３２ ＡＤＲＣ処理部， ３３ パターン分類部， ４１ 最大値検出部， ４２ 最小値検出部， ４３ 遅延部， ４４ 演算器， ４５ ＡＤＲＣコード決定部， ５１ ＲＯＭ， ６１ 係数ＲＯＭ，６２ 積和演算器， ６３ リミッタ， ７１ クラス分類用ブロック化回路， ７２ 間引き回路， ７３ 縮小回路， ７４ クラス分類回路， ７５ 学習ブロック化回路， ７６ 教師用ブロック化回路， ７７ 学習回路， ８１ 乗算回路， ８２ 加算回路， ８３ デコーダ， ８４ 演算回路， ９１ フレームメモリ， ９２ 学習用ブロック化回路， ９３ 教師用ブロック化回路， ９４ 学習回路， ９５ 係数ＲＡＭ， ９６ 予測回路[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus.,Image processing method, Parameter generation apparatus and parameter generation methodIn particular, for example, an image processing apparatus suitable for use in, for example, converting a standard resolution image into a high resolution image.,Image processing method, Parameter generation apparatus and parameter generation methodAbout.
[0002]
[Prior art]
For example, when converting a standard resolution or low resolution image (hereinafter referred to as an SD image as appropriate) to a high resolution image (hereinafter referred to as an HD image as appropriate) or enlarging the image, Interpolation (compensation) of the pixel values of the missing pixels is performed by a so-called interpolation filter or the like.
[0003]
[Problems to be solved by the invention]
However, even if pixel interpolation is performed using an interpolation filter, it is difficult to obtain a high-resolution image because the HD image component (high-frequency component) that is not included in the SD image cannot be restored.
[0004]
The present invention has been made in view of such a situation, and makes it possible to easily obtain a high-resolution image.
[0005]
[Means for Solving the Problems]
  The image processing apparatus according to claim 1,To form a first block of a predetermined size composed of a first pixel of interest that is a pixel of interest among the first pixels and a first pixel around the first pixel of interest,Blocking means for blocking the first image signal;First pixel of interestBelongs toFirst blockBecauseFirst pixel of interestOf a predetermined size including the position ofFirst blockRepresents a class based on the first pixel comprisingFirst class informationClassifying means for generating a prediction parameter storage means for storing a prediction parameter for predicting a second image signal using the first image signal;First class informationA prediction parameter read from the prediction parameter storage means according toFirst pixel of interestThe second pixel of the second image signal by the product-sum operation with the first pixel located aroundThe second pixel in the vicinity of the same position or corresponding position with respect to the position corresponding to the position of the first target pixel.Prediction prediction means for predicting, and the prediction parameter storage means,A second block having a predetermined size including a second target pixel that is a pixel of interest in calculation of a prediction parameter among the first pixels and a first pixel around the second target pixel. As described above, the first image signal is blocked, and a second block to which the second pixel of interest belongs and which is a second block having a predetermined size including the position of the second pixel of interest is formed. Based on the first pixel, the second class information representing the class is generated, and is a position corresponding to the predicted position of the second pixel, from the same position as the first pixel or a position in between, A second direction that is proportional to the first distance in the same direction with respect to the direction to each position of the first pixel that is sum-of-products calculated for the prediction of the second pixel and the first distance therebetween. The first pixel in the similar positional relationship that is the position of the distance of There are, of predicted values of the pixel values, by performing learning to minimize differences between true values, the prediction parameter calculated for each class indicated by the second class informationIs stored.
  The image processing apparatus according to claim 5, wherein the image processing apparatus has a predetermined size including a first pixel of interest which is a pixel of interest among the first pixels and a first pixel around the first pixel of interest. Block means for blocking the first image signal so as to constitute the first block, and a first block to which the first pixel of interest belongs, and includes a predetermined block including the position of the first pixel of interest In order to predict the second image signal using the first image signal and the class classification means for generating the first class information representing the class based on the first pixel constituting the first block of the size By a sum-of-products operation of a prediction parameter storage unit that stores the prediction parameter, a prediction parameter read from the prediction parameter storage unit according to the first class information, and a first pixel located around the first pixel of interest , Second of the second image signal Prediction operation means for predicting a second pixel in the vicinity of the same or corresponding position with respect to a position corresponding to the position of the first target pixel, and a predicted second pixel A position corresponding to the position, from a position that is the same as or between the first pixel to a position of each of the first pixels that is summed for prediction of the second pixel, and The first pixel in the same direction and the position of the second distance that is proportional to the first distance with respect to the first distance in the meantime using the first pixel having a similar positional relationship, And learning means for calculating a prediction parameter for each class by performing learning that minimizes an error with respect to a true value.
[0006]
  The image processing method according to claim 6 comprises:To form a first block of a predetermined size composed of a first pixel of interest that is a pixel of interest among the first pixels and a first pixel around the first pixel of interest,A block step of blocking the first image signal;First pixel of interestBelongs toFirst blockBecauseFirst pixel of interestOf a predetermined size including the position ofFirst blockRepresents a class based on the first pixel comprisingFirst class informationA classifying step for generating a prediction parameter storage means for storing a prediction parameter for predicting a second image signal using the first image signal,First class informationPrediction parameters read out according toFirst pixel of interestThe second pixel of the second image signal by the product-sum operation with the first pixel located aroundThe second pixel in the vicinity of the same position or corresponding position with respect to the position corresponding to the position of the first target pixel.Predictive calculation step for predicting the prediction parameter storage means,A second block having a predetermined size including a second target pixel that is a pixel of interest in calculation of a prediction parameter among the first pixels and a first pixel around the second target pixel. As described above, the first image signal is blocked, and a second block to which the second pixel of interest belongs and which is a second block having a predetermined size including the position of the second pixel of interest is formed. Based on the first pixel, the second class information representing the class is generated, and is a position corresponding to the predicted position of the second pixel, from the same position as the first pixel or a position in between, A second direction that is proportional to the first distance in the same direction with respect to the direction to each position of the first pixel that is sum-of-products calculated for the prediction of the second pixel and the first distance therebetween. The first pixel in the similar positional relationship that is the position of the distance of There are, of predicted values of the pixel values, by performing learning to minimize differences between true values, the prediction parameter calculated for each class indicated by the second class informationIs stored.
  The parameter generation device according to claim 7,In order to form a block of a predetermined size composed of a target pixel that is a target pixel of the first pixels and a first pixel around the target pixel,Block means for blocking the first image signal, and class information representing a class based on a first pixel constituting a block having a predetermined size including a position of the target pixel, to which the target pixel belongs. Class classification means to be generated and predicted second pixel positionIs the same position as the first pixel or a position in betweenTo the direction from the first pixel to the position where the first pixel is summed for prediction of the second pixel and the first distance therebetween, the same direction and proportional to the first distance Prediction parameters for each class are calculated by performing learning to minimize the error of the predicted value of the pixel value with respect to the true value using the first pixel having a similar positional relationship that is the position of the second distance. And learning means for performing.
  The parameter generation method according to claim 13 is:In order to form a block of a predetermined size composed of a target pixel that is a target pixel of the first pixels and a first pixel around the target pixel,A block step for blocking the first image signal, and class information representing a class based on a first pixel constituting a block of a predetermined size including a position of the pixel of interest to which the pixel of interest belongs. Class classification step to generate and predicted second pixel locationIs the same position as the first pixel or a position in betweenTo the direction from the first pixel to the position where the first pixel is summed for prediction of the second pixel and the first distance therebetween, the same direction and proportional to the first distance Prediction parameters for each class are calculated by performing learning to minimize the error of the predicted value of the pixel value with respect to the true value using the first pixel having a similar positional relationship that is the position of the second distance. And a learning step.
  An image processing apparatus according to claim 14 is provided.A predetermined number of pixels centered on a pixel of interest that is a pixel of interest among pixels of the first image signal, each of which is a predetermined number of pixels, and configured as a block composed of pixels of the first image signal.Block means for blocking the first image signal, class classification means for generating class information representing a class based on the pixels constituting the block to which the pixel of interest belongs, and the second image using the first image signal A prediction parameter storage unit that stores a prediction parameter for predicting a signal; a product of the prediction parameter read from the prediction parameter storage unit according to the class information and a pixel of the first image signal around the position of the target pixel A prediction calculation unit that generates a second image signal by sum calculation, and the prediction parameter storage unit reduces the resolution of the pixel of the first image signal used for the product-sum calculation and the first image signal. The prediction parameter for each class generated so as to minimize the error of the predicted value of the pixel value with respect to the true value is stored using the pixel of the third image signal that has been set. And wherein the door.
  The image processing method according to claim 15 comprises:A predetermined number of pixels centered on a pixel of interest that is a pixel of interest among pixels of the first image signal, each of which is a predetermined number of pixels, and configured as a block composed of pixels of the first image signal.A block step for blocking the first image signal, a class classification step for generating class information representing a class based on the pixels constituting the block to which the pixel of interest belongs, and a second image using the first image signal From a prediction parameter storage unit that stores a prediction parameter for predicting a signal, a product-sum operation is performed on the prediction parameter read according to the class information and the pixel of the first image signal around the position of the target pixel. A prediction calculation step for generating the second image signal, and the prediction parameter storage means includes a pixel of the first image signal used for the product-sum operation and a third resolution obtained by reducing the resolution of the first image signal. A prediction parameter for each class generated so as to minimize an error of a predicted value of a pixel value with respect to a true value using a pixel of an image signal is stored. That.
[0007]
  In the image processing apparatus according to claim 1, the block means includes:To form a first block of a predetermined size composed of a first pixel of interest that is a pixel of interest among the first pixels and a first pixel around the first pixel of interest,The first image signal is blocked, and the class classification means isFirst pixel of interestBelongs toFirst blockBecauseFirst pixel of interestOf a predetermined size including the position ofFirst blockRepresents a class based on the first pixel comprisingFirst class informationIs generated. The prediction parameter storage means stores a prediction parameter for predicting the second image signal using the first image signal. And the prediction calculation means isFirst class informationA prediction parameter read from the prediction parameter storage means according toFirst pixel of interestThe second pixel of the second image signal by the product-sum operation with the first pixel located aroundThe second pixel in the vicinity of the same position or corresponding position with respect to the position corresponding to the position of the first target pixel.Predict. Furthermore, the prediction parameter storage meansA second block having a predetermined size including a second target pixel that is a pixel of interest in calculation of a prediction parameter among the first pixels and a first pixel around the second target pixel. As described above, the first image signal is blocked, and a second block to which the second pixel of interest belongs and which is a second block having a predetermined size including the position of the second pixel of interest is formed. Based on the first pixel, the second class information representing the class is generated, and is a position corresponding to the predicted position of the second pixel, from the same position as the first pixel or a position in between, A second direction that is proportional to the first distance in the same direction with respect to the direction to each position of the first pixel that is sum-of-products calculated for the prediction of the second pixel and the first distance therebetween. The first pixel in the similar positional relationship that is the position of the distance of There are, of predicted values of the pixel values, by performing learning to minimize differences between true values, the prediction parameter calculated for each class indicated by the second class informationIt is made to memorize.
  In the image processing device according to claim 5, a predetermined size including a first pixel of interest which is a pixel of interest among the first pixels and a first pixel around the first pixel of interest. The first image signal is blocked to form the first block, and the first block to which the first pixel of interest belongs has a predetermined size including the position of the first pixel of interest. First class information representing a class is generated based on the first pixel constituting the first block of the first block, and a prediction parameter for predicting the second image signal using the first image signal is stored. The second pixel of the second image signal by the product-sum operation of the prediction parameter read from the prediction parameter storage unit according to the first class information and the first pixel located around the first target pixel And corresponding to the position of the first pixel of interest. A second pixel in the vicinity of the same or corresponding position with respect to the position is predicted and corresponds to the predicted position of the second pixel, and is the same position as or in between the first pixel The same direction and proportional to the first distance with respect to the direction from the position to the respective positions of the first pixels to be summed for prediction of the second pixel and the first distance therebetween By performing learning that minimizes the error of the predicted value of the pixel value with respect to the true value using the first pixel having a similar positional relationship that is the position of the second distance to be predicted, the prediction parameter for each class Calculated.
[0008]
  In the image processing method according to claim 6,To form a first block of a predetermined size composed of a first pixel of interest that is a pixel of interest among the first pixels and a first pixel around the first pixel of interest,The first image signal is blocked,First pixel of interestBelongs toFirst blockBecauseFirst pixel of interestOf a predetermined size including the position ofFirst blockRepresents a class based on the first pixel comprisingFirst class informationIs generated. And from the prediction parameter memory | storage means which memorize | stores the prediction parameter for predicting a 2nd image signal using a 1st image signal,First class informationPrediction parameters read out according toFirst pixel of interestThe second pixel of the second image signal by the product-sum operation with the first pixel located aroundThe second pixel in the vicinity of the same position or corresponding position with respect to the position corresponding to the position of the first target pixel.Predict. Further, the prediction parameter storage means includesA second block having a predetermined size including a second target pixel that is a pixel of interest in calculation of a prediction parameter among the first pixels and a first pixel around the second target pixel. As described above, the first image signal is blocked, and a second block to which the second pixel of interest belongs and which is a second block having a predetermined size including the position of the second pixel of interest is formed. Based on the first pixel, the second class information representing the class is generated, and is a position corresponding to the predicted position of the second pixel, from the same position as the first pixel or a position in between, A second direction that is proportional to the first distance in the same direction with respect to the direction to each position of the first pixel that is sum-of-products calculated for the prediction of the second pixel and the first distance therebetween. The first pixel in the similar positional relationship that is the position of the distance of There are, of predicted values of the pixel values, by performing learning to minimize differences between true values, the prediction parameter calculated for each class indicated by the second class informationIs memorized.
  In the parameter generation device according to claim 7, the block means includesIn order to form a block of a predetermined size composed of a target pixel that is a target pixel of the first pixels and a first pixel around the target pixel,Class information representing a class based on a first pixel which blocks the first image signal and the class classification means is a block to which the pixel of interest belongs and which constitutes a block of a predetermined size including the position of the pixel of interest Is generated. The learning means then predicts the position of the second pixelIs the same position as the first pixel or a position in betweenTo the direction from the first pixel to the position where the first pixel is summed for prediction of the second pixel and the first distance therebetween, the same direction and proportional to the first distance Prediction parameters for each class are calculated by performing learning to minimize the error of the predicted value of the pixel value with respect to the true value using the first pixel having a similar positional relationship that is the position of the second distance. It is made to do.
  In the parameter generation method according to claim 13,In order to form a block of a predetermined size composed of a target pixel that is a target pixel of the first pixels and a first pixel around the target pixel,The first image signal is blocked, and class information representing a class is generated on the basis of the first pixel constituting a block having a predetermined size including the position of the target pixel, to which the target pixel belongs. . And the predicted position of the second pixelIs the same position as the first pixel or a position in betweenTo the direction from the first pixel to the position where the first pixel is summed for prediction of the second pixel and the first distance therebetween, the same direction and proportional to the first distance A prediction parameter for each class is calculated by performing learning that minimizes an error of a predicted value of a pixel value with respect to a true value using a first pixel having a similar positional relationship that is a position of a second distance. It is made to be done.
  In the image processing device according to claim 14, the block means includes:A predetermined number of pixels centered on a pixel of interest that is a pixel of interest among pixels of the first image signal, each of which is a predetermined number of pixels, and configured as a block composed of pixels of the first image signal.The first image signal is blocked, and the class classification unit generates class information representing the class based on the pixels constituting the block to which the target pixel belongs. Further, the prediction parameter storage means stores a prediction parameter for predicting the second image signal using the first image signal. Then, the prediction calculation means calculates the second image signal by the product-sum calculation between the prediction parameter read from the prediction parameter storage means according to the class information and the pixel of the first image signal around the position of the target pixel. Generate. Further, the prediction parameter storage means predicts a pixel value using a pixel of the first image signal used for the product-sum operation and a pixel of the third image signal obtained by reducing the resolution of the first image signal. The prediction parameter for each class generated so as to minimize the error of the value with respect to the true value is stored.
  In the image processing method according to claim 15,A predetermined number of pixels centered on a pixel of interest that is a pixel of interest among pixels of the first image signal, each of which is a predetermined number of pixels, and configured as a block composed of pixels of the first image signal.The first image signal is blocked, and class information representing a class is generated based on the pixels constituting the block to which the target pixel belongs. Then, the prediction parameter storage means for storing the prediction parameter for predicting the second image signal using the first image signal is located around the position of the prediction parameter and the target pixel read according to the class information. A second image signal is generated by a product-sum operation with the pixels of the first image signal. Further, the prediction parameter storage means uses the pixel of the first image signal used for the product-sum operation and the pixel of the third image signal obtained by reducing the resolution of the first image signal, A prediction parameter for each class generated so as to minimize an error of the predicted value with respect to the true value is stored.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below, but before that, in order to clarify the correspondence between the respective means of the invention described in the claims and the following embodiments, after each means, A corresponding embodiment (however, an example) is added in parentheses to describe the characteristics of the present invention, and the following is obtained.
[0010]
  That is, the image processing apparatus according to claim 1To form a first block of a predetermined size composed of a first pixel of interest that is a pixel of interest among the first pixels and a first pixel around the first pixel of interest,Block means for blocking the first image signal (for example, the class classification blocking circuit 1 shown in FIG. 1),First pixel of interestBelongs toFirst blockBecauseFirst pixel of interestOf a predetermined size including the position ofFirst blockRepresents a class based on the first pixel comprisingFirst class informationClass classification means (for example, the class classification circuit 4 shown in FIG. 1) and prediction parameter storage means for storing prediction parameters for predicting the second image signal using the first image signal ( For example, the coefficient ROM 61 shown in FIG.First class informationA prediction parameter read from the prediction parameter storage means according toFirst pixel of interestThe second pixel of the second image signal by the product-sum operation with the first pixel located aroundThe second pixel in the vicinity of the same position or corresponding position with respect to the position corresponding to the position of the first target pixel.Prediction calculation means (for example, a product-sum operation unit 62 shown in FIG. 8), and the prediction parameter storage means includes:A second block having a predetermined size including a second target pixel that is a pixel of interest in calculation of a prediction parameter among the first pixels and a first pixel around the second target pixel. As described above, the first image signal is blocked, and a second block to which the second pixel of interest belongs and which is a second block having a predetermined size including the position of the second pixel of interest is formed. Based on the first pixel, the second class information representing the class is generated, and is a position corresponding to the predicted position of the second pixel, from the same position as the first pixel or a position in between, A second direction that is proportional to the first distance in the same direction with respect to the direction to each position of the first pixel that is sum-of-products calculated for the prediction of the second pixel and the first distance therebetween. The first pixel in the similar positional relationship that is the position of the distance of There are, of predicted values of the pixel values, by performing learning to minimize differences between true values, the prediction parameter calculated for each class indicated by the second class informationIs stored.
  The image processing apparatus according to claim 4 generates a third image signal by thinning out pixels constituting the block at regular intervals, and supplies the third image signal to the class classification unit (for example, the thinning circuit 2 shown in FIG. 1). Etc.) and a reduction circuit (for example, reduction circuit 3 shown in FIG. 1) that generates a fourth image signal by removing pixels outside the block and supplies them to the class classification means. Is further provided.
  An image processing apparatus according to claim 5 is provided.To form a first block of a predetermined size composed of a first pixel of interest that is a pixel of interest among the first pixels and a first pixel around the first pixel of interest, Block means for blocking the first image signal (for example, the class classification blocking circuit 1 shown in FIG. 1) and the first block to which the first pixel of interest belongs, Class classification means (for example, a class classification circuit 4 shown in FIG. 1) that generates first class information representing a class based on a first pixel constituting a first block having a predetermined size including a position; Prediction parameter storage means for storing a prediction parameter for predicting the second image signal using the first image signal (for example, coefficient ROM 61 shown in FIG. 8) and prediction according to the first class information Read from parameter storage The second pixel of the second image signal corresponding to the position of the first target pixel is obtained by multiplying the prediction parameters that are output and the first pixel located around the first target pixel. Prediction calculation means (for example, a product-sum calculation unit 62 shown in FIG. 8) for predicting a second pixel around the same or corresponding position with respect to the position, and the position of the predicted second pixel The corresponding position, from the same position as or between the first pixel to the respective positions of the first pixel to be summed for prediction of the second pixel, and between them The true value of the predicted value of the pixel value using the first pixel having a similar positional relationship that is the position of the second distance in the same direction and proportional to the first distance with respect to the first distance By learning to minimize the error for Learning means for calculating the over data (e.g., learning such as a circuit 77 shown in FIG. 9), characterized in that it comprises a.
[0011]
  The image processing method according to claim 6 comprises:To form a first block of a predetermined size composed of a first pixel of interest that is a pixel of interest among the first pixels and a first pixel around the first pixel of interest,A block step of blocking the first image signal;First pixel of interestBelongs toFirst blockBecauseFirst pixel of interestOf a predetermined size including the position ofFirst blockRepresents a class based on the first pixel comprisingFirst class informationFrom a classification step for generating a prediction parameter storage means (for example, a coefficient ROM 61 shown in FIG. 8) for storing a prediction parameter for predicting a second image signal using the first image signal,First class informationPrediction parameters read out according toFirst pixel of interestThe second pixel of the second image signal by the product-sum operation with the first pixel located aroundThe second pixel in the vicinity of the same position or corresponding position with respect to the position corresponding to the position of the first target pixel.Predictive calculation step for predicting the prediction parameter storage means,A second block having a predetermined size including a second target pixel that is a pixel of interest in calculation of a prediction parameter among the first pixels and a first pixel around the second target pixel. As described above, the first image signal is blocked, and a second block to which the second pixel of interest belongs and which is a second block having a predetermined size including the position of the second pixel of interest is formed. Based on the first pixel, the second class information representing the class is generated, and is a position corresponding to the predicted position of the second pixel, from the same position as the first pixel or a position in between, A second direction that is proportional to the first distance in the same direction with respect to the direction to each position of the first pixel that is sum-of-products calculated for the prediction of the second pixel and the first distance therebetween. The first pixel in the similar positional relationship that is the position of the distance of There are, of predicted values of the pixel values, by performing learning to minimize differences between true values, the prediction parameter calculated for each class indicated by the second class informationIs stored.
[0012]
  The parameter generation device according to claim 7,In order to form a block of a predetermined size composed of a target pixel that is a target pixel of the first pixels and a first pixel around the target pixel,Block means for blocking the first image signal (for example, the class classification blocking circuit 1 shown in FIG. 13) and a block to which the pixel of interest belongs and having a predetermined size including the position of the pixel of interest Class classifying means (for example, class classifying circuit 4 shown in FIG. 13) that generates class information representing a class based on the first pixel that constitutes the position of the second pixel to be predictedIs the same position as the first pixel or a position in betweenTo the direction from the first pixel to the position where the first pixel is summed for prediction of the second pixel and the first distance therebetween, the same direction and proportional to the first distance Prediction parameters for each class are calculated by performing learning to minimize the error of the predicted value of the pixel value with respect to the true value using the first pixel having a similar positional relationship that is the position of the second distance. Learning means (for example, a learning circuit 94 shown in FIG. 13).
  The parameter generation device according to claim 12 generates a third image signal by thinning out pixels constituting a block at a constant interval, and supplies the third image signal to the class classification means (for example, a thinning circuit 2 shown in FIG. 13). And a reduction circuit (for example, reduction circuit 3 shown in FIG. 13) that generates a fourth image signal by removing an outer pixel from the pixels constituting the block and supplies the fourth image signal to the class classification unit. Is further provided.
  The parameter generation method according to claim 13 is:In order to form a block of a predetermined size composed of a target pixel that is a target pixel of the first pixels and a first pixel around the target pixel,A block step for blocking the first image signal, and class information representing a class based on a first pixel constituting a block of a predetermined size including a position of the pixel of interest to which the pixel of interest belongs. Class classification step to generate and predicted second pixel locationIs the same position as the first pixel or a position in betweenTo the direction from the first pixel to the position where the first pixel is summed for prediction of the second pixel and the first distance therebetween, the same direction and proportional to the first distance Prediction parameters for each class are calculated by performing learning to minimize the error of the predicted value of the pixel value with respect to the true value using the first pixel having a similar positional relationship that is the position of the second distance. And a learning step.
[0013]
  An image processing apparatus according to claim 14 is provided.A predetermined number of pixels centered on a pixel of interest that is a pixel of interest among pixels of the first image signal, each of which is a predetermined number of pixels, and configured as a block composed of pixels of the first image signal.A class for generating class information representing a class based on block means for blocking the first image signal (for example, the class classification blocking circuit 1 shown in FIG. 1) and the pixels constituting the block to which the target pixel belongs. A classification unit (for example, the class classification circuit 4 shown in FIG. 1) and a prediction parameter storage unit (for example, FIG. 8) that stores a prediction parameter for predicting the second image signal using the first image signal. The second image signal is obtained by multiplying the prediction parameter read out from the prediction parameter storage unit according to the class information and the pixel of the first image signal around the position of the target pixel. For example, a product-sum operation unit 62 shown in FIG. 8, and the prediction parameter storage unit includes a first image signal used for product-sum operation. And a pixel of the third image signal obtained by reducing the resolution of the first image signal for each class generated so as to minimize the error of the predicted value of the pixel value with respect to the true value. Prediction parameters are stored.
  The image processing method according to claim 15 comprises:A predetermined number of pixels centered on a pixel of interest that is a pixel of interest among pixels of the first image signal, each of which is a predetermined number of pixels, and configured as a block composed of pixels of the first image signal.A block step for blocking the first image signal, a class classification step for generating class information representing a class based on the pixels constituting the block to which the pixel of interest belongs, and a second image using the first image signal The first image in the vicinity of the position of the prediction parameter and the target pixel read out according to the class information from the prediction parameter storage means (for example, the coefficient ROM 61 shown in FIG. 8) that stores the prediction parameter for predicting the signal A prediction calculation step of generating a second image signal by a product-sum operation with the pixel of the signal, and the prediction parameter storage means includes a first image signal pixel used for the product-sum operation, For each class generated so as to minimize the error of the predicted value of the pixel value with respect to the true value using the pixel of the third image signal whose image signal is reduced in resolution Wherein the prediction parameters are stored.
[0014]
Of course, this description does not mean that the respective means are limited to those described above.
[0015]
FIG. 1 shows the configuration of an embodiment of an image conversion apparatus to which the present invention is applied. This image conversion apparatus converts an SD image (first image) into an HD image (second image). That is, for example, in FIG. 2, the portion indicated by the symbol • is a pixel constituting the HD image (hereinafter referred to as “HD pixel” as appropriate), and the portion indicated by a circle in FIG. When a pixel (hereinafter, appropriately referred to as an SD pixel) is used, the image conversion apparatus in FIG. 1 converts an SD image indicated by a circle in FIG. 1 to an HD image indicated by a mark in FIG. .
[0016]
For example, an image of an SD image transmitted through a transmission line such as a ground line, a satellite line, or a CATV (CAble TeleVision) network, or reproduced from a recording medium such as an optical disk, a magneto-optical disk, or a magnetic tape. The signal is supplied to the class classification blocking circuit 1 and the prediction value calculation blocking circuit 5.
[0017]
The class classification blocking circuit 1 configures a class classification block including a predetermined target pixel from the SD image supplied thereto. That is, the class classification blocking circuit 1 includes, for example, a class classification block composed of 5 × 5 (horizontal × vertical) SD pixels centered on the target pixel, as shown by a solid line in FIG. Constitute.
[0018]
Here, 5 × 5 SD pixels (portions indicated by ◯ in FIG. 2) constituting the class classification block will be appropriately expressed as follows. That is, the SD pixel located at the i-th position from the left and the j-th position from the top in the class classification block is represented by B_ijIs written. Therefore, in the embodiment of FIG. 2, the classification block is the SD pixel B.₃₃Is configured as a pixel of interest. In addition, HD pixels generated from the SD pixels constituting the class classification block (portions indicated by marks in FIG. 2) are hereinafter referred to as “A” as appropriate._ijIs written.
[0019]
When the class classification block forming circuit 1 forms a class classification block, it outputs it to the thinning circuit 2 and the reduction circuit 3.
[0020]
When receiving the class classification block, the thinning circuit 2 reduces the number of pixels by, for example, thinning out the SD pixels constituting the class classification block, thereby forming a thinning block. That is, the thinning circuit 2 constitutes a thinning block (compressed image) by thinning out the class classification block by, for example, 1/2 in both the horizontal direction and the vertical direction. This thinning block is supplied to the class classification circuit 4.
[0021]
Here, the pixels in the class classification block remaining as a result of the thinning process by the thinning circuit 2 (portions indicated by the dotted circles in FIG. 2), that is, the pixels constituting the thinning block are appropriately thinned out as follows. The SD pixel is referred to as an SD pixel._ijIs written.
[0022]
On the other hand, in the reduction circuit 3, a reduced block is formed by reducing the class classification block around the target pixel. That is, the reduction circuit 3 removes the surrounding pixels from the 5 × 5 pixel class classification block, for example, the target pixel B₃₃A reduced block (compressed image) made up of 3 × 3 pixels centering on is formed. This reduced block is also supplied to the class classification circuit 4 in the same manner as the thinning block.
[0023]
Here, both the thinning block and the reduced block are generated by reducing the number of pixels of the class classification block, and reducing the number of pixels reduces the amount of information. In a sense, the thinned block and the reduced block can be referred to as a compressed image.
[0024]
When the class classification circuit 4 receives the thinned block and the reduced block, the class classification circuit 4 performs class classification using the two blocks, and the prediction value calculation blocking circuit 5 described later outputs an index representing the resulting class. The prediction value calculation block 6 is supplied to the prediction circuit 6 as being for the predicted value calculation block class.
[0025]
The prediction circuit 6 is also supplied with a prediction value calculation block from the prediction value calculation blocking circuit 5. In the predicted value calculation blocking circuit 5, for example, the target pixel B as surrounded by a dotted-line rectangle in FIG. 2.₃₃A prediction value calculation block of 3 × 3 pixels centering on is configured and supplied to the prediction circuit 6.
[0026]
Therefore, in the present embodiment, the prediction value calculation block and the reduced block are configured by the same SD pixel, but the SD pixels configuring both are not necessarily the same. That is, as described above, the reduced block only needs to be obtained by reducing the class classification block around the pixel of interest, and the predicted value calculation block is configured so that the feature is included in the class classification block. (To be precise, the prediction value calculation block may be configured in any way, and the class classification block needs to be configured to include the characteristics of the prediction value calculation block).
[0027]
When the prediction circuit 6 receives a prediction value calculation block and an index corresponding to the class, the prediction circuit 6 has a prediction coefficient as will be described later, which corresponds to the received index, and the SD constituting the prediction value calculation block. An adaptive process for obtaining a predicted value of the pixel value of the HD pixel is performed by linear combination with the pixel value of the pixel. That is, the prediction circuit 6 includes the prediction coefficient corresponding to the index and the SD pixel B constituting the prediction value calculation block._{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄For example, the target pixel B₃₃HD pixel A in a 3 × 3 range centered on₄₃, A₄₄, A₄₅, A₅₃, A₅₄, A₅₅, A₆₃, A₆₄, A₆₅Find the predicted value of.
[0028]
In the prediction circuit 6, the same processing is performed for the pixel B below.₃₃SD pixels other than are sequentially performed as the target pixel, and when the predicted values of all HD pixels constituting the HD image are obtained, the predicted values are supplied to the monitor 7. The monitor 7 includes, for example, a D / A converter, D / A converts the pixel value as a digital signal from the prediction circuit 6, and displays an image obtained as a result of the D / A conversion.
[0029]
Next, FIG. 3 shows a configuration example of the class classification circuit 4 of FIG.
[0030]
The decimation block from the decimation circuit 2 is supplied to the first class generation circuit 11, and the demagnification block from the demagnification circuit 3 (as described above, here, the predicted value calculation block surrounded by the dotted-line square in FIG. Is supplied to both the first class generation circuit 11 and the second class generation circuit 12.
[0031]
The first class generation circuit 11 calculates the self-similarity of the class classification block (similarity between the reduced block and the thinned block) based on the reduced block and the thinned block, and based on the self-similarity. Thus, the first class information for classifying the prediction value calculation block is output.
[0032]
The first class information obtained based on this self-similarity is supplied to the second class generation circuit 12 and the final class determination circuit 13.
[0033]
The second class generation circuit 12 recognizes the direction having the highest self-similarity based on the first class information from the first class information generation circuit 11. Further, the second class generation circuit 12 detects a pixel value pattern of pixels arranged in the direction having the highest self-similarity among the pixels constituting the reduced block, and determines the second class information corresponding to the pattern as the final class information. Output to the class decision circuit 13. The final class determination circuit 13 classifies the prediction value calculation blocks based on the first class information or the second class information from the first class generation circuit 11 or the second class generation circuit 12, respectively, and corresponds to the class. The index to be output is output to the prediction circuit 6 (FIG. 1).
[0034]
Next, FIG. 4 shows a configuration example of the first class generation circuit 11 of FIG.
[0035]
The thinning block and the reduced block are similarities calculating unit 21.₁Thru 21_FourTo be supplied. Similarity calculation unit 21₁Consists of pixel extraction units 26B and 26C and a norm calculation unit 27, and the reduced block or thinning block is supplied to the pixel extraction unit 26B or 26C, respectively.
[0036]
The pixel extraction unit 26 </ b> B is an SD pixel (FIG. 2) B that forms a reduced block._{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄Of the pixel of interest B₃₃Those that are on a straight line in a predetermined direction passing through are extracted. In other words, the pixel extraction unit 26B₃₃SD pixel B passing through, for example, on a straight line in the vertical direction_{twenty three}, B₃₃, B₄₃To extract. This extracted SD pixel B_{twenty three}, B₃₃, B₄₃Is output to the norm calculation unit 27.
[0037]
On the other hand, the pixel extraction unit 26C performs the thinning SD pixel (FIG. 2) C constituting the thinning block.₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty two}, C_{twenty three}, C₃₁, C₃₂, C₃₃Of the pixel of interest B₃₃Thinned SD pixel C corresponding to_{twenty two}That are on the straight line in the same direction as in the pixel extraction unit 26B. Therefore, in this case, the pixel extraction unit 26B performs the thinning SD pixel C.₁₂, C_{twenty two}, C₃₂Is extracted. This extracted thinned SD pixel C₁₂, C_{twenty two}, C₃₂Is also output to the norm calculation unit 27.
[0038]
The norm calculation unit 27 calculates a norm between vectors each having the output of the pixel extraction unit 26B or 26C as a component, and the calculation result is sent to the maximum similarity direction determination unit 22 as self-similarity in the vertical direction. Output.
[0039]
Similarity calculation unit 21₂Thru 22_FourAlso in the similarity calculation unit 21₁In the same manner as in, the vector norm is calculated using pixels passing through the pixel of interest on the straight line in the other direction, and is output to the maximum similarity direction determination unit 22 as self-similarity in that direction. . That is, the similarity calculation unit 21₂Thru 22_FourThen, for example, the self-similarity in each of the horizontal direction, the upper left diagonal direction (the lower right diagonal direction), or the upper right diagonal direction (the lower left diagonal direction) is obtained and output to the maximum similarity direction determining unit 22.
[0040]
  The maximum similarity direction determination unit 22 determines the direction with the highest self-similarity based on the outputs of the similarity calculation units 211 to 214.TheJudgmentYouThe That is, in this case, since the self-similarity is given by the norm of the vector, the maximum similarity direction determining unit 22 detects the one having the smallest value and determines the direction in which the norm is obtained as the self-similarity. The direction with the highest similarity is determined. When the maximum similarity direction determination unit 22 obtains the direction having the highest self-similarity in this way, the maximum similarity direction determination unit 22 outputs, for example, a 2-bit code corresponding to the direction as the first class information. That is, when the direction with the highest self-similarity is the horizontal direction, the vertical direction, the left diagonally upward direction, or the diagonally right upward direction, the maximum similarity direction determination unit 22 sets, for example, “00 ”,“ 01 ”,“ 10 ”,“ 11 ”, or the like.
[0041]
In the above-described case, the norm of the vector having the pixel value as a component is used as the evaluation amount of the self-similarity in each direction. It is also possible to use it as an evaluation amount.
[0042]
That is, for example, the SD pixel B constituting the reduced block_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄To the thinned-out SD pixel C that does not exist in the reduced block.₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty three}, C₃₁, C₃₂, C₃₃Is obtained by extrapolation, and the extrapolation result and true decimation SD pixel C₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty three}, C₃₁, C₃₂, C₃₃The error from the pixel value of is calculated. Then, this error can be used as the evaluation amount of similarity, and the direction with the smallest error can be set as the direction of the highest similarity.
[0043]
In addition, for example, the thinned SD pixels constituting the thinned block (FIG. 2) C₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty two}, C_{twenty three}, C₃₁, C₃₂, C₃₃SC pixel B of the reduced block that does not exist in the thinned block_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₄, B₄₂, B₄₃, B₄₄Is obtained by interpolation, and the interpolation result and the true SD pixel B_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₄, B₄₂, B₄₃, B₄₄The error from the pixel value of is calculated. Then, this error can be used as an evaluation amount of similarity, and the direction with the smallest error can be set as the direction of the highest similarity.
[0044]
Next, FIG. 5 shows a configuration example of the second class generation circuit 12 of FIG.
[0045]
The reduced block from the reduction circuit 3 and the first class information from the first class generation circuit 11 are supplied to the maximum similarity direction pixel extraction unit 31. The maximum similarity direction pixel extraction unit 31 recognizes the direction having the maximum similarity based on the first class information, and from the reduced block, the target pixel B₃₃SD pixels arranged on a straight line in the direction passing through (hereinafter referred to as the maximum similarity pixel as appropriate) (in this embodiment, SD pixels B constituting a reduced block)_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄Are extracted in the horizontal direction, vertical direction, left diagonally upward direction, or right diagonally upward direction). The three maximum similarity pixels are supplied to the pattern classification unit 33 via the ADRC processing unit 32. The pattern classification unit 33 detects a pattern of pixel values of the three maximum similarity pixels supplied via the ADRC processing unit 32, and outputs second class information corresponding to the pattern.
[0046]
Here, normally, for example, about 8 bits are assigned to each pixel in order to express the pixel value. Further, in the present embodiment, as described above, three maximum similarity pixels are obtained. Therefore, if the pixel value pattern is divided into 3 pixels each represented by 8 bits, (2⁸)^ThreeThis results in a huge number of patterns. Therefore, the number of classes is enormous, and processing is complicated when processing is performed in correspondence with such an enormous number of classes.
[0047]
Therefore, in the present embodiment, the ADRC processing unit 32 performs ADRC (Adaptive Dynamic Range Coding) processing on the three maximum similarity pixels, whereby the maximum similarity pixel. By reducing the number of bits, the number of pixel value patterns is reduced.
[0048]
That is, in the ADRC process, as shown in FIG. 6A, the maximum value MAX and the minimum value MIN of the pixel value are detected from the three maximum similarity pixels arranged on a certain straight line. Then, DR = MAX−MIN is set as the local dynamic range of the maximum similarity pixel, and based on the dynamic range DR, the pixel values of the three maximum similarity pixels are requantized to K bits.
[0049]
Specifically, the minimum value MIN is subtracted from each pixel value of the three maximum similarity pixels, and the subtraction value is DR / 2.^KDivide by. Then, it is converted into a code (ADRC code) corresponding to the division value obtained as a result. That is, for example, when K = 2, as shown in FIG. 6B, the division value has a dynamic range DR of 4 (= 2²) It is determined which range is obtained by equally dividing, and the division value is the range of the lowest level, the range of the second level from the bottom, the range of the third level from the bottom, or the top In the case of belonging to the level range, for example, each bit is encoded into 2 bits such as 00B, 01B, 10B, or 11B (B represents a binary number).
[0050]
When decoding based on pixel values, the ADRC code 00B, 01B, 10B, or 11B is the center value L of the lowest level range obtained by dividing the dynamic range DR into four equal parts.₀₀, Center value L of the second level range from the bottom₀₁, Center value L of the third level range from the bottom_TenOr the center value L of the range of the highest level₁₁And the minimum value MIN is added to the value.
[0051]
Here, the details of the ADRC processing as described above are disclosed in, for example, Japanese Patent Application Laid-Open No. 3-53778 filed by the applicant of the present application.
[0052]
As described above, the number of patterns can be reduced by performing ADRC processing that performs requantization with a smaller number of bits than the number of bits allocated to the SD pixel. This is performed in the section 32.
[0053]
That is, the three maximum similarity pixels output from the maximum similarity direction pixel extraction unit 31 are supplied to the maximum value detection unit 41, the minimum value detection unit 42, and the delay unit 43 of the ADRC processing unit 32. In the maximum value detection unit 41 or the minimum value detection unit 42, the maximum value MAX or the minimum value MIN of the pixel value is detected from the three maximum similarity pixels, and both are supplied to the calculator 44. The calculator 44 calculates the difference between the maximum value MAX and the minimum value MIN, that is, the dynamic range DR (= MAX−MIN), and supplies it to the ADRC code determination unit 45. Further, the minimum value MIN output from the minimum value detection unit 42 is also supplied to the ADRC code determination unit 45.
[0054]
On the other hand, in the delay unit 43, the maximum similarity pixel is delayed by the time required for processing in the maximum value detection unit 41 (or minimum value detection unit 42) and the computing unit 45 and supplied to the ADRC code determination unit 45. In the ADRC code determination unit 45, the minimum value MIN is subtracted from each of the pixel values of the three maximum similarity pixels, and each subtraction value is requantized to, for example, 1 bit based on the dynamic range DR. Then, a 1-bit ADRC code for each of the three maximum similarity pixels obtained as a result, that is, a 3-bit ADRC code in total is output as the second class information.
[0055]
Next, FIG. 7 is a block diagram showing a configuration example of the final class determination circuit 13 of FIG.
[0056]
The first class information and the second class information are supplied to the address terminal AD of the ROM 51. For example, the ROM 51 stores a value having the first class information as the upper bits and the second class information as the lower bits at the address indicated by both the first class information and the second class information. Then, when the first class information and the second class information are given to the address terminal AD, the ROM 51 reads the stored value of the address indicated by both the first class information and the second class information, and stores the stored value. Is output as an index indicating the class of the predicted value calculation blocking circuit 5. Therefore, in this case, 5-bit data in which 3-bit second class information is added after 2-bit first class information is output as an index.
[0057]
As described above, since classification is performed not only for pixel value level patterns but also for self-similarity of images, images can be processed more appropriately according to their properties. It becomes possible.
[0058]
The index may be 6 bits in total by adding 1 bit indicating whether the unit of processing is a frame or a field to the above 5 bits.
[0059]
Further, the ROM 51 can store the same index at several different addresses, thereby reducing the number of bits of the index. That is, according to the first class information of 2 bits and the second class information of 3 bits, 5 bits, that is, 32 (= 2^FiveHowever, there are cases in which there are no problems even if the prediction circuit 6 uses the same prediction coefficient to obtain the prediction value of the pixel value. . Thus, a plurality of classes that can use the same prediction coefficient is treated as one class, and the number of classes can be reduced by doing so.
[0060]
Further, in the above-described case, the pattern classification unit 33 in FIG. 5 performs pattern classification based on the ADRC code. However, for example, the pattern classification may be DPCM (predictive coding), BTC ( It is also possible to perform the processing on data subjected to Block Truncation Coding), VQ (Vector Quantization), DCT (Discrete Cosine Transform), Hadamard Transform, and the like.
[0061]
Next, FIG. 8 shows a configuration example of the prediction circuit 6 of FIG.
[0062]
A coefficient ROM (Read Only Memory) 61 stores prediction coefficients for each class obtained in advance by learning (described later), receives an index output from the class classification circuit 4, and sets an address corresponding to the index. The stored prediction coefficient, that is, the prediction coefficient corresponding to the class of the prediction value calculation block output by the prediction value calculation block forming circuit 5 is read and output to the product-sum calculator 62.
[0063]
The product-sum calculator 62 is supplied with a prediction value calculation block from the prediction value calculation blocking circuit 5 in addition to the prediction coefficient. And a prediction coefficient corresponding to the class, a linear linear expression shown in Equation (1) described below (specifically, for example, Equation (8)) is calculated (product-sum operation is performed), Thereby, an adaptive process for calculating the predicted value of the pixel value of the HD pixel is performed. That is, for example, as shown in FIG. 2 described above, the SD pixel B constituting the prediction value calculation block_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄Pixel B₃₃HD pixel A in a 3 × 3 range centered on₄₃, A₄₄, A₄₅, A₅₃, A₅₄, A₅₅, A₆₃, A₆₄, A₆₅Is predicted. The predicted value of the HD pixel obtained in the product-sum calculator 62 is supplied to the limiter 63, where the value is limited so as not to exceed the dynamic range in the D / A converter built in the monitor 7. , Supplied to the monitor 7.
[0064]
Here, the adaptive processing will be described in detail.
[0065]
For example, now, the predicted value E [y] of the pixel value y of the HD pixel is converted into a pixel value of some SD pixels (hereinafter referred to as learning data as appropriate) x.₁, X₂, ... and a predetermined prediction coefficient w₁, W₂Consider a linear primary combination model defined by the linear combination of. In this case, the predicted value E [y] can be expressed by the following equation.
[0066]

[0067]
Therefore, in order to generalize, a matrix W composed of a set of prediction coefficients w, a matrix X composed of a set of learning data, and a matrix Y ′ composed of a set of predicted values E [y],
[Expression 1]

Then, the following observation equation holds.
[0068]

[0069]
Then, it is considered to apply the least square method to this observation equation to obtain a predicted value E [y] close to the pixel value y of the HD pixel. In this case, a matrix Y composed of a set of pixel values of HD pixels (hereinafter referred to as teacher data as appropriate) y and a matrix E composed of a set of residuals e of predicted values E [y] for the pixel values y of HD pixels. ,
[Expression 2]

From the equation (2), the following residual equation is established.
[0070]

[0071]
In this case, the prediction coefficient w for obtaining the predicted value E [y] close to the pixel value y of the HD pixel_iIs the square error
[Equation 3]

Can be obtained by minimizing.
[0072]
Therefore, the above square error is converted into the prediction coefficient w._iWhen the value differentiated by 0 is 0, that is, the prediction coefficient w satisfying the following equation:_iHowever, this is the optimum value for obtaining the predicted value E [y] close to the pixel value y of the HD pixel.
[0073]
[Expression 4]

[0074]
Therefore, first, Equation (3) is converted into the prediction coefficient w._iIs differentiated by the following equation.
[0075]
[Equation 5]

[0076]
From equations (4) and (5), equation (6) is obtained.
[0077]
[Formula 6]

[0078]
Further, considering the relationship among the learning data x, the prediction coefficient w, the teacher data y, and the residual e in the residual equation of Equation (3), the following normal equation can be obtained from Equation (6). .
[0079]
[Expression 7]

[0080]
The normal equation of the equation (7) can be formed by the same number as the number of prediction coefficients w to be obtained. Therefore, by solving the equation (7) (however, to solve the equation (7), the equation (7) 7), the matrix composed of the coefficients related to the prediction coefficient w needs to be regular), and the optimal prediction coefficient w can be obtained. In solving equation (7), for example, a sweep-out method (Gauss-Jordan elimination method) or the like can be applied.
[0081]
As described above, the optimum prediction coefficient w is obtained, and further, using the prediction coefficient w, it is adaptive to obtain the prediction value E [y] close to the pixel value y of the HD pixel by the equation (1). This adaptive processing is performed in the product-sum operation unit 62.
[0082]
Note that the adaptive processing is different from the interpolation processing in that a component included in the HD image that is not included in the SD image is reproduced. In other words, the adaptive process is the same as the interpolation process using a so-called interpolation filter as long as only Expression (1) is seen, but the prediction coefficient w corresponding to the tap coefficient of the interpolation filter uses the teacher data y. In other words, since it is obtained by learning, the components included in the HD image can be reproduced. That is, a high-resolution image can be easily obtained. From this, it can be said that the adaptive process is a process having an image creating action.
[0083]
The adaptive processing described above is based on the prediction coefficient w obtained by learning using the pixel value y of the HD pixel as teacher data.₁, W₂,..., And such an adaptive process (hereinafter referred to as a first adaptive process as appropriate) has been proposed by the present applicant.
[0084]
By the way, according to the first adaptive processing, the prediction coefficient w₁, W₂,..., That is, in learning, HD images are required as teacher data. That is, for example, in FIG._{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄To HD pixel A₄₃, A₄₄, A₄₅, A₅₃, A₅₄, A₅₅, A₆₃, A₆₄, A₆₅When the predicted value of is determined by the first adaptive processing, the prediction coefficient w for that₁, W₂,... Are calculated using HD pixel A₄₃, A₄₄, A₄₅, A₅₃, A₅₄, A₅₅, A₆₃, A₆₄, A₆₅Are required.
[0085]
However, even if there is no HD image, the prediction coefficient w₁, W₂If it is required, it is convenient.
[0086]
Therefore, using the self-similarity of the image, for example, the prediction coefficient w is calculated from only the SD pixel as follows.₁, W₂,... Can be used by the prediction circuit 6 to perform adaptive processing (this adaptive processing is hereinafter referred to as second adaptive processing as appropriate).
[0087]
That is, in FIG. 2, the SD pixel B constituting the prediction value calculation block_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄HD pixel A₄₃, A₄₄, A₄₅, A₅₃, A₅₄, A₅₅, A₆₃, A₆₄, A₆₅When attention is paid to the positional relationship between and the SD pixel C₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty two}, C_{twenty three}, C₃₁, C₃₂, C₃₃SD pixel B_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄It is similar to the positional relationship between the two.
[0088]
Therefore, from the self-similarity of the image, the thinned SD pixel C₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty two}, C_{twenty three}, C₃₁, C₃₂, C₃₃As learning data and SD pixel B_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄Is used as teacher data, so that the prediction coefficient w₁, W₂, ..., and the prediction coefficient w₁, W₂,... Are SD pixels B constituting the prediction value calculation block._{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄To HD pixel A₄₃, A₄₄, A₄₅, A₅₃, A₅₄, A₅₅, A₆₃, A₆₄, A₆₅Can be used to predict the predicted value.
[0089]
FIG. 9 shows a configuration example of an image processing apparatus that performs a learning process for obtaining a prediction coefficient for performing the second adaptive process.
[0090]
An SD image is supplied to the class classification blocking circuit 71, the learning blocking circuit 75, and the teacher blocking circuit 76.
[0091]
In the class classification blocking circuit 71, the thinning circuit 72, the reduction circuit 73, or the class classification circuit 74, the class classification blocking circuit 1, the thinning circuit 2, the reduction circuit 3, or the class classification circuit 4 in FIG. The same processing is performed for each, whereby an index corresponding to a class of learning blocks output from a learning blocking circuit 75 described later is supplied to the learning circuit 77.
[0092]
On the other hand, as described above, the learning blocking circuit 75 uses the SD pixels (decimated SD pixels) C as learning data from the SD image.₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty two}, C_{twenty three}, C₃₁, C₃₂, C₃₃A block larger than the prediction value calculation block is configured, and is output to the learning circuit 77 as a learning block. At the same time, in the teacher blocking circuit 76, an SD pixel B as teacher data is obtained from the SD image._{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄This is also output to the learning circuit 77 as a teacher block.
[0093]
In the learning circuit 77, the SD pixel constituting the learning block is used as learning data, and the prediction coefficient that minimizes the error is calculated by, for example, the least square method using the SD pixel constituting the teacher block as the teacher data. The
[0094]
That is, for example, the pixel value of the SD pixel (decimation SD pixel) constituting the learning block is set to x₁, X₂, X_Three, ..., and the prediction coefficient to be obtained is w₁, W₂, W_Three,..., In order to obtain the pixel value y of a certain SD pixel constituting the teacher block by these linear linear combinations, the prediction coefficient w₁, W₂, W_Three,... Must satisfy the following equation.
[0095]
y = w₁x₁+ W₂x₂+ W_Threex_Three+ ...
[0096]
Therefore, in the learning circuit 77, the predicted value w for the true value y is determined from the learning block and the teacher block.₁x₁+ W₂x₂+ W_Threex_ThreePrediction coefficient w that minimizes square error of + ...₁, W₂, W_Three,... Are obtained by building and solving the normal equation shown in the above equation (7).
[0097]
The prediction coefficient obtained by the learning circuit 77 is output as the prediction coefficient of the class corresponding to the index from the class classification circuit 74.
[0098]
That is, in the present embodiment, SD pixels (decimated SD pixels) C constituting the learning block₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty two}, C_{twenty three}, C₃₁, C₃₂, C₃₃To 9 SD pixels B constituting the teacher block_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄It is necessary to calculate a prediction coefficient for obtaining.
[0099]
For this reason, in the learning circuit 77, the SD pixel B for the class CL corresponding to the index output from the class classification circuit 74 is displayed._{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄Each is used as teacher data, and SD pixel (decimated SD pixel) C₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty two}, C_{twenty three}, C₃₁, C₃₂, C₃₃As a learning data, the normal equation shown in the equation (7) is established.
[0100]
Further, in the learning circuit 77, a normal equation is similarly established for other learning blocks classified into the class CL, and the SD pixel B_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄Each predicted value E [B_{twenty two}], E [B_{twenty three}], E [B_{twenty four}], E [B₃₂], E [B₃₃], E [B₃₄], E [B₄₂], E [B₄₃], E [B₄₄] Prediction coefficient w for obtaining₁(B_{twenty two}) To w₉(B_{twenty two}), W₁(B_{twenty three}) To w₉(B_{twenty three}), W₁(B_{twenty four}) To w₉(B_{twenty four}), W₁(B₃₂) To w₉(B₃₂), W₁(B₃₃) To w₉(B₃₃), W₁(B₃₄) To w₉(B₃₄), W₁(B₄₂) To₉(B₄₂), W₁(B₄₃) To w₉(B₄₃), W₁(B₄₄) To w₉(B₄₄) (In this embodiment, nine learning data are used to obtain one prediction value, and accordingly nine prediction coefficients w are required correspondingly). Once the equation is obtained (and thus the normal equation is repeated in the learning circuit 77 until such a number of normal equations is obtained), the SD pixel B for class CL is solved by solving the normal equation._{3 + m, 3 + n}Predicted value E [B_{3 + m, 3 + n}] Is the optimal prediction coefficient w₁(B_{3 + m, 3 + n}) To w₉(B_{3 + m, 3 + n}) Is calculated (where m = -1, 0, +1, n = -1, 0, +1).
[0101]
The coefficient ROM 61 (FIG. 8) constituting the prediction circuit 6 of FIG. 1 can store the prediction coefficient output from the learning circuit 77 as described above. , HD pixel A in the prediction value calculation block according to the following equation corresponding to equation (1)₄₃, A₄₄, A₄₅, A₅₃, A₅₄, A₅₅, A₆₃, A₆₄, A₆₅Each predicted value E [A₄₃], E [A₄₄], E [A₄₅], E [A₅₃], E [A₅₄], E [A₅₅], E [A₆₃], E [A₆₄], E [A₆₅] Is required.
[0102]

[0103]
As described above, the SD pixel B_{twenty two}, B_{twenty three}, B_{twenty four}, B₃₂, B₃₃, B₃₄, B₄₂, B₄₃, B₄₄Each pixel value (predicted value) is reduced to the thinned SD pixel C.₁₁, C₁₂, C₁₃, C_{twenty one}, C_{twenty two}, C_{twenty three}, C₃₁, C₃₂, C₃₃The prediction coefficient is obtained by performing learning so that the prediction coefficient can be calculated from a predetermined learning block (predetermined block) configured by using the prediction coefficient and smaller than the learning block (the target pixel B₃₃According to the case where adaptive processing is applied to a prediction value calculation block (small block), which is a reduced learning block centered on the image, the HD pixel A is determined by the self-similarity of the image.₄₃, A₄₄, A₄₅, A₅₃, A₅₄, A₅₅, A₆₃, A₆₄, A₆₅An appropriate prediction value for can be obtained.
[0104]
Next, FIG. 10 shows a configuration example of the learning circuit 77 of FIG.
[0105]
The multiplication circuit 81 has learning data x constituting a learning block.₁, X₂, ..., x_mAnd teacher data y constituting the teacher block are input, where the learning data x to be subject to the summation (Σ) in the normal equation of Equation (7).₁, X₂, ..., x_mProduct between and learning data x₁, X₂, ..., x_mThe product of each and the teacher data y is obtained and supplied to the adding circuit 82.
[0106]
The adder circuit 82 is supplied with the output of the decoder 83 in addition to the output of the multiplier circuit 81. An index is supplied from the class classification circuit 74 (FIG. 9) to the decoder 83, and the decoder 83 uses the learning data x based on the index.₁, X₂, ..., x_mAre recognized, and the recognition result is output to the adder circuit 82.
[0107]
The adder circuit 82 uses the output of the multiplier circuit 81 to perform an operation corresponding to the summation in the normal equation of Expression (7) independently for each class from the decoder 83, and outputs the operation result to the operation circuit 84. To supply. The arithmetic circuit 84 uses the output of the adder circuit 82 to perform an operation based on the sweep-out method, thereby calculating and outputting a prediction coefficient.
[0108]
Next, FIG. 11 shows a configuration example of the multiplication circuit 81 of FIG.
[0109]
As shown in the figure, the multiplier circuit 81 is constituted by a multiplier array (multipliers are arranged in a predetermined shape). In other words, the multiplication circuit 81 includes a coefficient of the prediction coefficient w (summation portion) on the left side of the normal equation of Expression (7) and a multiplier corresponding to each term on the right side.
[0110]
In addition, in the embodiment of FIG. 11, since a matrix composed of coefficients related to the prediction coefficient w on the left side of the normal equation of Expression (7) (hereinafter referred to as a coefficient matrix as appropriate) and its transpose matrix are equal. Only the multiplier corresponding to the right-hand side term of Equation (7) and the right upper part component including the diagonal component of the coefficient matrix are provided.
[0111]
In the multiplier circuit 81 configured as described above, in each multiplier, as described above, the learning data x to be subject to the summation (Σ) in the normal equation of the equation (7)₁, X₂, ..., x_mProduct between and learning data x₁, X₂, ..., x_mThe product of each and the teacher data y is obtained and supplied to the adding circuit 82.
[0112]
Next, FIG. 12 shows a configuration example of the adder circuit 82 of FIG.
[0113]
The adder circuit 82 is composed of an adder array or a memory array (register array) in which adders or memory cells are respectively arranged, like the multiplier constituting the multiplier circuit 81 of FIG. Note that only one adder array is provided in the same manner as the multiplier array, but as many memory arrays as the number corresponding to the class are provided.
[0114]
In the adder circuit 82 configured as described above, the output of the corresponding multiplier of the multiplier array is supplied to each adder constituting the adder array. Further, each adder is supplied with the stored value of the corresponding memory cell constituting the memory array corresponding to the class from the decoder 83. In each adder, the outputs of the multiplier and the memory cell are added, and the addition result is supplied to the original memory cell. Then, in each memory cell, the addition result supplied from the adder is stored, and the adder circuit 82 performs the operation corresponding to the summation in the normal equation of Expression (7) by repeating the same processing thereafter. .
[0115]
As a result, each memory array stores the coefficient of each term of the normal equation for the corresponding class.
[0116]
Then, in the arithmetic circuit 84 of FIG. 10, the prediction coefficient for each class is obtained by the sweep-out method using the stored value of each memory cell of the memory array.
[0117]
By the way, for example, a receiving apparatus that receives an SD image conforming to the NTSC system or the like is incorporated with the image conversion apparatus of FIG. 1, and the prediction circuit 6 obtains a predicted value of the HD pixel by the first adaptive processing. In this case, as described above, since HD pixels are necessary as teacher data in learning, it is difficult to update the prediction coefficient in the receiving device.
[0118]
On the other hand, when the predicted value of the HD pixel is obtained by the second adaptive processing, the HD pixel is not necessary as the teacher data, and learning can be performed using only the SD pixel (including the thinned SD pixel). Therefore, the prediction coefficient can be updated in the receiving apparatus.
[0119]
Therefore, FIG. 13 shows a configuration of an embodiment of an image conversion apparatus that converts an SD image into an HD image while updating a prediction coefficient. In the figure, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, this image conversion apparatus is newly provided with a frame memory 91, a learning blocking circuit 92, a teacher blocking circuit 93, a learning circuit 94, and a coefficient RAM (for example, SRAM (Static Read Only Memory)) 95. 1 except that the prediction circuit 96 is provided in place of the prediction circuit 6.
[0120]
In the frame memory 91, an SD image transmitted via a transmission path or reproduced from a recording medium is stored in units of frames (or fields), for example. The SD image stored in the frame memory 91 is supplied to the class classification blocking circuit 1, and the learning block 92 described later outputs from the class classification circuit 4 in the same manner as in FIG. The index corresponding to the block class is output.
[0121]
The SD image stored in the frame memory 91 is also supplied to the learning blocking circuit 92 and the teacher blocking circuit 93 at the same time.
[0122]
The learning blocking circuit 92, the teacher blocking circuit 93, or the learning circuit 94 is configured similarly to the learning blocking circuit 75, the teacher blocking circuit 76, or the learning circuit 77 in FIG. The learning circuit 94 is supplied with an index from the class classification circuit 4 corresponding to the class classification circuit 74 (FIG. 9). Therefore, the learning circuit 94 performs learning as described with reference to FIG. 9 and supplies the prediction coefficient obtained as a result to the coefficient RAM 95.
[0123]
The coefficient RAM 95 is supplied with a prediction coefficient from the learning circuit 94 and is supplied with an index from the class classification circuit 4. In the coefficient RAM 95, an address corresponding to the index is sent from the learning circuit 94. The prediction coefficient is stored (overwritten).
[0124]
When the above learning process is performed on the SD image stored in the frame memory 91, for example, all the pixels are used as the pixel of interest, the predicted value calculation blocking circuit 5 stores the SD stored in the frame memory 91. Prediction value calculation blocks are sequentially constructed from the image and output to the prediction circuit 96.
[0125]
At this time, the SD image stored in the frame memory 91 is also supplied again to the class classification blocking circuit 1, and the prediction value calculation blocking circuit 5 is configured in the same manner as described above. The index corresponding to the class of the predicted value calculation block to be output is output from the class classification circuit 4.
[0126]
This index is given to the coefficient RAM 95 as an address, and the prediction coefficient stored at the address is read from the coefficient RAM 95 and supplied to the prediction circuit 96.
[0127]
The prediction circuit 96 includes a sum-of-products calculator 62 and a limiter 63, excluding the coefficient ROM 61, of the blocks constituting the prediction circuit 6 shown in FIG. 8, where the prediction coefficient from the coefficient RAM 95 is used, As described in Expression (8), the predicted value of the HD pixel is obtained.
[0128]
As described above, according to the case where a prediction coefficient is obtained from an SD image to be converted into an HD image and the SD image is converted into an HD image using the prediction coefficient, a more accurate HD image can be obtained. It becomes possible.
[0129]
As described above, the case where the present invention is applied to an image conversion apparatus that converts an SD image into an HD image has been described. However, the present invention can also be applied to, for example, an image enlargement process.
[0130]
In this embodiment, the coefficient ROM 61 (FIG. 8) (the same applies to the coefficient RAM 95 in FIG. 13) stores the prediction coefficient at the address corresponding to each class. In addition, for example, it is possible to store an average value of pixel values constituting the teacher block. In this case, when an index for a class is given, a pixel value corresponding to the class is output, and it is not necessary to provide the prediction value calculation blocking circuit 5 and the prediction circuit 6 (or the prediction circuit 96). It becomes like this.
[0131]
Furthermore, in the present embodiment, the final class is determined from both the first class information and the second class information, but from either one of the first class information or the second class information. It is also possible to determine the final class.
[0132]
In the present embodiment, the thinning circuit 2 is configured by simply thinning out the pixels to form a thinning block. However, the thinning block is configured such that, for example, an average value of several pixels is 1 or the like. It is also possible to generate the image by assigning it to a pixel.
[0133]
Furthermore, in the present embodiment, the shape of each block is a square, but the shape of the block is not limited to a square. That is, the block in this specification means a set of several pixels, and the shape thereof can be, for example, a rectangle, a cross, a circle, or any other shape besides a square.
[0134]
In this embodiment, class classification is performed in consideration of image self-similarity. However, class classification does not consider image self-similarity, and is simply a pattern of pixel values. It is also possible to perform based on this. That is, for example, as shown in FIG. 14A, a block of 2 × 2 pixels is formed by a certain pixel of interest and three pixels adjacent thereto, and each pixel is expressed by 1 bit. (Takes a level of either 0 or 1). In this case, as shown in FIG. 14B, the block of 2 × 2 4 pixels is 16 (= (2¹)^Four) Can be classified into patterns. The classification can be performed only by such pattern division.
[0135]
Furthermore, the present invention can be realized by both hardware and software.
[0136]
【The invention's effect】
  According to the image processing device according to claim 1 and the image processing method according to claim 6, the prediction parameter is:A second block having a predetermined size including a second target pixel that is a pixel of interest in calculation of a prediction parameter among the first pixels and a first pixel around the second target pixel. As described above, the first image signal is blocked, and a second block to which the second pixel of interest belongs and which is a second block having a predetermined size including the position of the second pixel of interest is formed. Based on the first pixel, the second class information representing the class is generated, and is a position corresponding to the predicted position of the second pixel, from the same position as the first pixel or a position in between, A second direction that is proportional to the first distance in the same direction with respect to the direction to each position of the first pixel that is sum-of-products calculated for the prediction of the second pixel and the first distance therebetween. The first pixel in the similar positional relationship that is the position of the distance of There are, of predicted values of the pixel values, by performing learning to minimize differences between true values were calculated for each class indicated by the second class informationThe pixel value of the second image is obtained by performing an adaptive process using the prediction parameter. Therefore, a high-resolution image can be easily obtained as the second image.
  According to the image processing device of claim 5, the predetermined pixel composed of the first pixel of interest that is the pixel of interest among the first pixels and the first pixel around the first pixel of interest. The first image signal is blocked so as to constitute the first block having the size, and is a first block to which the first target pixel belongs, and has a predetermined size including the position of the first target pixel. Based on the first pixels constituting the first block, first class information representing a class is generated, and a prediction parameter for predicting the second image signal using the first image signal is stored. The second sum of the second image signal is calculated by multiplying the prediction parameter read from the prediction parameter storage unit according to the first class information and the first pixel located around the first target pixel. A pixel corresponding to the position of the first pixel of interest A second pixel around the same or corresponding position is predicted and corresponds to the predicted second pixel position, and is the same position as or in between the first pixel The same direction and proportional to the first distance with respect to the direction from the position to the respective positions of the first pixels to be summed for prediction of the second pixel and the first distance therebetween By performing learning that minimizes the error of the predicted value of the pixel value with respect to the true value using the first pixel having a similar positional relationship that is the position of the second distance to be calculated, the prediction parameter for each class is determined. Since it is calculated, a high-resolution image can be easily obtained as the second image.
  According to the parameter generation device according to claim 7 and the parameter generation method according to claim 13, the prediction parameter isThe first image signal is blocked so as to form a block of a predetermined size including a target pixel that is a target pixel of the first pixels and a first pixel around the target pixel. Class information representing a class is generated based on a first pixel constituting a block having a predetermined size including the position of the target pixel, to which the target pixel belongs. Then, a first sum-of-products operation is performed for the prediction of the second pixel from a position corresponding to the position of the second pixel to be predicted and the same position as the first pixel or a position in between. A first pixel having a similar positional relationship that is a position of a second distance that is in the same direction and proportional to the first distance with respect to the direction to the respective positions of the pixels and the first distance therebetween. And is calculated for each class by learning to minimize the error of the predicted value of the pixel value with respect to the true value.That is, the prediction parameter can be obtained by performing learning that minimizes the error of the predicted value of the pixel value with respect to the true value using only the pixel of the first image signal.
  Further, according to the image processing device according to claim 14 and the image processing method according to claim 15, the prediction parameter includes a pixel of the first image signal used for the product-sum operation, and the first image signal. The pixel of the third image signal whose resolution has been reduced is generated so as to minimize the error of the predicted value of the pixel value with respect to the true value. By performing an adaptive process using a prediction parameter generated using only pixels of the image signal, the pixel value of the second image is obtained. Therefore, a high-resolution image can be easily obtained as the second image.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a first embodiment of an image conversion apparatus to which the present invention is applied.
FIG. 2 is a diagram for explaining processing of the image conversion apparatus in FIG. 1;
FIG. 3 is a block diagram illustrating a configuration example of a class classification circuit 4 in FIG. 1;
4 is a block diagram illustrating a configuration example of a first class generation circuit 11 in FIG. 3;
5 is a block diagram illustrating a configuration example of a second class generation circuit 12 in FIG. 3;
FIG. 6 is a diagram for explaining ADRC processing;
7 is a block diagram illustrating a configuration example of a final class determination circuit 13 in FIG. 3;
8 is a block diagram illustrating a configuration example of a prediction circuit 6 in FIG.
FIG. 9 is a block diagram illustrating a configuration example of an image processing apparatus that performs a learning process for obtaining a prediction coefficient.
10 is a block diagram illustrating a configuration example of a learning circuit 77 in FIG. 9;
11 is a block diagram illustrating a configuration example of a multiplication circuit 81 in FIG.
12 is a block diagram illustrating a configuration example of an adder circuit 82 in FIG. 10;
FIG. 13 is a block diagram showing a configuration of a second embodiment of an image conversion apparatus to which the present invention is applied.
FIG. 14 is a diagram for explaining class classification performed based only on a pattern of pixel value levels.
[Explanation of symbols]
1 block classification circuit, 2 decimation circuit, 3 reduction circuit, 4 class classification circuit, 5 block calculation circuit, 6 prediction circuit, 7 monitor, 11 first class generation circuit, 12 second class generation circuit , 13 Final class decision circuit, 21₁Thru 21_Four  Similarity calculation unit, 22 maximum similarity direction determination unit, 26B and 26C pixel extraction unit, 27 norm calculation unit, 31 maximum similarity direction pixel extraction unit, 32 ADRC processing unit, 33 pattern classification unit, 41 maximum value detection unit, 42 Minimum value detection unit, 43 delay unit, 44 arithmetic unit, 45 ADRC code determination unit, 51 ROM, 61 coefficient ROM, 62 multiply-add arithmetic unit, 63 limiter, 71 class classification blocking circuit, 72 decimation circuit, 73 reduction Circuit, 74 class classification circuit, 75 learning block circuit, 76 teacher block circuit, 77 learning circuit, 81 multiplier circuit, 82 adder circuit, 83 decoder, 84 arithmetic circuit, 91 frame memory, 92 learning block circuit, 93 teacher blocking circuit, 94 learning circuit, 5 coefficient RAM, 96 prediction circuit

Claims (15)

The first image signal composed of the first pixels is a second image signal having a higher resolution than the first image signal, and is the same as or in the position between the positions of the first pixels. In the image processing apparatus for converting to a second image signal composed of the second pixels arranged,
A first block having a predetermined size is formed by a first pixel of interest which is a pixel of interest among the first pixels and the first pixel around the first pixel of interest. as such, the blocking means for blocking said first image signal,
The first block to which the first pixel of interest belongs, and a class is determined based on the first pixel constituting the first block of a predetermined size including the position of the first pixel of interest. Class classification means for generating first class information to be represented;
Prediction parameter storage means for storing a prediction parameter for predicting the second image signal using the first image signal;
The second image signal is obtained by a product-sum operation between the prediction parameter read from the prediction parameter storage unit according to the first class information and the first pixel located around the first pixel of interest. Prediction calculation means for predicting the second pixel in the vicinity of the same or corresponding position with respect to the position corresponding to the position of the first target pixel .
The prediction parameter storage means includes a second pixel of interest that is a pixel of interest in calculation of the prediction parameter of the first pixel, and the first pixel around the second pixel of interest. The first image signal is blocked so as to form a second block of a predetermined size, and the second target pixel to which the second target pixel belongs, Second class information representing a class is generated based on the first pixel constituting the second block having a predetermined size including the position corresponding to the position of the predicted second pixel. A position from the same position as the first pixel or a position between them to a position of each of the first pixels that is summed for prediction of the second pixel, and between them For the first distance of Using the first pixels in the same direction and having a similar positional relationship that is the position of the second distance proportional to the first distance, the error of the predicted value of the pixel value with respect to the true value is minimized. The image processing apparatus is characterized in that the prediction parameter calculated for each class indicated by the second class information is stored by performing learning . Positional relationship similar to the positional relationship between the first pixel of the first image signal used for the product-sum operation and the second pixel of the second image signal predicted by the product-sum operation The first pixel of the first image signal is the first pixel of the first image signal, with an interval between the first pixels adjacent to the first image signal being a sampling interval. The first pixels arranged at a wider interval than the sampling interval, and the first pixels adjacent to each other at the same interval as the sampling interval. Image processing apparatus. The class classification means uses the interval between the first pixels adjacent to the first image signal as a sampling interval,
Of the first pixel constituting the block, and the third image signal to be in the first pixel that exists in a wider interval than the sampling interval, is present at the same interval as the sampling interval, the Based on the fourth image signal composed of the first pixel close to the first pixel of interest, the predetermined pixel that passes through the first pixel of interest among the first pixels constituting the third image signal. Among the first vector having the pixel value of the first pixel on the straight line in the direction of the element and the first pixel constituting the fourth image signal, the element of the first vector is determined. A norm calculated for each direction with a second vector that is a straight line in the same direction as the straight line and that has the pixel value of the first pixel on the straight line passing through the first target pixel as an element, The third direction indicating the direction of the straight line from which the lowest norm is obtained And class information,
The first target pixel and a pixel selected from the pixels of the fourth image signal on a straight line in the direction indicated by the third class information and passing through the first target pixel The image processing apparatus according to claim 1, wherein the first class information is generated from fourth class information indicating a pattern of pixel values . A thinning circuit that generates the third image signal by thinning out the pixels constituting the block at regular intervals, and supplies the third image signal to the class classification unit;
4. The reduction circuit according to claim 3, further comprising: a reduction circuit that generates the fourth image signal by removing an outer pixel of the pixels constituting the block and supplies the fourth image signal to the class classification unit. Image processing apparatus. The first image signal composed of the first pixels is a second image signal having a higher resolution than the first image signal, and the same position as the position of the first pixel or a position between them. In an image processing apparatus for converting to a second image signal composed of second pixels arranged,
A first block having a predetermined size is formed by a first pixel of interest, which is a pixel of interest among the first pixels, and the first pixel around the first pixel of interest. Block means for blocking the first image signal,
The first block to which the first pixel of interest belongs, which is based on the first pixel constituting the first block having a predetermined size including the position of the first pixel of interest. Class classification means for generating first class information to be represented;
Prediction parameter storage means for storing a prediction parameter for predicting the second image signal using the first image signal;
The second image signal is obtained by a product-sum operation between the prediction parameter read from the prediction parameter storage unit according to the first class information and the first pixel located around the first pixel of interest. Prediction operation means for predicting the second pixel in the vicinity of the same or corresponding position with respect to the position corresponding to the position of the first target pixel,
The position corresponding to the position of the second pixel to be predicted, and the product-sum operation is performed for the prediction of the second pixel from the same position as the first pixel or a position therebetween. The first position in the same direction and the position of the second distance proportional to the first distance with respect to the direction to the respective positions of the first pixels and the first distance therebetween are similar to each other. Learning means for calculating the prediction parameter for each class by performing learning to minimize an error of a predicted value of a pixel value with respect to a true value using one pixel;
An image processing apparatus comprising: The first image signal composed of the first pixels is a second image signal having a higher resolution than the first image signal, and is the same as or in the position between the positions of the first pixels. In an image processing method for converting to a second image signal composed of second pixels arranged,
A first block having a predetermined size is formed by a first pixel of interest which is a pixel of interest among the first pixels and the first pixel around the first pixel of interest. as such, the block step of blocking said first image signal,
The first block to which the first pixel of interest belongs, and a class is determined based on the first pixel constituting the first block of a predetermined size including the position of the first pixel of interest. A classifying step for generating first class information representing;
The prediction parameter read out in accordance with the first class information and the first parameter from a prediction parameter storage unit that stores a prediction parameter for predicting the second image signal using the first image signal . the product-sum operation between the first pixel located around the pixel of interest, a said second pixel of the second image signal, with respect to a position corresponding to the position of the first pixel of interest Predicting the second pixel around the same or corresponding position , and
The prediction parameter storage means includes a second pixel of interest that is a pixel of interest in calculation of the prediction parameter of the first pixel, and the first pixel around the second pixel of interest. The first image signal is blocked to form a second block having a predetermined size, and the second block to which the second pixel of interest belongs is the second block of pixels to which the second pixel of interest belongs. Second class information representing a class is generated based on the first pixel constituting the second block having a predetermined size including the position corresponding to the position of the predicted second pixel. A position from the same position as the first pixel or a position between them to a position of each of the first pixels to be multiplied and summed for prediction of the second pixel, and between them For the first distance of Using the first pixels in the same direction and having a similar positional relationship that is the position of the second distance proportional to the first distance, the error of the predicted value of the pixel value with respect to the true value is minimized. By performing the learning, the prediction parameter calculated for each class indicated by the second class information is stored. The first image signal composed of the first pixels is a second image signal having a higher resolution than the first image signal, and is the same as or in the position between the positions of the first pixels. In a parameter generation device that generates a prediction parameter used for a product-sum operation when converting to a second image signal composed of second pixels to be arranged,
The first image signal so as to form a block of a predetermined size including a target pixel that is a target pixel of the first pixels and the first pixel around the target pixel. Blocking means for blocking
A said block in which the pixel of interest belongs, a predetermined on the basis of the first pixels constituting the block size, classification means for generating class information indicating a class including the position of the pixel of interest,
The position corresponding to the position of the second pixel to be predicted, and the product-sum operation is performed for the prediction of the second pixel from the same position as the first pixel or a position therebetween. The first position in the similar direction, which is the position of the second distance in the same direction and proportional to the first distance, with respect to the direction to each position of the first pixel and the first distance therebetween. And a learning unit that calculates the prediction parameter for each class by performing learning that minimizes an error of the predicted value of the pixel value with respect to the true value using one pixel. . Positional relationship similar to the positional relationship between the first pixel of the first image signal used for the product-sum operation and the second pixel of the second image signal predicted by the product-sum operation The first pixel of the first image signal is the first pixel of the first image signal, with an interval between the first pixels adjacent to the first image signal being a sampling interval. The first pixels arranged at an interval wider than the sampling interval, and the first pixels adjacent to each other at the same interval as the sampling interval. Parameter generator. The learning means obtains the first pixels adjacent to each other at the same interval as the sampling interval, which is obtained by a product-sum operation of the first pixels arranged at intervals wider than the sampling interval and the prediction parameter. The parameter generation device according to claim 8, wherein the prediction parameter that minimizes an error of a predicted value of a pixel value with respect to a true value is calculated. The first pixels adjacent to each other at the same interval as the sampling interval are pixels within a range in which the first pixels existing at intervals wider than the sampling interval are distributed. The parameter generation device described. The class classification means uses the interval between the first pixels adjacent to the first image signal as a sampling interval,
Of the first pixels constituting the block, a third image signal composed of the first pixels existing at a wider interval than the sampling interval, and the same as the sampling interval, based on the fourth image signal consisting of the first pixel near the target pixel, among the first pixels constituting the third image signal, the predetermined direction through the target pixel in the straight line A first vector having the pixel value of the first pixel as an element and a straight line in the same direction as a straight line that determines an element of the first vector among the first pixels constituting the fourth image signal The norm having the smallest value among the norms calculated for each direction with the second vector having the pixel value of the first pixel on the straight line passing through the target pixel as an element was obtained. First class information indicating the direction of the straight line ;
A pattern of pixel values of the target pixel and a pixel selected from the pixels of the fourth image signal on the straight line in the direction indicated by the third class information and passing through the target pixel. The parameter generation device according to claim 7, wherein the class information is generated from second class information . A thinning circuit that generates the third image signal by thinning out the pixels constituting the block at regular intervals, and supplies the third image signal to the class classification unit;
12. The reduction circuit according to claim 11, further comprising: a reduction circuit that generates the fourth image signal by removing an outer pixel of the pixels constituting the block and supplies the fourth image signal to the class classification unit. Parameter generator. The first image signal composed of the first pixels is a second image signal having a higher resolution than the first image signal, and is the same as or in the position between the positions of the first pixels. In a parameter generation method for generating a prediction parameter used for a prediction calculation when converting to a second image signal composed of second pixels to be arranged,
The first image signal so as to form a block of a predetermined size including a target pixel that is a target pixel of the first pixels and the first pixel around the target pixel. A block step for blocking
A said block in which the pixel of interest belongs, a class classification step of generating the basis of the first pixel, the class information representing the class that constitutes the predetermined size of the block including the position of the pixel of interest,
The position corresponding to the position of the second pixel to be predicted, and the product-sum operation is performed for the prediction of the second pixel from the same position as the first pixel or a position therebetween. The first position in the similar direction, which is the position of the second distance in the same direction and proportional to the first distance, with respect to the direction to each position of the first pixel and the first distance therebetween. And a learning step of calculating the prediction parameter for each class by performing learning that minimizes an error of a predicted value of a pixel value with respect to a true value using one pixel. . In an image processing apparatus for converting a first image signal into a second image signal having a higher resolution than the first image signal,
A predetermined number of pixels centered on a pixel of interest which is a pixel of interest among the pixels of the first image signal, each of which has a predetermined number of pixels, and includes a block of pixels of the first image signal to a blocking means for blocking said first image signal,
Based on the pixels constituting the block in which the pixel of interest belongs, and class classification means for generating class information representing the class,
Prediction parameter storage means for storing a prediction parameter for predicting the second image signal using the first image signal;
The second image signal is generated by a product-sum operation between the prediction parameter read from the prediction parameter storage unit according to the class information and the pixel of the first image signal around the position of the target pixel. And a predictive calculation means for
The prediction parameter storage means uses a pixel value of the first image signal used for the product-sum operation and a pixel value of the third image signal obtained by lowering the resolution of the first image signal. An image processing apparatus, wherein the prediction parameter for each class generated so as to minimize an error of a predicted value of the true value with respect to a true value is stored. In an image processing method for converting a first image signal into a second image signal having a higher resolution than the first image signal,
A predetermined number of pixels centered on a pixel of interest which is a pixel of interest among the pixels of the first image signal, each of which has a predetermined number of pixels, and includes a block of pixels of the first image signal in the block step of blocking said first image signal,
Based on the pixels constituting the block in which the pixel of interest belongs, a class classification step of generating class information representing the class,
The prediction parameter read out in accordance with the class information from the prediction parameter storage means for storing the prediction parameter for predicting the second image signal using the first image signal, and the vicinity of the position of the target pixel A predictive operation step of generating the second image signal by a product-sum operation with the pixels of the first image signal in
The prediction parameter storage means uses a pixel value of the first image signal used for the product-sum operation and a pixel value of the third image signal obtained by lowering the resolution of the first image signal. An image processing method, wherein the prediction parameter for each class generated so as to minimize an error of the predicted value of the true value with respect to the true value is stored.

Images