JP4419287B2 - Image processing apparatus, image processing method, and recording medium - Google Patents

Description

但し、式（１）において、Σは、ｉ，ｊを、１からＩ，Ｊまでに変えてのサメーションを表し、ＩとＪは、ブロックを構成する画素のライン数と列数を、それぞれ表す。従って、本実施の形態では、Ｉ，Ｊは、ともに８である。
【００７８】
なお、アクティビティ算出部２２は、注目ブロックについてアクティビティを計算するとともに、その注目ブロックの上下左右にそれぞれ隣接するブロックのアクティビティも計算する。但し、アクティビティ算出部２２は、注目ブロックの上下左右にそれぞれ隣接するブロックに関しては、サイドインフォメーション抽出部１３から供給されるＤＣＴタイプを参照することにより、そのブロックを構成するライン構造を認識し、注目ブロックと同一構成のライン構造のブロックについて、アクティビティを計算する。
【００７９】
即ち、例えば、図５に示したように、注目マクロブロックＭＢ_Nと、マクロブロックＭＢ_U，ＭＢ_D，ＭＢ_Lが、いずれもフレームＤＣＴモードである場合には、アクティビティ算出部２２は、注目ブロックＢ_N-1の上下左右に隣接するブロックのアクティビティを、それぞれ、注目ブロックＢ_N-1の上に隣接するブロックＢ_U-3の第１乃至第８ライン、下に隣接するブロックＢ_N-3の第１乃至第８ライン、左に隣接するブロックＢ_L-2の第１乃至第８ライン、右に隣接するブロックＢ_N-2の第１乃至第８ラインから、式（１）にしたがって演算する。
【００８０】
また、例えば、図６に示したように、注目マクロブロックＭＢ_NがフィールドＤＣＴモードであり、マクロブロックＭＢ_U，ＭＢ_D，ＭＢ_Lが、いずれもフレームＤＣＴモードである場合には、アクティビティ算出部２２は、注目ブロックＢ_N-1の上下左右に隣接するブロックのアクティビティを、それぞれ、注目ブロックＢ_N-1の上に隣接するマクロブロックＭＢ_UのブロックＢ_U-1の４つの奇数ラインおよびブロックＢ_U-3の４つの奇数ライン、下に隣接するマクロブロックＭＢ_DのブロックＢ_D-1の４つの奇数ラインおよびブロックＢ_D-3の４つの奇数ライン、左に隣接するマクロブロックＭＢ_LのブロックＢ_L-2の４つの奇数ラインおよびブロックＢ_L-4の４つの奇数ライン、右に隣接するブロックＢ_N-2の第１乃至第８ラインから、式（１）にしたがって演算する。
【００８１】
さらに、例えば、図７に示したように、注目マクロブロックＭＢ_NがフレームＤＣＴモードであり、マクロブロックＭＢ_U，ＭＢ_D，ＭＢ_Lが、いずれもフィールドＤＣＴモードである場合には、アクティビティ算出部２２は、注目ブロックＢ_N-1の上下左右に隣接するブロックのアクティビティを、それぞれ、注目ブロックＢ_N-1の上に隣接するマクロブロックＭＢ_UのブロックＢ_U-1の第５乃至第８ラインおよびブロックＢ_U-3の第５乃至第８ライン、下に隣接するブロックＢ_N-3の第１乃至第８ライン、左に隣接するマクロブロックＭＢ_LのブロックＢ_L-2の第１乃至第４ラインおよびブロックＢ_L-4の第１乃至第４ライン、右に隣接するブロックＢ_N-2の第１乃至第８ラインから、式（１）にしたがって演算する。
【００８２】
また、例えば、図８に示したように、注目マクロブロックＭＢ_N、マクロブロックＭＢ_U，ＭＢ_D，ＭＢ_Lが、いずれもフィールドＤＣＴモードである場合には、アクティビティ算出部２２は、注目ブロックＢ_N-1の上下左右に隣接するブロックのアクティビティを、それぞれ、注目ブロックＢ_N-1の上に隣接するマクロブロックＭＢ_UのブロックＢ_U-1の第１乃至第８ライン、下に隣接するマクロブロックＭＢ_DのブロックＢ_D-1の第１乃至第８ライン、左に隣接するブロックＢ_L-2の第１乃至第８ライン、右に隣接するブロックＢ_N-2の第１乃至第８ラインから、式（１）にしたがって演算する。
【００８３】
次に、図１の歪補正値算出部２を構成する補正値算出部２３の処理について説明する。
【００８４】
補正値算出部２３は、ブロック境界段差検出部２１から供給される注目ブロックの境界の差分情報を、アクティビティ算出部２２からの注目ブロックおよびその上下左右に隣接するブロックのアクティビティに基づいて処理することにより、補正値を算出する。
【００８５】
即ち、補正値算出部２３は、注目ブロックと、それに隣接するブロック（以下、適宜、隣接ブロックという）とのアクティビティがほぼ等しい場合には、その注目ブロックの境界画素の画素値と、その境界画素に隣接する隣接ブロックの画素値とを、それらの２つの画素値の中間値に近づけるような補正値を求める。
【００８６】
具体的には、例えば、いま差分情報として用いている、注目ブロックの境界画素の画素値と、その境界画素に隣接する隣接ブロックの画素値との差分を、差分画素値というものとすると、補正値算出部２３は、例えば、図９（Ａ）に示すように、ブロック歪みを低減するために、注目ブロックの境界画素の画素値を、差分画素値の７／１６だけ、その境界画素に隣接する隣接ブロックの画素値に近づけるような補正値を求める。
【００８７】
なお、境界画素に隣接する隣接ブロックの画素値については、その隣接ブロックが注目ブロックとなった場合に、上述の場合と同様にして、差分画素値の７／１６だけ画素値を補正するような補正値が求められる。また、この場合、注目ブロックの境界画素の画素値と、その境界画素に隣接する隣接ブロックの画素値とが、互いに、差分画素値の７／１６だけ近づくように補正されることとなり、従って、それらの２つの補正後の画素値の間には、差分画素値の１／８（＝２／１６）だけの差が生じるが、これは、隣接するブロックの境界画素どうしの間に、ある程度の差がある方が、そのブロックどうしが滑らかに繋がり、自然に見えるからである。
【００８８】
一方、補正値算出部２３は、注目ブロックと、隣接ブロックとのアクティビティに、ある程度の差がある場合には、その注目ブロックの境界画素の画素値、またはその境界画素に隣接する隣接ブロックの画素値のうちの、アクティビティが大きい方のブロックの画素値を、アクティビティが小さい方のブロックの画素値に近づけるような補正値を求める。
【００８９】
即ち、注目ブロックと、隣接ブロックとのアクティビティに、ある程度の差がある場合には、そのうちのアクティビティの大きい方のブロックがモスキートノイズを含んでいる可能性が高いため、そのモスキートノイズが、他方のブロック（アクティビティが小さい方のブロック）に伝搬（影響）することを防止する必要がある。
【００９０】
そこで、補正値算出部２３は、注目ブロックと隣接ブロックとのアクティビティに、ある程度の差がある場合において、そのうちの注目ブロックのアクティビティの方が大きいときには、例えば、図９（Ｂ）に示すように、注目ブロックの境界画素の画素値を、差分画素値の分だけ、その境界画素に隣接する隣接ブロックの画素値に近づけるような補正値、即ち、注目ブロックの境界画素の画素値を、その境界画素に隣接する隣接ブロックの画素値に一致させるような補正値を求める。
【００９１】
補正値算出部２３は、以上のような補正値を、ブロック境界段差検出部２１からの差分画素値である差分情報と、アクティビティ算出部２２からのアクティビティを用いて、例えば、次のようにして求める。
【００９２】
即ち、例えば、いま、図１０に示すように、注目ブロックのアクティビティをＣで表すとともに、その上下左右に隣接するブロックのアクティビティを、それぞれＡ，Ｅ，Ｂ，Ｄでそれぞれ表すとすると、補正値算出部２３は、注目ブロックの上下左右の境界の差分情報としての差分画素値それぞれに対して付す重みＧｕ，Ｇｄ，Ｇｌ，Ｇｒを、次式にしたがって演算する。
【００９３】

【００９４】
そして、補正値算出部２３は、図１１に示すように、注目ブロックの上下左右の境界それぞれの差分情報としての差分画素値を、Ｇｕ，Ｇｄ，Ｇｌ，Ｇｒ倍して補正値を得る。
【００９５】
なお、補正値算出部２３は、差分情報のうちの、例えば、最も左上の差分画素値については、重みＧｕとＧｌとの平均値を重みとして付す。同様に、補正値算出部２３は、最も右上の差分画素値については、重みＧｕとＧｒとの平均値を、最も左下の差分画素値については、重みＧｌとＧｄとの平均値を、最も右下の差分画素値については、重みＧｒとＧｄとの平均値を、それぞれ重みとして付す。
【００９６】
従って、補正値算出部２３では、境界部分に、重みが付された差分画素値が配置され、残りの部分に、０が配置されている８×８のブロックの補正値が得られる。
【００９７】
また、上述の場合には、差分情報に対して、アクティビティにのみ基づく重みを付して補正値を求めるようにしたが、補正値は、その他、例えば、注目ブロックに隣接する隣接ブロックが、注目マクロブロックではないマクロブロックに属し、かつ、注目ブロックと隣接ブロックとの量子化スケールが異なるときには、差分情報に対して、アクティビティと量子化スケールとに基づく重みを付して求めるようにすること等が可能である。
【００９８】
次に、図１の歪補正値算出部２を構成する高域低減部２５の処理について説明する。
【００９９】
高域低減部２５には、補正値算出部２３において得られる空間領域の８×８のブロックの補正値が、ＤＣＴ変換部２４において、図１２（Ａ）に示すようにＤＣＴ処理され、８×８個の周波数領域のＤＣＴ係数とされたものが供給される。
【０１００】
高域低減部２５は、そこに供給される補正値としてのＤＣＴ係数のうちの高次のものを低減することにより、即ち、例えば、図１２（Ｂ）に示すように、補正値としての８×８個のＤＣＴ係数のうちの第５行乃至第８行、および第５列乃至第８列のものを０とすること等により、その補正値を修正し、補正値加算部３１に供給する。
【０１０１】
即ち、ＤＣＴ変換部２４が出力する補正値を、そのまま、ＤＣＴ係数抽出／逆量子化部１２が出力するブロックのＤＣＴ係数の補正に用いると、そのブロックの境界付近の画素値しか補正されず、その結果、例えば、図１３（Ａ）に示すような急峻な段差のあるブロックの境界部分の画素値は、図１３（Ｂ）に示すように補正される。従って、ブロックの境界は、補正前よりも目立たなくはなるが、それでも、ブロックの境界部分の段差が、まだ急峻であるため、復号画像にブロック構造が現れる。
【０１０２】
そこで、高域低減部２５は、補正値としてのＤＣＴ係数の高次のものを低減することにより、その補正値を修正する。このような修正後の補正値を、ＤＣＴ係数抽出／逆量子化部１２が出力するブロックのＤＣＴ係数の補正に用いた場合には、そのブロックの境界付近だけでなく、その内部の画素値も補正され、その結果、例えば、図１３（Ａ）に示すような急峻な段差のあるブロックの境界部分の画素値は、図１３（Ｃ）に示すように補正される。従って、ブロックの境界部分の段差が滑らかになるため、復号画像にブロック構造が現れることを、十分低減することができる。
【０１０３】
次に、高域低減部２５において、補正値としてのＤＣＴ係数の高次のものを低減するように、補正値を修正する方法としては、図１２（Ｂ）に示したような、高次のＤＣＴ係数を０とする方法以外の方法を採用することが可能である。
【０１０４】
即ち、例えば、図１４（Ａ）に示すように、補正値は、第５行乃至第８行、および第５列乃至第８列のものを０に修正するのではなく、任意の第ｉ行以上、および第ｉ列以上のものを０に修正するようにすることができる。そして、この場合、変数ｉは、注目ブロックのアクティビティや、注目ブロックの量子化に用いられた量子化スケール等に基づいて決定することができる。即ち、例えば、注目ブロックのアクティビティが大きい場合には、より高次のＤＣＴ係数だけを０に修正するようにし、そのアクティビティが小さい場合には、より低次のＤＣＴ係数も０に修正するようにすることができる。また、例えば、注目ブロックの量子化スケールが小さい場合には、より高次のＤＣＴ係数だけを０に修正するようにし、その量子化スケールが大きい場合には、より低次のＤＣＴ係数も０に修正するようにすることができる。
【０１０５】
なお、注目ブロックのアクティビティや量子化スケール等に基づいて、０に修正するＤＣＴ係数の次数を決定する場合には、その次数の下限値を設定しておくようにすることができる。即ち、例えば、第４行以下および第４列以下のＤＣＴ係数は、０に修正しないような設定をしておくことが可能である。この場合、０に修正するＤＣＴ係数は、第５行乃至第８行、および第５列乃至第８列の範囲で変化することになる。
【０１０６】
また、補正値は、図１４（Ａ）に示したように、８×８のＤＣＴ係数のうち、同一の行と列以上のものを０に修正する他、０に修正する行と列を、独立に決定することも可能である。
【０１０７】
即ち、例えば、図１４（Ｂ）に示すように、注目ブロックの左と右にそれぞれ隣接するブロックのアクティビティが大きい場合には、水平方向の、より高次のＤＣＴ係数だけを０に修正するようにし、その左右に隣接するブロックのアクティビティが小さい場合には、水平方向の、より低次のＤＣＴ係数も０に修正するようにすることができる。また、例えば、注目ブロックの上と下にそれぞれ隣接するブロックのアクティビティが大きい場合には、垂直方向の、より高次のＤＣＴ係数だけを０に修正するようにし、上下に隣接するブロックのアクティビティが小さい場合には、垂直方向の、より低次のＤＣＴ係数も０に修正するようにすることができる。
【０１０８】
さらに、例えば、注目ブロックの左と右にそれぞれ隣接するブロックの量子化スケールが小さい場合には、水平方向の、より高次のＤＣＴ係数だけを０に修正するようにし、その左右に隣接するブロックの量子化スケールが大きい場合には、水平方向の、より低次のＤＣＴ係数も０に修正するようにすることができる。また、例えば、注目ブロックの上と下にそれぞれ隣接するブロックの量子化スケールが小さい場合には、垂直方向の、より高次のＤＣＴ係数だけを０に修正するようにし、その上下に隣接するブロックの量子化スケールが大きい場合には、垂直方向の、より低次のＤＣＴ係数も０に修正するようにすることができる。
【０１０９】
さらに、図１４（Ａ）や図１４（Ｂ）に示した場合においては、注目ブロックと、その上下に隣接するブロックや左右に隣接するブロックとのアクティビティの和等に基づいて、０に修正するＤＣＴ係数の次数を決定することも可能である。
【０１１０】
また、補正値としてのＤＣＴ係数は、例えば、そのＤＣＴ係数に、所定の重み付けを行うことにより修正することも可能である。
【０１１１】
即ち、例えば、図１４（Ｃ）に示すように、補正値としての８×８のＤＣＴ係数のうち、第ｖ行第ｕ列のＤＣＴ係数を、Ｆ（ｕ，ｖ）と表すとともに、修正後の補正値としてのＤＣＴ係数を、Ｆ’（ｕ，ｖ）と表すこととすると、補正値としてのＤＣＴ係数は、例えば、次の２つの式のいずれかにしたがって修正することが可能である。
【０１１２】

但し、式（３）において、ａ，ｂ，ｃは、注目ブロックや、その上下左右に隣接するブロックのアクティビティ、あるいは量子化スケール等に基づいて設定される定数を表す。
【０１１３】
また、補正値としてのＤＣＴ係数は、例えば、注目ブロックの量子化に用いられた量子化テーブルに基づく重み付けを行うことにより修正することも可能である。
【０１１４】
即ち、注目ブロックの量子化に用いられた量子化テーブルｑが、例えば、図１４（Ｄ）に示すようなものであった場合において、その量子化テーブルｑの第ｖ行第ｕ列の値をｑ（ｕ，ｖ）と表すと、補正値としてのＤＣＴ係数は、例えば、次式にしたがって修正することが可能である。
【０１１５】

但し、式（４）において、ａは、式（３）における場合と同様に、注目ブロック等のアクティビティ等に基づいて設定される定数を表す。
【０１１６】
次に、図１の画像再構成部３を構成する補正値加算部３１の処理について説明する。
【０１１７】
補正値加算部３１は、図１５（Ａ）に示すように、ＤＣＴ係数抽出／逆量子化部１２から供給される、ビデオストリームから抽出された注目ブロックのＤＣＴ係数から、高域低減部２５から供給される、注目ブロックに対応する補正値としてのＤＣＴ係数を減算することで、注目ブロックのＤＣＴ係数を補正し、逆ＤＣＴ変換部３２に供給する。逆ＤＣＴ変換部３２では、このようにして補正値加算部３１から供給される補正後の注目ブロックのＤＣＴ係数が逆ＤＣＴ処理され、これにより、図１５（Ｂ）に示すような空間領域の８×８画素のブロックが得られる。
【０１１８】
なお、Ｐ，Ｂピクチャについては、ＭＰＥＧエンコードの際、元の画像と、その予測画像との差分（予測残差）がＤＣＴ処理され、さらに量子化されるから、その量子化の結果得られるＤＣＴ係数は、すべて０となることがあり、この場合、ビデオストリームに、ＤＣＴ係数は含められない。このような場合には、補正値加算部３１は、ブロックのＤＣＴ係数が、すべて０であるとして、その０から、補正値を減算して、補正後のＤＣＴ係数を得る。
【０１１９】
また、上述の場合には、補正値加算部３１において、注目ブロックのＤＣＴ係数を、補正値によって必ず補正するようにしたが、注目ブロックのＤＣＴ係数によって十分な画質が得られる場合（ブロック歪み等が目立たない場合等）には、補正を行わないようにすることが可能である。
【０１２０】
即ち、補正値加算部３１では、例えば、マクロブロック単位で、そのマクロブロックを構成するブロックの補正を行うかどうかを、サイドインフォメーション抽出部１３から供給される量子化スケール等に基づいて判定するようにすることが可能である。具体的には、例えば、ブロックの量子化スケールが所定の閾値以下（未満）の場合（量子化が細かい場合）には、ブロックの補正を行わず、ブロックの量子化スケールが所定の閾値より大（以上）の場合（量子化が粗い）には、ブロックの補正を行うようにすることが可能である。
【０１２１】
次に、図１６および図１７は、本件発明者が行ったシミュレーション結果を示している。なお、図１６および図１７の左側に四角形で囲んである部分は、それぞれの画像の一部分を拡大したものである。
【０１２２】
図１６は、従来のＭＰＥＧ方式によりエンコードした画像を、従来のＭＰＥＧ方式によりデコードしたデコード結果を示している。図１６の左側に示した拡大部分から明らかなように、顕著に、ブロック歪みが現れている。
【０１２３】
図１７は、従来のＭＰＥＧ方式によりエンコードした画像を、図１の画像処理装置によりデコードしたデコード結果を示している。図１７の左側に示した拡大部分から明らかなように、図１６においてモザイク状に現れているブロック歪みが十分に低減されていることが分かる。
【０１２４】
なお、ＭＰＥＧでは、Ｐ，Ｂピクチャにおいて、スキップマクロブロックが生じると、スキップマクロブロックには、ＤＣＴタイプが付加されないため、ブロック境界段差検出部２１やアクティビティ算出部２２等において、ブロックのライン構造が、フレームＤＣＴモードまたはフィールドＤＣＴモードのうちのいずれであるのか認識することができなくなる。このことは、ＭＰＥＧデコード部１１において得られたＩピクチャに動き補償を施して予測画像を得る場合に問題となるが、図１で説明したように、出力画像作成部３３で得られたＩピクチャに動き補償を施して予測画像を得る場合には、スキップマクロブロックについては、特に処理を行わずにスキップするだけで良いので、特に問題は生じない。
【０１２５】
以上のように、ブロックの境界部分の差分情報に、アクティビティに基づく重みを付すことにより補正値を得て、その補正値によって、ブロックのＤＣＴ係数を補正するようにしたので、特に平坦な部分で生じやすいブロック歪みを十分に低減した高画質の復号画像を得ることができる。
【０１２６】
さらに、エッジを含むブロック（アクティビティの高いブロック）で発生するモスキートノイズが、平坦なブロックに伝搬することを防止することができる。
【０１２７】
また、モスキートノイズが生じているブロック（アクティビティの高いブロック）の画素値は、図９（Ｂ）で説明したように、平坦なブロック（アクティビティの低いブロック）の画素値に近づくように補正されるため、モスキートノイズが生じているブロックの、そのモスキートノイズを目立たなくすることができる。
【０１２８】
さらに、図１の画像処理装置は、ブロック歪み等を除去するための補正値としてのＤＣＴ係数を求め、その補正値によって、ブロックのＤＣＴ係数を補正するため、逆ＤＣＴ処理を行うＭＰＥＧデコード処理との親和性が高く、ＭＰＥＧデコーダに組み込んで、リアルタイムで処理を行うようにすることが可能である。
【０１２９】
また、補正値は、ブロックの境界部分の差分情報、即ち、いわばブロック境界における歪みそのものから生成されるため、その補正値による補正の効果は、基本的に、ＭＰＥＧエンコード時における圧縮率の影響をほとんど受けない。
【０１３０】
さらに、高域低減部２５における補正値の修正や、補正値加算部３１におけるブロックの補正においては、量子化スケールその他のサイドインフォメーションを利用することで、適応的な歪み除去を行うことができる。即ち、例えば、補正値加算部３１では、上述したように、マクロブロックの量子化スケールを参照することにより、量子化誤差による画質の劣化の程度を推測して、その劣化の程度によって、ブロックの補正を行ったり、または行わないようにすることができる。
【０１３１】
次に、図１の画像処理装置では、逆ＤＣＴ変換部３２において、ＤＣＴ係数に対して逆ＤＣＴ処理を施すことにより、そのＤＣＴ係数を画素値に変換するようにしたが、ＤＣＴ係数から画素値への変換は、その他、例えば、本件出願人が先に提案したクラス分類適応処理を利用して行うことも可能である。
【０１３２】
クラス分類適応処理は、クラス分類処理と適応処理とからなり、クラス分類処理によって、データを、その性質に基づいてクラス分けし、各クラスごとに適応処理を施すものであり、適応処理は、以下のような手法のものである。
【０１３３】
即ち、適応処理では、例えば、ＤＣＴ係数と、所定のタップ係数との線形結合により、そのＤＣＴ係数に対応する元の画素値の予測値を求めることで、ＤＣＴ係数が、元の画素値に復号される。
【０１３４】
具体的には、例えば、いま、ある画像を教師データとするとともに、その画像を、ブロック単位でＤＣＴ処理して得られるＤＣＴ係数を生徒データとして、教師データである画素の画素値ｙの予測値Ｅ［ｙ］を、幾つかのＤＣＴ係数ｘ1，ｘ2，・・・の集合と、所定のタップ係数ｗ1，ｗ2，・・・の線形結合により規定される線形１次結合モデルにより求めることを考える。この場合、予測値Ｅ［ｙ］は、次式で表すことができる。
【０１３５】

【０１３６】
式（５）を一般化するために、タップ係数ｗjの集合でなる行列Ｗ、生徒データｘijの集合でなる行列Ｘ、および予測値Ｅ［ｙj］の集合でなる行列Ｙ’を、
【数１】

で定義すると、次のような観測方程式が成立する。
【０１３７】

ここで、行列Ｘの成分ｘijは、ｉ件目の生徒データの集合（ｉ件目の教師データｙiの予測に用いる生徒データの集合）の中のｊ番目の生徒データを意味し、行列Ｗの成分ｗjは、生徒データの集合の中のｊ番目の生徒データとの積が演算されるタップ係数を表す。また、ｙiは、ｉ件目の教師データを表し、従って、Ｅ［ｙi］は、ｉ件目の教師データの予測値を表す。なお、式（５）の左辺におけるｙは、行列Ｙの成分ｙiのサフィックスｉを省略したものであり、また、式（５）の右辺におけるｘ1，ｘ2，・・・も、行列Ｘの成分ｘijのサフィックスｉを省略したものである。
【０１３８】
そして、この観測方程式に最小自乗法を適用して、元の画素値ｙに近い予測値Ｅ［ｙ］を求めることを考える。この場合、教師データとなる真の画素値ｙの集合でなる行列Ｙ、および画素値ｙに対する予測値Ｅ［ｙ］の残差ｅの集合でなる行列Ｅを、
【数２】

で定義すると、式（６）から、次のような残差方程式が成立する。
【０１３９】

【０１４０】
この場合、元の画素値ｙに近い予測値Ｅ［ｙ］を求めるためのタップ係数ｗjは、自乗誤差
【数３】

を最小にすることで求めることができる。
【０１４１】
従って、上述の自乗誤差をタップ係数ｗjで微分したものが０になる場合、即ち、次式を満たすタップ係数ｗjが、元の画素値ｙに近い予測値Ｅ［ｙ］を求めるため最適値ということになる。
【０１４２】
【数４】

・・・（８）
【０１４３】
そこで、まず、式（７）を、タップ係数ｗjで微分することにより、次式が成立する。
【０１４４】
【数５】

・・・（９）
【０１４５】
式（８）および（９）より、式（１０）が得られる。
【０１４６】
【数６】

・・・（１０）
【０１４７】
さらに、式（７）の残差方程式における生徒データｘij、タップ係数ｗj、教師データｙi、および残差ｅiの関係を考慮すると、式（１０）から、次のような正規方程式を得ることができる。
【０１４８】
【数７】

・・・（１１）
【０１４９】
なお、式（１１）に示した正規方程式は、行列（共分散行列）Ａおよびベクトルｖを、
【数８】

で定義するとともに、ベクトルＷを、数１で示したように定義すると、式

で表すことができる。
【０１５０】
式（１１）における各正規方程式は、生徒データｘijおよび教師データｙiのセットを、ある程度の数だけ用意することで、求めるべきタップ係数ｗjの数Ｊと同じ数だけたてることができ、従って、式（１２）を、ベクトルＷについて解くことで（但し、式（１２）を解くには、式（１２）における行列Ａが正則である必要がある）、最適なタップ係数（ここでは、自乗誤差を最小にするタップ係数）ｗjを求めることができる。なお、式（１２）を解くにあたっては、例えば、掃き出し法（Gauss-Jordanの消去法）などを用いることが可能である。
【０１５１】
以上のようにして、最適なタップ係数ｗjを求めておく学習処理を行い、さらに、そのタップ係数ｗjを用い、式（５）により、元の画素値ｙに近い予測値Ｅ［ｙ］を求める予測処理を行うのが適応処理である。
【０１５２】
図１８は、以上のようなクラス分類適応処理により、ＤＣＴ係数を画素値に復号する、逆ＤＣＴ変換部３２の構成例を示している。
【０１５３】
補正値加算部３１が出力する８×８のブロックごとのＤＣＴ係数は、予測タップ抽出回路４１およびクラスタップ抽出回路４２に供給されるようになっている。
【０１５４】
予測タップ抽出回路４１は、そこに供給されるＤＣＴ係数のブロック（以下、適宜、ＤＣＴブロックという）に対応する画素値のブロック（この画素値のブロックは、現段階では存在しないが、仮想的に想定される）（以下、適宜、画素ブロックという）を、順次、注目画素ブロックとし、さらに、その注目画素ブロックを構成する各画素を、例えば、いわゆるラスタスキャン順に、順次、注目画素とする。さらに、予測タップ抽出回路４１は、注目画素の画素値を予測するのに用いるＤＣＴ係数を抽出し、予測タップとする。
【０１５５】
即ち、予測タップ抽出回路４１は、例えば、注目画素が属する画素ブロックに対応するＤＣＴブロックのすべてのＤＣＴ係数、即ち、８×８の６４個のＤＣＴ係数を、予測タップとして抽出する。従って、本実施の形態では、ある画素ブロックのすべての画素について、同一の予測タップが構成される。但し、予測タップは、注目画素ごとに、異なるＤＣＴ係数で構成することが可能である。
【０１５６】
予測タップ抽出回路４１において得られる、画素ブロックを構成する各画素についての予測タップ、即ち、６４画素それぞれについての６４セットの予測タップは、積和演算回路４５に供給される。但し、本実施の形態では、上述したように、画素ブロックのすべての画素について、同一の予測タップが構成されるので、実際には、１つの画素ブロックに対して、１セットの予測タップを、積和演算回路４５に供給すれば良い。
【０１５７】
クラスタップ抽出回路４２は、注目画素を、幾つかのクラスのうちのいずれかに分類するためのクラス分類に用いるＤＣＴ係数を抽出して、クラスタップとする。
【０１５８】
なお、ＭＰＥＧエンコードでは、画像が、画素ブロックごとにＤＣＴ処理されることから、ある画素ブロックに属する画素は、例えば、すべて同一のクラスにクラス分類することとする。従って、クラスタップ抽出回路４２は、ある画素ブロックの各画素については、同一のクラスタップを構成する。即ち、クラスタップ抽出回路４２は、例えば、予測タップ抽出回路４１における場合と同様に、注目画素が属する画素ブロックに対応するＤＣＴブロックの８×８個のすべてのＤＣＴ係数を、クラスタップとして抽出する。
【０１５９】
ここで、画素ブロックに属する各画素を、すべて同一のクラスにクラス分類するということは、その画素ブロックをクラス分類することと等価である。従って、クラスタップ抽出回路４２には、注目画素ブロックを構成する６４画素それぞれをクラス分類するための６４セットのクラスタップではなく、注目画素ブロックをクラス分類するための１セットのクラスタップを構成させれば良く、このため、クラスタップ抽出回路４２は、画素ブロックごとに、その画素ブロックをクラス分類するために、その画素ブロックに対応するＤＣＴブロックの６４個のＤＣＴ係数を抽出して、クラスタップとするようになっている。
【０１６０】
なお、予測タップやクラスタップを構成するＤＣＴ係数は、上述したパターンのものに限定されるものではない。
【０１６１】
クラスタップ抽出回路４２において得られる、注目画素ブロックのクラスタップは、クラス分類回路４３に供給されるようになっており、クラス分類回路４３は、クラスタップ抽出回路４２からのクラスタップに基づき、注目画素ブロックをクラス分類し、その結果得られるクラスに対応するクラスコードを出力する。
【０１６２】
ここで、クラス分類を行う方法としては、例えば、ADRC(Adaptive Dynamic Range Coding)等を採用することができる。
【０１６３】
ADRCを用いる方法では、クラスタップを構成するＤＣＴ係数が、ADRC処理され、その結果得られるADRCコードにしたがって、注目画素ブロックのクラスが決定される。
【０１６４】
なお、KビットADRCにおいては、例えば、クラスタップを構成するＤＣＴ係数の最大値MAXと最小値MINが検出され、DR=MAX-MINを、集合の局所的なダイナミックレンジとし、このダイナミックレンジDRに基づいて、クラスタップを構成するＤＣＴ係数がKビットに再量子化される。即ち、クラスタップを構成するＤＣＴ係数の中から、最小値MINが減算され、その減算値がDR/2^Kで除算（量子化）される。そして、以上のようにして得られる、クラスタップを構成するKビットの各ＤＣＴ係数を、所定の順番で並べたビット列が、ADRCコードとして出力される。従って、クラスタップが、例えば、１ビットADRC処理された場合には、そのクラスタップを構成する各ＤＣＴ係数は、最小値MINが減算された後に、最大値MAXと最小値MINとの平均値で除算され、これにより、各ＤＣＴ係数が１ビットとされる（２値化される）。そして、その１ビットのＤＣＴ係数を所定の順番で並べたビット列が、ADRCコードとして出力される。
【０１６５】
なお、クラス分類回路４３には、例えば、クラスタップを構成するＤＣＴ係数のレベル分布のパターンを、そのままクラスコードとして出力させることも可能であるが、この場合、クラスタップが、Ｎ個のＤＣＴ係数で構成され、各ＤＣＴ係数に、Ｋビットが割り当てられているとすると、クラス分類回路４３が出力するクラスコードの場合の数は、（２^N）^K通りとなり、ＤＣＴ係数のビット数Ｋに指数的に比例した膨大な数となる。
【０１６６】
従って、クラス分類回路４３においては、クラスタップの情報量を、上述のADRC処理や、あるいはベクトル量子化等によって圧縮してから、クラス分類を行うのが好ましい。
【０１６７】
ところで、本実施の形態では、クラスタップは、上述したように、６４個のＤＣＴ係数で構成される。従って、例えば、仮に、クラスタップを１ビットADRC処理することにより、クラス分類を行うこととしても、クラスコードの場合の数は、２⁶⁴通りという大きな値となる。
【０１６８】
そこで、本実施の形態では、クラス分類回路４３において、クラスタップを構成するＤＣＴ係数から、重要性の高い特徴量を抽出し、その特徴量に基づいてクラス分類を行うことで、クラス数を低減するようになっている。
【０１６９】
即ち、図１９は、図１８のクラス分類回路４３の構成例を示している。
【０１７０】
クラスタップは、電力演算回路５１に供給されるようになっており、電力演算回路５１は、クラスタップを構成するＤＣＴ係数を、幾つかの空間周波数帯域のものに分け、各周波数帯域の電力を演算する。
【０１７１】
即ち、電力演算回路５１は、クラスタップを構成する８×８個のＤＣＴ係数を、例えば、図２０に示すような４つの空間周波数帯域Ｓ0，Ｓ1，Ｓ2，Ｓ3に分割する。
【０１７２】
さらに、電力演算回路５１は、空間周波数帯域Ｓ0，Ｓ1，Ｓ2，Ｓ3それぞれについて、ＤＣＴ係数のＡＣ成分の電力（ＡＣ成分の２乗和）Ｐ0，Ｐ1，Ｐ2，Ｐ3を演算し、クラスコード生成回路５２に出力する。
【０１７３】
クラスコード生成回路５２は、電力演算回路５１からの電力Ｐ0，Ｐ1，Ｐ2，Ｐ3を、閾値テーブル記憶部５３に記憶された、対応する閾値ＴＨ０，ＴＨ１，ＴＨ２，ＴＨ３とそれぞれ比較し、それぞれの大小関係に基づいて、クラスコードを出力する。即ち、クラスコード生成回路５２は、電力Ｐ0と閾値ＴＨ０とを比較し、その大小関係を表す１ビットのコードを得る。同様に、クラスコード生成回路５２は、電力Ｐ1と閾値ＴＨ１、電力Ｐ2と閾値ＴＨ２、電力Ｐ3と閾値ＴＨ３を、それぞれ比較することにより、それぞれについて、１ビットのコードを得る。そして、クラスコード生成回路５２は、以上のようにして得られる４つの１ビットのコードを、例えば、所定の順番で並べることにより得られる４ビットのコード（従って、０乃至１５のうちのいずれかの値）を、注目画素ブロックのクラスを表すクラスコードとして出力する。従って、本実施の形態では、注目画素ブロックは、２⁴（＝１６）個のクラスのうちのいずれかにクラス分類されることになる。
【０１７４】
閾値テーブル記憶部５３は、空間周波数帯域Ｓ0乃至Ｓ3の電力Ｐ0乃至Ｐ3とそれぞれ比較する閾値ＴＨ０乃至ＴＨ３を記憶している。
【０１７５】
なお、上述の場合には、クラス分類処理に、ＤＣＴ係数のＤＣ成分が用いられないが、このＤＣ成分をも用いてクラス分類処理を行うことも可能である。
【０１７６】
図１８に戻り、以上のようなクラス分類回路４３が出力するクラスコードは、係数テーブル記憶部４４に、アドレスとして与えられる。
【０１７７】
係数テーブル記憶部４４は、上述したような教師データと生徒データとを用いた学習処理が行われることにより得られるタップ係数が登録された係数テーブルを記憶しており、クラス分類回路４３が出力するクラスコードに対応するアドレスに記憶されているタップ係数を積和演算回路４５に出力する。
【０１７８】
ここで、本実施の形態では、画素ブロックがクラス分類されるから、注目画素ブロックについて、１つのクラスコードが得られる。一方、画素ブロックは、本実施の形態では、８×８画素の６４画素で構成されるから、注目画素ブロックについて、それを構成する６４画素それぞれを復号するための６４セットのタップ係数が必要である。従って、係数テーブル記憶部４４には、１つのクラスコードに対応するアドレスに対して、６４セットのタップ係数が記憶されている。
【０１７９】
積和演算回路４５は、予測タップ抽出回路４１が出力する予測タップと、係数テーブル記憶部４４が出力するタップ係数とを取得し、その予測タップとタップ係数とを用いて、式（５）に示した線形予測演算（積和演算）を行い、その結果得られる注目画素ブロックの８×８画素の画素値を、対応するＤＣＴブロックの復号結果（逆ＤＣＴ結果）として、出力画像作成部３３（図１）に出力する。
【０１８０】
ここで、予測タップ抽出回路４１においては、上述したように、注目画素ブロックの各画素が、順次、注目画素とされるが、積和演算回路４５は、注目画素ブロックの、注目画素となっている画素の位置に対応した動作モード（以下、適宜、画素位置モードという）となって、処理を行う。
【０１８１】
即ち、例えば、注目画素ブロックの画素のうち、ラスタスキャン順で、ｉ番目の画素を、ｐiと表し、画素ｐiが、注目画素となっている場合、積和演算回路４５は、画素位置モード＃ｉの処理を行う。
【０１８２】
具体的には、上述したように、係数テーブル記憶部４４は、注目画素ブロックを構成する６４画素それぞれを復号するための６４セットのタップ係数を出力するが、そのうちの画素ｐiを復号するためのタップ係数のセットをＷiと表すと、積和演算回路４５は、動作モードが、画素位置モード＃ｉのときには、予測タップと、６４セットのタップ係数のうちのセットＷiとを用いて、式（５）の積和演算を行い、その積和演算結果を、画素ｐiの復号結果とする。
【０１８３】
次に、図２１のフローチャートを参照して、図１８の逆ＤＣＴ変換部３２の処理について説明する。
【０１８４】
補正値加算部３１（図１）が出力するブロックごとのＤＣＴ係数は、予測タップ抽出回路４１およびクラスタップ抽出回路４２において順次受信され、予測タップ抽出回路４１は、そこに供給されるＤＣＴ係数のブロック（ＤＣＴブロック）に対応する画素ブロックを、順次、注目画素ブロックとする。
【０１８５】
そして、クラスタップ抽出回路４２は、ステップＳ２１において、そこで受信したＤＣＴ係数の中から、注目画素ブロックをクラス分類するのに用いるものを抽出して、クラスタップを構成し、クラス分類回路４３に供給する。
【０１８６】
クラス分類回路４３は、ステップＳ２２において、クラスタップ抽出回路４２からのクラスタップを用いて、注目画素ブロックをクラス分類し、その結果得られるクラスコードを、係数テーブル記憶部４４に出力する。
【０１８７】
即ち、ステップＳ２２では、クラス分類回路４３（図１９）の電力演算回路５１が、クラスタップを構成する８×８個のＤＣＴ係数を、図２０に示した４つの空間周波数帯域Ｓ0乃至Ｓ3に分割し、それぞれの電力Ｐ0乃至Ｐ3を演算する。この電力Ｐ0乃至Ｐ3は、電力演算回路５１からクラスコード生成回路５２に出力される。
【０１８８】
クラスコード生成回路５２は、閾値テーブル記憶部５３から閾値ＴＨ０乃至ＴＨ３を読み出し、電力演算回路５１からの電力Ｐ0乃至Ｐ3それぞれと、閾値ＴＨ０乃至ＴＨ３それぞれとを比較し、それぞれの大小関係に基づいたクラスコードを生成する。
【０１８９】
以上のようにして得られるクラスコードは、クラス分類回路４３から係数テーブル記憶部４４に対して、アドレスとして与えられる。
【０１９０】
係数テーブル記憶部４４は、クラス分類回路４３からのアドレスとしてのクラスコードを受信すると、ステップＳ２３において、そのアドレスに記憶されている６４セットのタップ係数を読み出し、積和演算回路４５に出力する。
【０１９１】
そして、ステップＳ２４に進み、予測タップ抽出回路４１は、注目画素ブロックの画素のうち、ラスタスキャン順で、まだ、注目画素とされていない画素を、注目画素として、その注目画素の画素値を予測するのに用いるＤＣＴ係数を抽出し、予測タップとして構成する。この予測タップは、予測タップ抽出回路４１から積和演算回路４５に供給される。
【０１９２】
ここで、本実施の形態では、各画素ブロックごとに、その画素ブロックのすべての画素について、同一の予測タップが構成されるので、実際には、ステップＳ２４の処理は、注目画素ブロックについて、最初に注目画素とされる画素に対してだけ行えば、残りの６３画素に対しては、行う必要がない。
【０１９３】
積和演算回路４５は、ステップＳ２５において、ステップＳ２３で係数テーブル記憶部４４が出力する６４セットのタップ係数のうち、注目画素に対する画素位置モードに対応するタップ係数のセットを取得し、そのタップ係数のセットと、ステップＳ２４で予測タップ抽出回路４１から供給される予測タップとを用いて、式（５）に示した積和演算を行い、注目画素の画素値の復号値を得る。
【０１９４】
そして、ステップＳ２６に進み、予測タップ抽出回路４１は、注目画素ブロックのすべての画素を、注目画素として処理を行ったかどうかを判定する。ステップＳ２６において、注目画素ブロックのすべての画素を、注目画素として、まだ処理を行っていないと判定された場合、ステップＳ２４に戻り、予測タップ抽出回路４１は、注目画素ブロックの画素のうち、ラスタスキャン順で、まだ、注目画素とされていない画素を、新たに注目画素として、以下、同様の処理を繰り返す。
【０１９５】
また、ステップＳ２６において、注目画素ブロックのすべての画素を、注目画素として処理を行ったと判定された場合、即ち、注目画素ブロックのすべての画素の復号値が得られた場合、積和演算回路４５は、その復号値で構成される画素ブロック（復号ブロック）を、出力画像作成部３３（図１）に出力し、処理を終了する。
【０１９６】
なお、係数テーブル記憶部４４には、ある画像を教師データとするとともに、その画像をＭＰＥＧエンコードしたものを図１の画像処理装置の入力とした場合の補正値加算部３１の出力を生徒データとして、学習処理を行うことにより得られるタップ係数を記憶させておく必要がある。
【０１９７】
また、上述の場合には、ＤＣＴブロックのＤＣＴ係数のみを用いてクラス分類を行うようにしたが、クラス分類は、ＤＣＴ係数の他、例えば、そのＤＣＴブロックのアクティビティや、量子化スケール、Ｉ，Ｐ，Ｂピクチャの別等をも用いて行うことが可能である。この場合、学習処理においても、ＤＣＴ係数、アクティビティ、量子化スケール、Ｉ，Ｐ，Ｂピクチャの別等を用いてクラス分類を行う必要があるから、タップ係数が、ＤＣＴ係数、アクティビティ、量子化スケール、Ｉ，Ｐ，Ｂピクチャの別等に応じて求められることとなる。従って、そのようなタップ係数を用いて、ＤＣＴ係数を画素値に変換する場合には、その画素値で構成される画像の高画質化を図ることが可能となる。
【０１９８】
さらに、クラス分類適応処理は、ＤＣＴ係数を画素値に変換する場合の他、その逆に、画素値をＤＣＴ係数に変換する場合にも利用することができる。即ち、例えば、図１のＤＣＴ変換部２４でも、クラス分類適応処理を利用して、補正値算出部２３の出力をＤＣＴ係数に変換することが可能である。但し、この場合には、補正値算出部２３の出力を生徒データとするとともに、その出力をＤＣＴ処理して得られるＤＣＴ係数を教師データとして学習処理を行い、タップ係数を求めておく必要がある。
【０１９９】
次に、図２２は、本発明を適用した画像処理装置の他の実施の形態の構成例を示している。なお、図中、図１における場合と対応する部分については、同一の符号を付してあり、以下では、その説明は、適宜省略する。即ち、図２２の画像処理装置は、第１に、演算器４が新たに設けられ、第２に、入力画像分析部１にＤＣＴ変換部１４が新たに設けられ、第３に、歪補正値算出部２のブロック段差検出部２１またはアクティビティ算出部２２に替えて、ブロック段差検出部２６またはアクティビティ算出部２７がそれぞれ設けられている他は、図１における場合と同様に構成されている。
【０２００】
図１の画像処理装置では、ブロックの境界部分の差分情報、およびブロックのアクティビティを、いずれも、ＭＰＥＧデコード部１１が出力する画素値から求めるようになっていたが、図２２の画像処理装置では、差分情報およびアクティビティを、ＤＣＴ係数から求めるようになっている。
【０２０１】
即ち、ＤＣＴ変換部１４は、ＭＰＥＧデコード部１１で得られる画素値のブロックをＤＣＴ処理し、ＤＣＴ係数のブロックとして、演算器４に供給する。
【０２０２】
なお、ＤＣＴ変換部１４においても、上述したクラス分類適応処理を利用して、画素値をＤＣＴ係数に変換することが可能である。
【０２０３】
演算器４には、ＤＣＴ変換部１４が出力するＤＣＴ係数のブロックの他、ＤＣＴ係数抽出／逆量子化部１２が出力するＤＣＴ係数のブロックも供給されるようになっており、演算器４は、それらの２つのＤＣＴ係数のブロックを、必要に応じて加算して、ブロック境界段差検出部２６、アクティビティ算出部２７、および補正値加算部３１に供給する。
【０２０４】
即ち、Ｉピクチャについては、ＤＣＴ係数抽出／逆量子化部１２が出力するＤＣＴ係数のブロックは、元の画像の画素値のブロックをＤＣＴ処理したものであり、そのＤＣＴ係数を逆ＤＣＴ処理することにより、元の画像の画素値を得ることができるから、演算器４は、ＤＣＴ係数抽出／逆量子化部１２が出力するＤＣＴ係数のブロックを、そのまま、ブロック境界段差検出部２６、アクティビティ算出部２７、および補正値加算部３１に供給する。
【０２０５】
一方、ＰおよびＢピクチャについては、ＤＣＴ係数抽出／逆量子化部１２が出力するＤＣＴ係数のブロックは、元の画像の画素値のブロックと予測画像との差分（予測残差）をＤＣＴ処理したものであり、従って、そのＤＣＴ係数を逆ＤＣＴ処理したのでは、元の画像の画素値を得ることができない。即ち、ＰおよびＢピクチャについては、ＤＣＴ係数抽出／逆量子化部１２が出力するＤＣＴ係数のブロックと、予測画像をＤＣＴ処理して得られるＤＣＴ係数とを加算して得られるＤＣＴ係数を逆ＤＣＴ処理することにより、元の画像の画素値を得ることができる。そこで、この場合、ＤＣＴ変換部１４は、ＭＰＥＧデコード部１１で得られる予測画像をＤＣＴ処理し、その結果得られるＤＣＴ係数を、演算器４に供給する。そして、演算器４は、ＤＣＴ係数抽出／逆量子化部１２が出力するＤＣＴ係数のブロックと、ＤＣＴ変換部１４が出力するＤＣＴ係数とを加算し、その結果得られるＤＣＴ係数のブロックを、ブロック境界段差検出部２６、アクティビティ算出部２７、および補正値加算部３１に供給する。
【０２０６】
従って、演算器４が出力するＤＣＴ係数のブロックは、そのピクチャタイプによらず（Ｉ，Ｐ，Ｂピクチャのいずれであっても）、逆ＤＣＴ処理することにより、元の画像の画素値のブロックが得られるものとなっている。
【０２０７】
ここで、以上から、図２２では、ＭＰＥＧデコード部１１は、ＭＰＥＧデコード処理全体を行うものである必要はない。即ち、図２２のＭＰＥＧデコード部１１は、画像メモリ３４に記憶された参照画像を用いて動き補償を行うことにより、予測画像を生成することができるものであれば良い。
【０２０８】
ブロック境界段差検出部２６は、演算器４が出力するＤＣＴ係数のブロックについて、図１のブロック境界段差検出部２１における場合と同様にして、差分情報を求める。従って、ブロック境界段差検出部２６で求められる差分情報は、画素値の差分ではなく、ＤＣＴ係数の差分となっている。
【０２０９】
アクティビティ算出部２７は、演算器４が出力するＤＣＴ係数のブロックから、そのブロックのアクティビティを算出する。即ち、ブロックのアクティビティは、そのブロックのＤＣＴ係数のうちのＡＣ成分の２乗和との間に高い相関を有する。そこで、アクティビティ算出部２７は、演算器４が出力するＤＣＴ係数のブロックのＡＣ成分の２乗和を求め、そのブロックのアクティビティとして、補正値算出部２３および高域低減部２５に出力する。
【０２１０】
以下、図２２の画像処理装置では、図１における場合と同様にして、ブロック歪み等を低減した画像が復号される。
【０２１１】
但し、演算器４が補正値加算部３１に出力するＤＣＴ係数のブロックは、上述したように、そのピクチャタイプによらず、逆ＤＣＴ処理することにより、元の画像の画素値のブロックが得られるものとなっているため、Ｉピクチャのみならず、ＰピクチャやＢピクチャであっても、出力画像作成部３３において、予測画像を加算する必要はない。
【０２１２】
次に、図２３は、本発明を適用した伝送システムの一実施の形態の構成例を示している。
【０２１３】
この伝送システムは、送信装置６１および受信装置６２から構成され、送信装置６１から、ＭＰＥＧエンコードされた画像データが、例えば、公衆網や、インターネット、ＣＡＴＶ網、衛星回線等のネットワーク６３を介して、受信装置６２に伝送されるようになっている。
【０２１４】
送信装置６１には、画像データが入力されるようになっており、この画像データは、ＭＰＥＧエンコード部７１に供給される。ＭＰＥＧエンコード部７１は、そこに供給される画像データをＭＰＥＧエンコードし、その結果得られる符号化データを、入力画像分析部７２と、ＭＵＸ（マルチプレクサ）７４に供給する。
【０２１５】
入力画像分析部７２は、ＭＰＥＧエンコード部７１からの符号化データに対して、例えば、図１の入力画像分析部１と同様の処理を施し、その処理結果を、歪補正値算出部７３に供給する。歪補正値算出部７３は、入力画像分析部７２の出力を用いて、例えば、図１の歪補正値算出部２と同様の処理を行い、図１の画像再構成部３に出力されるのと同様の補正値を得て、ＭＵＸ７４に供給する。
【０２１６】
ＭＵＸ７４は、ＭＰＥＧエンコード部７１からの符号化データと、歪補正値算出部７３からの補正値とを多重化し、その結果得られる多重化データを、通信Ｉ／Ｆ(Interface)７５に供給する。通信Ｉ／Ｆ７５は、ＭＵＸ７４からの多重化データを、ネットワーク６３を介して、受信装置６２に伝送する。
【０２１７】
受信装置６２では、通信Ｉ／Ｆ８１が、上述のようにして、ネットワーク６３を介して、送信装置６１から伝送されてくる多重化データを受信し、ＤＭＵＸ（デマルチプレクサ）８２に供給する。ＤＭＵＸ８２は、通信Ｉ／Ｆ８１からの多重化データを、符号化データと補正値とに分離し、符号化データを入力画像分析部８３に、補正値を画像再構成部８４に、それぞれ供給する。
【０２１８】
入力画像分析部８３は、ＤＭＵＸ８２からの符号化データに対して、例えば、図１の入力画像分析部１と同様の処理を施し、その処理結果を、画像再構成部８４に供給する。
【０２１９】
画像再構成部８４は、操作部８５の操作にしたがい、入力画像分析部８３の出力を、ＤＭＵＸ８２からの補正値によって補正し、あるいは補正せずに処理し、復号画像を得て出力する。
【０２２０】
即ち、ユーザが、操作部８５を、高画質の画像を要求するように操作しなかった場合には、画像再構成部８４は、入力画像分析部８３の出力を、ＤＭＵＸ８２からの補正値によって補正せずに処理し、復号画像を得て出力する。従って、この場合、ユーザには、画像のアクティビティ、あるいは送信装置６１のＭＰＥＧエンコード部７１における画像の圧縮率等によっては、ブロック歪み等の目立つ復号画像が提供される。
【０２２１】
一方、ユーザが、操作部８５を、高画質の画像を要求するように操作した場合には、画像再構成部８４は、入力画像分析部８３の出力を、ＤＭＵＸ８２からの補正値によって補正して処理し、復号画像を得て出力する。従って、この場合、ユーザには、ブロック歪み等の低減された高画質の復号画像が提供される。
【０２２２】
さらに、この場合、画像再構成部８４は、補正値による補正を行った高画質の画像を提供した旨のメッセージ（以下、適宜、高画質画像提供メッセージという）を、通信Ｉ／Ｆ８１に供給する。通信Ｉ／Ｆ８１は、その高画質画像提供メッセージを、受信装置６２にあらかじめ付されている受信装置ＩＤ(Identification)とともに、ネットワーク６３を介して、送信装置６１に伝送する。
【０２２３】
送信装置６１では、通信Ｉ／Ｆ７５が、受信装置６２からの高画質画像提供メッセージおよび受信装置ＩＤを受信し、課金処理部７６に供給する。課金処理部７６は、受信装置ＩＤと、その受信装置ＩＤに対応するユーザに対する課金情報とを対応付けて管理しており、高画質画像提供メッセージおよび受信装置ＩＤを受信すると、その受信装置ＩＤに対応するユーザに対する課金処理を行い、課金情報を更新する。ユーザに対しては、この課金情報に基づいて、高画質の画像の提供に対する対価としての料金請求が行われる。
【０２２４】
なお、上述の場合には、操作部８５の操作に対応して、補正値による補正を行った高画質の画像を提供し、その後、課金処理を行うようにしたが、その他、例えば、ユーザから、あらかじめ料金を徴収しておき、料金を支払ったユーザにのみ、補正値の利用を許可して、その補正値による補正を行った高画質の画像を提供するようにすることも可能である。さらに、補正値による補正を行った高画質の画像の提供は、料金の徴収なしで行うことも可能である。また、補正値は、送信装置６１で生成するのではなく、受信装置６２で生成することも可能である。
【０２２５】
次に、上述した一連の処理は、ハードウェアにより行うこともできるし、ソフトウェアにより行うこともできる。一連の処理をソフトウェアによって行う場合には、そのソフトウェアを構成するプログラムが、汎用のコンピュータ等にインストールされる。
【０２２６】
そこで、図２４は、上述した一連の処理を実行するプログラムがインストールされるコンピュータの一実施の形態の構成例を示している。
【０２２７】
プログラムは、コンピュータに内蔵されている記録媒体としてのハードディスク１０５やＲＯＭ１０３に予め記録しておくことができる。
【０２２８】
あるいはまた、プログラムは、フロッピーディスク、CD-ROM(Compact Disc Read Only Memory)，MO(Magneto optical)ディスク，DVD(Digital Versatile Disc)、磁気ディスク、半導体メモリなどのリムーバブル記録媒体１１１に、一時的あるいは永続的に格納（記録）しておくことができる。このようなリムーバブル記録媒体１１１は、いわゆるパッケージソフトウエアとして提供することができる。
【０２２９】
なお、プログラムは、上述したようなリムーバブル記録媒体１１１からコンピュータにインストールする他、ダウンロードサイトから、ディジタル衛星放送用の人工衛星を介して、コンピュータに無線で転送したり、LAN(Local Area Network)、インターネットといったネットワークを介して、コンピュータに有線で転送し、コンピュータでは、そのようにして転送されてくるプログラムを、通信部１０８で受信し、内蔵するハードディスク１０５にインストールすることができる。
【０２３０】
コンピュータは、CPU(Central Processing Unit)１０２を内蔵している。CPU１０２には、バス１０１を介して、入出力インタフェース１１０が接続されており、CPU１０２は、入出力インタフェース１１０を介して、ユーザによって、キーボードや、マウス、マイク等で構成される入力部１０７が操作等されることにより指令が入力されると、それにしたがって、ROM(Read Only Memory)１０３に格納されているプログラムを実行する。あるいは、また、CPU１０２は、ハードディスク１０５に格納されているプログラム、衛星若しくはネットワークから転送され、通信部１０８で受信されてハードディスク１０５にインストールされたプログラム、またはドライブ１０９に装着されたリムーバブル記録媒体１１１から読み出されてハードディスク１０５にインストールされたプログラムを、RAM(Random Access Memory)１０４にロードして実行する。これにより、CPU１０２は、上述したフローチャートにしたがった処理、あるいは上述したブロック図の構成により行われる処理を行う。そして、CPU１０２は、その処理結果を、必要に応じて、例えば、入出力インタフェース１１０を介して、LCD(Liquid CryStal Display)やスピーカ等で構成される出力部１０６から出力、あるいは、通信部１０８から送信、さらには、ハードディスク１０５に記録等させる。
【０２３１】
ここで、本明細書において、コンピュータに各種の処理を行わせるためのプログラムを記述する処理ステップは、必ずしもフローチャートとして記載された順序に沿って時系列に処理する必要はなく、並列的あるいは個別に実行される処理（例えば、並列処理あるいはオブジェクトによる処理）も含むものである。
【０２３２】
また、プログラムは、１のコンピュータにより処理されるものであっても良いし、複数のコンピュータによって分散処理されるものであっても良い。さらに、プログラムは、遠方のコンピュータに転送されて実行されるものであっても良い。
【０２３３】
なお、本実施の形態では、ＭＰＥＧエンコードされた画像を復号する場合を対象としたが、本発明は、その他、ブロック単位で直交変換することによりエンコードされた画像を復号する場合に適応可能である。
【０２３４】
【発明の効果】
以上の如く、本発明によれば、ブロック歪み等を、容易に低減し、高画質の復号画像を得ることが可能となる。
【図面の簡単な説明】
【図１】本発明を適用した画像処理装置の第１実施の形態の構成例を示すブロック図である。
【図２】図１の画像処理装置の処理を説明するフローチャートである。
【図３】ブロック境界段差検出部２１の処理を説明するための図である。
【図４】ＭＰＥＧにおけるフレームＤＣＴモードとフィールドＤＣＴモードのマクロブロックを示す図である。
【図５】ブロック境界段差検出部２１の処理を説明するための図である。
【図６】ブロック境界段差検出部２１の処理を説明するための図である。
【図７】ブロック境界段差検出部２１の処理を説明するための図である。
【図８】ブロック境界段差検出部２１の処理を説明するための図である。
【図９】補正値算出部２３の処理を説明するための図である。
【図１０】補正値算出部２３の処理を説明するための図である。
【図１１】補正値算出部２３の処理を説明するための図である。
【図１２】高域低減部２５の処理を説明するための図である。
【図１３】高域低減部２５の処理による効果を説明するための図である。
【図１４】高域低減部２５の処理を説明するための図である。
【図１５】補正値加算部３１の処理を説明するための図である。
【図１６】本件発明者によるシミュレーション結果を示すディスプレイ上に表示された中間階調の写真である。
【図１７】本件発明者によるシミュレーション結果を示すディスプレイ上に表示された中間階調の写真である。
【図１８】逆ＤＣＴ変換部３２の構成例を示すブロック図である。
【図１９】クラス分類回路４３の構成例を示すブロック図である。
【図２０】電力演算回路５１の処理を説明するための図である。
【図２１】図１８の逆ＤＣＴ変換部３２の処理を説明するフローチャートである。
【図２２】本発明を適用した画像処理装置の第２実施の形態の構成例を示すブロック図である。
【図２３】本発明を適用した伝送システムの一実施の形態の構成例を示すブロック図である。
【図２４】本発明を適用したコンピュータの一実施の形態の構成例を示すブロック図である。
【符号の説明】
１ 入力画像分析部， ２ 歪補正値算出部， ３ 画像再構成部， ４ 演算器， １１ ＭＰＥＧデコード部， １２ ＤＣＴ係数抽出／逆量子化部， １３ サイドインフォメーション抽出部， １４ ＤＣＴ変換部， ２１ ブロック境界段差検出部， ２２ アクティビティ算出部， ２３ 補正値算出部，２４ ＤＣＴ変換部， ２５ 高域低減部， ２６ ブロック境界段差検出部， ２７ アクティビティ算出部， ３１ 補正値加算部， ３２ 逆ＤＣＴ変換部， ３３ 出力画像作成部， ３４ 画像メモリ， ３５ ピクチャ選択部， ４１ 予測タップ抽出回路， ４２ クラスタップ抽出回路， ４３クラス分類回路， ４４ 係数テーブル記憶部， ４５積和演算回路， ５１電力演算回路， ５２ クラスコード生成回路， ５３ 閾値テーブル記憶部， ６１ 送信装置， ６２ 受信装置， ６３ ネットワーク， ７１ ＭＰＥＧエンコード部， ７２ 入力画像分析部， ７３ 歪補正値算出部， ７４ＭＵＸ， ７５ 通信Ｉ／Ｆ， ７６ 課金処理部， ８１ 通信Ｉ／Ｆ， ８２ ＤＭＵＸ， ８３ 入力画像分析部， ８４ 画像再構成部， ８５ 操作部， １０１ バス， １０２ CPU， １０３ ROM， １０４ RAM， １０５ ハードディスク， １０６ 出力部， １０７ 入力部， １０８ 通信部， １０９ ドライブ， １１０ 入出力インタフェース， １１１ リムーバブル記録媒体[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus, an image processing method, and a recording medium, and in particular, image processing that makes it possible to easily reduce, for example, distortion generated in an MPEG (Moving Picture Experts Group) encoded image. The present invention relates to an apparatus, an image processing method, and a recording medium.
[0002]
[Prior art]
For example, a DCT (Discrete Cosine Transform) coefficient is quantized in a decoded image obtained by MPEG decoding image data encoded based on a standard such as MPEG1 or 2 (MPEG encoding). As a result, various distortions such as block distortion occur.
[0003]
That is, when the DCT coefficient is quantized, a part of the DCT coefficient is lost, the pattern of the block which is a unit for performing the DCT processing is simplified, and discontinuous steps due to the pixel values being greatly different in adjacent blocks. Although it appears, this is called block distortion.
[0004]
Block distortion is unavoidable in MPEG encoding in which block-unit pixels are quantized by DCT processing. As long as an MPEG-encoded video stream is decoded in accordance with the MPEG standard, it is not a little. appear. Note that the block distortion is particularly noticeable in a flat image, and appears prominently when the compression rate is increased (when the quantization scale is increased).
[0005]
Therefore, for example, in the re-published patent WO 98/54892, etc., the position of the block boundary is determined for the decoded image obtained by MPEG decoding, the step of the pixel value occurring at the boundary is detected, and A method of reducing the step by manipulating the pixel value near the block boundary is disclosed.
[0006]
[Problems to be solved by the invention]
However, in the MPEG encoder, with respect to P (forward prediction coding) pictures and B (bidirectional prediction coding) pictures, an already encoded and locally decoded image is used as a reference image, and motion compensation is performed on the reference image. The difference (prediction residual) from the predicted image obtained by the above is subjected to DCT processing and quantized. Therefore, even in an MPEG decoder, a P picture and a B picture are obtained by adding a prediction image obtained by applying motion compensation to an already decoded image as a reference image and a prediction residual thereof. Although it is decoded, the above-described method is troublesome because it is necessary to track the boundary of the block of the reference image that is moved by the motion compensation performed when obtaining the predicted image by performing motion detection on the reference image. It was.
[0007]
Furthermore, in the motion detection, it is necessary to obtain the same motion vector as the motion vector detected by the MPEG encoder, but even if motion detection is performed using a reference image, that is, an image decoded by the MPEG decoder, It is not always possible to detect the same motion vector as that detected by the MPEG encoder. In this case, a block image generated in the reference image is a predicted image obtained by performing motion compensation on the reference image. Will appear in the image decoded using.
[0008]
The present invention has been made in view of such a situation, and it is possible to easily reduce distortion generated in an MPEG encoded image or the like, thereby obtaining a high-quality decoded image. It is.
[0009]
[Means for Solving the Problems]
  The image processing apparatus of the present inventionThe image constituting the input stream is blocked, coefficient extraction means for extracting orthogonal transform coefficients for each block of a predetermined size, decoding means for decoding the stream, and orthogonality of each block included in the stream Corresponds to the difference between the extraction means for extracting the type of transformation, the pixel value at the boundary of the block constituting the image decoded by the decoding means, and the pixel value at the boundary of the adjacent block which is a block adjacent to the block Difference information calculation means for generating difference information with reference to the orthogonal transformation type extracted by the extraction means so as to obtain a difference between adjacent pixels, and an image decoded by the decoding means Activity detecting means for detecting an activity of the block to be performed, and the difference When the activity of the block is equal to the activity of the adjacent block based on the activity and the activity detected by the activity detection means, the boundary pixel which is the pixel adjacent to the adjacent block among the pixels of the block When a correction value is calculated so as to bring the pixel value and the pixel value of the adjacent block pixel adjacent to the boundary pixel closer to an intermediate pixel value, and there is a predetermined difference between the block activity and the adjacent block activity Among the pixel value of the boundary pixel of the block and the pixel value of the pixel of the adjacent block adjacent to the boundary pixel, the pixel value of the block with the larger activity is the pixel value of the smaller activity. A correction value calculating means for calculating a correction value that approaches the correction value, and the correction value calculating means The orthogonal transform unit that performs orthogonal transform processing on the correction value that is calculated as a pixel value difference, and the correction that corrects the orthogonal transform coefficient obtained by performing orthogonal transform processing by the orthogonal transform unit And a correcting means for obtaining a corrected orthogonal transform coefficient by adding or subtracting the orthogonal transform coefficient corrected by the correcting means to or from the orthogonal transform coefficient extracted by the coefficient extracting means, and the corrected orthogonal transform An inverse orthogonal transform unit that performs inverse orthogonal transform of the coefficients and converts the coefficient into a pixel value, an inverse orthogonal transform result, and a prediction image obtained by the decoding unit by performing motion compensation on a predetermined reference image, An image creating means for creating the image created by the image creating means as a reference image for decoding the stream by the decoding means. Used.
[0010]
  The image processing method of the present invention includes:The image constituting the input stream is blocked, the orthogonal transform coefficient for each block of a predetermined size is extracted, the stream is decoded by the decoding means, and the orthogonal transform type of each block included in the stream is set. The type of the orthogonal transform in which the difference information corresponding to the difference between the pixel value at the boundary of the block constituting the extracted and decoded image and the pixel value at the boundary of the adjacent block that is a block adjacent to the block is extracted , Generating the difference between adjacent pixels and detecting the activity of the block constituting the decoded image, and based on the difference information and the detected activity, adjacent to the activity of the block If the block activity is equal, the pixel of the block A correction value is calculated such that the pixel value of a boundary pixel that is a pixel adjacent to the adjacent block and the pixel value of the adjacent block adjacent to the boundary pixel approach an intermediate pixel value; When there is a predetermined difference between the activity and the activity of the adjacent block, the larger of the pixel value of the boundary pixel of the block and the pixel value of the pixel of the adjacent block adjacent to the boundary pixel By calculating a correction value that brings the pixel value of the block closer to the pixel value of the smaller activity, subjecting the calculated correction value expressed as the difference between the pixel values to orthogonal transform processing, and performing orthogonal transform processing The correction is performed by correcting the obtained orthogonal transform coefficient and adding or subtracting the corrected orthogonal transform coefficient to the extracted orthogonal transform coefficient. An orthogonal transform coefficient is obtained, the corrected orthogonal transform coefficient is subjected to inverse orthogonal transform, converted into a pixel value, an inverse orthogonal transform result, and a predicted image obtained by the decoding unit by performing motion compensation on a predetermined reference image, and Is used, and the created image is used by the decoding means as a reference image for decoding the stream.
[0011]
  The recording medium of the present invention isThe image constituting the input stream is blocked, the orthogonal transform coefficient for each block of a predetermined size is extracted, the stream is decoded by the decoding means, and the orthogonal transform type of each block included in the stream is set. The type of the orthogonal transform in which the difference information corresponding to the difference between the pixel value at the boundary of the block constituting the extracted and decoded image and the pixel value at the boundary of the adjacent block that is a block adjacent to the block is extracted , Generating the difference between adjacent pixels and detecting the activity of the block constituting the decoded image, and based on the difference information and the detected activity, adjacent to the activity of the block If the block activity is equal, the pixel of the block A correction value is calculated such that the pixel value of a boundary pixel that is a pixel adjacent to the adjacent block and the pixel value of the adjacent block adjacent to the boundary pixel approach an intermediate pixel value; When there is a predetermined difference between the activity and the activity of the adjacent block, the larger of the pixel value of the boundary pixel of the block and the pixel value of the pixel of the adjacent block adjacent to the boundary pixel By calculating a correction value that brings the pixel value of the block closer to the pixel value of the smaller activity, subjecting the calculated correction value expressed as the difference between the pixel values to orthogonal transform processing, and performing orthogonal transform processing The correction is performed by correcting the obtained orthogonal transform coefficient and adding or subtracting the corrected orthogonal transform coefficient to the extracted orthogonal transform coefficient. An orthogonal transform coefficient is obtained, the corrected orthogonal transform coefficient is subjected to inverse orthogonal transform, converted into a pixel value, an inverse orthogonal transform result, and a predicted image obtained by the decoding unit by performing motion compensation on a predetermined reference image, and A program for causing a computer to execute processing used by the decoding means as a reference image for decoding the stream is recorded.
[0012]
  In the image processing apparatus, the image processing method, and the recording medium of the present invention,The image constituting the input stream is blocked, orthogonal transform coefficients for each block of a predetermined size are extracted, the stream is decoded by a decoding means, and the type of orthogonal transform of each block included in the stream Is extracted. Further, the difference information corresponding to the difference between the pixel value at the boundary of the block constituting the decoded image and the pixel value at the boundary of the adjacent block which is a block adjacent to the block is the type of the extracted orthogonal transform. It is generated by referring to and obtaining a difference between adjacent pixels. If the activity of the block constituting the decoded image is detected, and the activity of the block and the activity of the adjacent block are equal based on the difference information and the detected activity, A correction value is calculated so that a pixel value of a boundary pixel that is a pixel adjacent to the adjacent block and a pixel value of a pixel of the adjacent block adjacent to the boundary pixel are approximated to an intermediate pixel value, and the activity of the block If there is a predetermined difference between the activity of the adjacent block and the adjacent block, the block having the larger activity of the pixel value of the boundary pixel of the block and the pixel value of the pixel of the adjacent block adjacent to the boundary pixel A correction value that brings the pixel value closer to the pixel value with the smaller activity It is issued. The correction value expressed as the difference between the calculated pixel values is subjected to the orthogonal transform process, the orthogonal transform coefficient obtained by performing the orthogonal transform process is corrected, and the corrected orthogonal transform coefficient is extracted. By adding or subtracting to the orthogonal transform coefficient, a corrected orthogonal transform coefficient is obtained, the corrected orthogonal transform coefficient is subjected to inverse orthogonal transform, converted into a pixel value, an inverse orthogonal transform result, and a predetermined reference image An image is created using the predicted image obtained by the decoding means by performing motion compensation. Further, the created image is used by the decoding means as a reference image for decoding the stream.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows a configuration example of an embodiment of an image processing apparatus to which the present invention is applied.
[0014]
This image processing apparatus includes an input image analysis unit 1, a distortion correction value calculation unit 2, and an image reconstruction unit 3, and decodes image data encoded in accordance with standards such as MPEG1 and 2, for example. It has become.
[0015]
The input image analysis unit 1 includes an MPEG decoding unit 11, a DCT coefficient extraction / inverse quantization unit 12, and a side information extraction unit 13, which includes MPEG encoding the image data. The obtained video stream (hereinafter referred to as MPEG video stream as appropriate) is input.
[0016]
In addition to the MPEG video stream input to the input image analysis unit 1, the MPEG decoding unit 11 is supplied with a decoded image with reduced distortion stored in the image memory 34 of the image reconstruction unit 3 as a reference image. It has become so. The MPEG decoding unit 11 decodes the MPEG video stream supplied thereto, and supplies the decoded image obtained as a result to the block boundary step detection unit 21 and the activity calculation unit 22 of the distortion correction value calculation unit 2. In addition, the MPEG decoding unit 11 predicts an image used for decoding a P picture and a B picture among I (intra-encoded) pictures, P pictures, and B pictures (an already decoded picture is used as a reference picture and a reference picture thereof). The image obtained by performing motion compensation on the image according to the motion vector is supplied to the output image creation unit 33 of the image reconstruction unit 3.
[0017]
Note that the MPEG decoding unit 11 uses, as a reference image, a decoded image with reduced distortion supplied from the image memory 34 as described above, instead of a decoded image obtained therein, with respect to the reference image. The predicted image is obtained by performing motion compensation according to the motion vector included in the MPEG video stream.
[0018]
Therefore, since motion compensation in the MPEG decoding unit 11 is performed using a decoded image with reduced distortion as a reference image, block distortion occurring in the reference image can be obtained by performing motion compensation on the reference image. It can be prevented from appearing in an image decoded using the predicted image. That is, in the MPEG decoding unit 11, when a predicted image is generated by motion compensation using an MPEG decoded I or P picture as a reference image, block distortion occurring in the I or P picture used as the reference image is Move by motion compensation. Furthermore, the prediction residual of the P or B picture is added to the prediction image obtained by the motion compensation, and the P or B picture is decoded. The decoded P or B picture includes the prediction image. In addition to distortion in, distortion in the prediction residual also appears, making it difficult to detect the distortion itself. In contrast, the MPEG decoding unit 11 uses the decoded image with reduced distortion supplied from the image memory 34 as described above as a reference image, and the reference image is included in the MPEG video stream. Since motion compensation is performed according to the motion vector, the problem of distortion detection as described above does not occur.
[0019]
The DCT coefficient extraction / inverse quantization unit 12 is supplied with an MPEG video stream input to the input image analysis unit 1. The DCT coefficient extraction / inverse quantization unit 12 extracts and inverse-quantizes the quantized DCT coefficient from the MPEG video stream supplied thereto, and obtains the DCT coefficient for each 8 × 8 pixel block obtained as a result. , And supplied to the correction value adding unit 31 of the image reconstruction unit 3.
[0020]
The side information extraction unit 13 is supplied with an MPEG video stream input to the input image analysis unit 1. The side information extraction unit 13 extracts, for example, side information such as a quantization scale, a quantization table, and a DCT type included in the MPEG video stream supplied thereto, and the distortion correction value calculation unit 2 and the image reconstruction unit 3 is supplied. That is, for example, the quantization scale is supplied to the correction value addition unit 31 of the image reconstruction unit 3, and the DCT type is the block boundary step detection unit 21 and activity calculation unit 22 of the distortion correction value calculation unit 2, and the image reconstruction unit. The correction value is added to the correction value adding unit 31 of the configuration unit 3.
[0021]
The distortion correction value calculation unit 2 includes a block level detection unit 21, an activity calculation unit 22, a correction value calculation unit 23, a DCT conversion unit 24, and a high frequency reduction unit 25. A correction value used for correcting the inversely quantized DCT coefficient output from the inverse quantization unit 12 is calculated.
[0022]
That is, the block boundary step detection unit 21 sequentially sets each block as a target block in the block-unit decoded image from the MPEG decoding unit 11, sets the pixel value constituting the boundary of the target block, and blocks adjacent to the target block. The difference between the pixel value and the pixel value constituting the boundary is calculated by referring to the DCT type supplied from the side information unit 13 and supplied to the correction value calculation unit 23 as difference information on the boundary of the target block.
[0023]
The activity calculation unit 22 calculates the activity of the block of interest for the decoded image of the block unit from the MPEG decoding unit 11, and the DCT type supplied from the side information unit 13 for the activity of the block adjacent to the block of interest. The calculation is performed by referring to the correction value calculation unit 23 and the high frequency reduction unit 25.
[0024]
The correction value calculation unit 23 performs weighting on the difference information of the block boundary from the block boundary step detection unit 21 based on the activity from the activity calculation unit 22, and the weighted result is subjected to DCT coefficient extraction / inverse quantization. The correction value used for correction of the inversely quantized DCT coefficient output from the unit 12 is supplied to the DCT conversion unit 24.
[0025]
The DCT conversion unit 24 performs DCT processing on the correction value from the correction value calculation unit 23 and supplies it to the high frequency reduction unit 25. That is, in FIG. 1, the correction value output from the correction value calculation unit 23 is a spatial region value (pixel value difference), so the DCT conversion unit 24 performs DCT processing on the correction value of the spatial region. Thus, it is converted into a DCT coefficient as a correction value in the frequency domain and supplied to the high frequency reduction unit 25.
[0026]
The high frequency reduction unit 25 corrects the DCT coefficient as the correction value from the DCT conversion unit 24 based on the activity from the activity calculation unit 22, and uses the corrected correction value as the correction value of the image reconstruction unit 3. It supplies to the addition part 31.
[0027]
The image reconstruction unit 3 includes a correction value addition unit 31, an inverse DCT conversion unit 32, an output image creation unit 33, an image memory 34, and a picture selection unit 35. The DCT coefficient extraction / inverse quantum of the input image analysis unit 1 The block unit DCT coefficient output from the conversion unit 12 is corrected based on the correction value output from the distortion correction value calculation unit 2, and the image is decoded using the corrected DCT coefficient.
[0028]
That is, the correction value addition unit 31 adds (subtracts) the DCT coefficient of the block output from the DCT coefficient extraction / inverse quantization unit 12 and the DCT coefficient as the correction value output from the high frequency reduction unit 25, Thus, the DCT coefficient of the block is corrected and supplied to the inverse DCT transform unit 32.
[0029]
The inverse DCT conversion unit 32 performs inverse DCT processing on the corrected DCT coefficient of the block output from the correction value addition unit 31, decodes the pixel value block, and supplies the decoded block to the output image creation unit 33.
[0030]
The output image creation unit 33 refers to the DCT type output from the side information unit 13 for the predicted image output from the MPEG decoding unit 11 for the block of pixel values from the inverse DCT conversion unit 32 as necessary. In this way, a block of the decoded image is obtained. Further, the output image creation unit 33 supplies the block of the decoded image to the picture selection unit 35 and also supplies it to the image memory 34 as necessary.
[0031]
The image memory 34 stores a decoded image output by the output image creation unit 33 that serves as a reference image for creating a predicted image of P or B picture. The image stored in the image memory 34 is read out as necessary, supplied to the MPEG decoding unit 11 as a reference image, and supplied to the picture selection unit 35. In MPEG1 and 2, the reference picture used to create the predicted picture is an I picture or a P picture. In this embodiment, the picture memory 34 stores the I picture and the P picture. Only the B picture is not stored.
[0032]
The picture selection unit 35 selects and outputs either the image output from the output image creation unit 33 or the image read from the image memory 34 as necessary. That is, in MPEG, since the image decoding (encoding) order and the display order do not match, the picture selection unit 35 selects the image output from the output image creation unit 33 or the image stored in the image memory 34. Select the one that should be displayed now and output it.
[0033]
In the image processing apparatus of FIG. 1 (the same applies to the image processing apparatus of FIG. 22 described later), in fact, a timing adjustment memory and a synchronization signal are required to cope with the delay in each block. However, in order to avoid the figure from becoming complicated, the illustration is omitted.
[0034]
Next, image decoding processing by the image processing apparatus of FIG. 1 will be described with reference to the flowchart of FIG.
[0035]
When the MPEG video stream is supplied to the input image analysis unit 1, the input image analysis unit 1 determines in step S1 whether or not the current decoding target is an I picture.
[0036]
If it is determined in step S1 that the current decoding target is an I picture, the process proceeds to step S2, where the MPEG decoding unit 11 performs MPEG decoding on the I picture, and the block boundary step detection unit 21 and the activity. It supplies to the calculation part 22. Further, in step S2, the DCT coefficient extraction / inverse quantization unit 12 extracts the quantized DCT coefficient of the I picture decoded by the MPEG decoding unit 11 from the MPEG video stream, performs inverse quantization, and corrects the correction value. While being supplied to the adding unit 31, the side information unit 13 extracts the quantization scale and DCT type for the I picture decoded by the MPEG decoding unit 11 from the MPEG video stream. The quantization scale is supplied to the correction value adding unit 31, and the DCT type is supplied to the block boundary step detecting unit 21, the activity calculating unit 22, and the correction value adding unit 31.
[0037]
On the other hand, if it is determined in step S1 that the current decoding target is not an I picture, that is, if the decoding target is a P or B picture, the process proceeds to step S3, where the MPEG decoding unit 11 A reference image is read out from 34 and motion compensation is performed, so that a predicted image of the P or B picture that is the current decoding target is generated, supplied to the output image generating unit 33, and the process proceeds to step S4.
[0038]
In step S4, the MPEG decoding unit 11 performs inverse quantization and inverse DCT processing on the quantized DCT coefficient of the P or B picture that is to be decoded, and obtains the prediction residual of the pixel value obtained as a result. By adding the predicted images obtained in step S3, the P or B picture that is currently the decoding target is decoded. Then, the MPEG decoding unit 11 supplies the decoding result of the P or B picture to the block level detection unit 31 and the activity calculation unit 22. Further, in step S4, the DCT coefficient extraction / inverse quantization unit 12 extracts the quantized DCT coefficient of the P or B picture decoded by the MPEG decoding unit 11 from the MPEG video stream, and performs inverse quantization. While being supplied to the correction value adding unit 31, the side information unit 13 extracts a quantization scale and a DCT type for the P or B picture decoded by the MPEG decoding unit 11 from the MPEG video stream. The quantization scale is supplied to the correction value adding unit 31, and the DCT type is supplied to the block boundary step detecting unit 21, the activity calculating unit 22, and the correction value adding unit 31.
[0039]
After the processing in step S2 or S4, the process proceeds to step S5, and the block boundary step detection unit 21 sequentially sets the blocks of the decoded image from the MPEG decoding unit 11 as the attention block, and configures the boundary of the attention block. The difference between the pixel value and the pixel value constituting the boundary of the block adjacent to the target block is calculated with reference to the DCT type supplied from the side information unit 13, and corrected as difference information of the target block boundary. The value is supplied to the value calculation unit 23, and the process proceeds to step S6.
[0040]
In step S6, the activity calculation unit 22 refers to the DCT type supplied from the side information unit 13 for the activity of the target block in the decoded image from the MPEG decoding unit 11 and the activity of the block adjacent to the target block. The calculated value is supplied to the correction value calculation unit 23 and the high frequency reduction unit 25, and the process proceeds to step S7.
[0041]
In step S <b> 7, the correction value calculation unit 23 assigns a weight based on the activity from the activity calculation unit 22 to the block boundary difference information from the block boundary step detection unit 21, and uses the weighted result as the correction value. Is supplied to the DCT conversion unit 24, and the process proceeds to step S8. In step S8, the DCT conversion unit 24 performs DCT processing on the correction value from the correction value calculation unit 23, supplies the correction value to the high frequency reduction unit 25, and proceeds to step S9. In step S9, the high frequency reduction unit 25 corrects higher-order DCT coefficients as correction values from the DCT conversion unit 24 based on the activity from the activity calculation unit 22, and corrects the corrected values. The value is supplied to the correction value adding unit 31, and the process proceeds to step S10.
[0042]
In step S10, the image reconstruction unit 3 determines whether or not the current decoding target is an I picture. If it is determined that the image is an I picture, the process proceeds to step S11, where the correction value adding unit 31 The DCT coefficient of the block output from the DCT coefficient extraction / inverse quantization unit 12 and the DCT coefficient as the correction value output from the high frequency reduction unit 25 are added, that is, here, the DCT coefficient extraction / inverse quantization is added. The DCT coefficient as the correction value output from the high frequency reduction unit 25 is subtracted from the DCT coefficient of the block output from the unit 12, whereby the DCT coefficient of the block is corrected and supplied to the inverse DCT transform unit 32.
[0043]
In step S <b> 12, the inverse DCT conversion unit 32 performs inverse DCT processing on the corrected DCT coefficient of the block output from the correction value addition unit 31, decodes the block of the pixel value, and supplies the block to the output image creation unit 33. .
[0044]
On the other hand, if it is determined in step S10 that the decoding target is not an I picture, that is, if the decoding target is a P or B picture, the process proceeds to step S13, and the correction value adding unit 31 Similarly to the case in S11, the DCT coefficient of the block output by the DCT coefficient extraction / inverse quantization unit 12 (in this case, this DCT coefficient is obtained by converting the prediction residual that is the difference between the P or B picture and the predicted image to the DCT. And the DCT coefficient as the correction value output from the high frequency reduction unit 25 are added, thereby correcting the DCT coefficient of the block, and the inverse DCT This is supplied to the conversion unit 32.
[0045]
In step S14, the inverse DCT transform unit 32 performs inverse DCT processing on the corrected DCT coefficient of the block output by the correction value addition unit 31 in the same manner as in step S12, and the pixel value (prediction residual) block Is supplied to the output image creation unit 33, and the process proceeds to step S15.
[0046]
In step S <b> 15, the output image creating unit 33 obtains a decoded image by adding the predicted image supplied from the MPEG decoding unit 11 to the prediction residual block from the inverse DCT transform unit 32. That is, in this case, the decoding target is an image of a P or B picture in which a prediction residual that is a difference value from the prediction image is encoded. Therefore, the block from the inverse DCT transform unit 32 is a block of the prediction residual. It has become. Therefore, in step S15, a predicted image is added to the prediction residual, thereby decoding a P or B picture image.
[0047]
After the processing in step S12 or S15, the process proceeds to step S16, and the output image creation unit 33 determines whether the image that has just been decoded is a B picture.
[0048]
If it is determined in step S16 that the decoded image is not a B picture, that is, if the decoded image is an I or P picture that can be a reference image, the process proceeds to step S17, and the output image creation unit 33 The decoded I or P picture is supplied to and stored in the image memory 34, and the process proceeds to step S18. In step S18, the picture selection unit 35 selects and outputs the previously decoded I or P picture stored in the image memory 34. And it returns to step S1 and repeats the same process hereafter.
[0049]
If it is determined in step S16 that the decoded image is a B picture that is not a reference image, the output image creation unit 33 supplies the decoded B picture to the picture selection unit 35, and Proceed to step S19. In step S19, the picture selection unit 35 selects and outputs the B picture output from the output image creation unit 33. And it returns to step S1 and repeats the same process hereafter.
[0050]
Next, processing in the distortion correction value calculation unit 2 and the image reconstruction unit 3 in FIG. 1 will be described in detail.
[0051]
As described above, the block boundary step detecting unit 21 constituting the distortion correction calculating unit 2 calculates the difference between the pixel value constituting the boundary of the target block and the pixel value constituting the boundary of the block adjacent to the target block. Calculate and output as difference information of the boundary of the target block.
[0052]
That is, as shown in FIG. 3, the block boundary step detection unit 21 calculates a pixel value between 28 pixel values forming the boundary of the target block and pixel values forming block boundaries adjacent to the target block. Calculate the difference. Note that, for example, the upper left pixel value of the target block is adjacent to both the upper block of the target block and the left block. In such a case, the difference between each block is calculated. The That is, for the upper left pixel value, two differences are calculated: the difference between the pixel value of the upper adjacent block and the difference between the pixel value of the left adjacent block. The same applies to the upper right, lower left, and lower right pixels of the block of interest. Therefore, the block boundary level detector 21 calculates 32 (= 8 × 4) difference values for the pixels constituting the boundary of the block of interest.
[0053]
Here, in MPEG, an image is divided into 16 × 16 pixel macroblocks, which are further divided into 8 × 8 pixel blocks and subjected to DCT processing and quantization processing in units of blocks. A frame DCT mode and a field DCT mode are prepared as modes for performing DCT processing by dividing a macroblock into blocks.
[0054]
That is, for example, as shown in FIG. 4A, when the top line of a 16 × 16 pixel macroblock is the first line, in the frame DCT mode, when attention is paid to the luminance signal, the macroblock is As shown in FIG. 4B, the block is divided into four blocks each composed of a frame in which odd lines and even lines are alternately arranged. On the other hand, in the field DCT mode, when attention is paid to the luminance signal, as shown in FIG. 4 (C), the macroblock is composed of four fields including only odd lines and even lines. Divided into blocks.
[0055]
The frame DCT mode and the field DCT mode can be switched in units of macroblocks. Refer to the DCT type which is one of the side information included in the MPEG video stream to determine which mode is used for MPEG encoding. Can be recognized.
[0056]
Therefore, there is no problem when the DCT type of the macroblock to which the target block belongs (hereinafter referred to as the target macroblock as appropriate) and the macroblock adjacent to the target macroblock are the same, but the DCT type is different. The pixel of the block adjacent to the pixel constituting the boundary of the block of interest (hereinafter referred to as the boundary pixel as appropriate) is not originally adjacent to the pixel (finally not in the state of the image of one frame) Pixel).
[0057]
Therefore, the block boundary step detection unit 21 refers to the DCT type supplied from the side information unit 13 as described above in order to calculate the difference between the boundary pixel and the originally adjacent pixel. Recognize whether the block and the macroblock adjacent to it are MPEG-encoded in frame DCT mode or field DCT mode, and calculate the difference from the boundary pixel of the target block as follows, for example It is supposed to be.
[0058]
That is, as shown in FIG._NAnd macroblocks adjacent to the top, bottom, and left are respectively MBs._U, MB_D, MB_LIt expresses. Furthermore, for example, macroblock MB_NAmong the four blocks constituting the i-th block in the raster scan order,_NiIt shall be expressed as Furthermore, attention macroblock MB_NThe four blocks B that make up_N-1, B_N-2, B_N-3, B_N-4For example, the upper left block B_N-1Is a block of interest.
[0059]
And now, for example, as shown in FIG._N, Macroblock MB_U, MB_D, MB_LAre both in the frame DCT mode, the target block B_N-1For each of the upper, lower, left, and right boundary pixels, the difference from the pixels adjacent to the upper, lower, left, and right is taken.
[0060]
That is, attention block B_N-1For each of the border pixels above, the target macroblock MB_NMacroblock MB adjacent on top of_UBlock B at the lower left of_U-3A difference from each of the pixels in the eighth line is taken. For the lower boundary pixel, the target macroblock MB_NBlock B at the lower left of_N-3The difference from each of the pixels in the first line is taken. Further, for the left boundary pixel, the target macroblock MB_NMacroblock MB adjacent to the left of_LBlock B in the upper right of_L-2Differences from the last (eighth column) pixels of the first to eighth lines are taken. For the right boundary pixel, the target macroblock MB_NBlock B in the upper right of_N-2Differences from the first (first column) pixels of the first to eighth lines are taken.
[0061]
Next, for example, as shown in FIG._NIs the field DCT mode and the macroblock MB_U, MB_D, MB_LHowever, if both are in the frame DCT mode, first, the target block B_N-1Block B adjacent to the right of_N-2Is the attention macroblock MB_NTherefore, the target block B_N-1And adjacent block B to the right_N-2Has the same line structure, so the target block B_N-1For the right boundary pixel, the difference from the right adjacent pixel is taken. That is, for the right boundary pixel, the target macroblock MB_NBlock B in the upper right of_N-2Differences from the first pixels of the first to eighth lines are taken.
[0062]
In addition, attention block B_N-1For each of the boundary pixels above, the adjacent pixel above it, that is, the target macroblock MB_NMacroblock MB adjacent on top of_UBlock B at the lower left of_U-3The difference from each of the pixels in the seventh line is taken.
[0063]
Furthermore, attention block B_N-1As for the left boundary pixel of each of the first to fourth line boundary pixels, the target macroblock MB_NMacroblock MB next to the left_LBlock B in the upper right that constitutes_L-2Are compared with the last pixels of the first, third, fifth, and seventh lines, and for each of the boundary pixels of the fifth to eighth lines, the target macroblock MB_NMacroblock MB next to the left_LBlock B on the bottom right_L-4Differences from the last pixels of the first, third, fifth and seventh lines are taken.
[0064]
In addition, attention block B_N-1For each lower boundary pixel, the target macroblock MB_NMacroblock MB adjacent under_DBlock B in the upper left_D-1The difference from each of the pixels constituting the first line is taken.
[0065]
Next, for example, as shown in FIG._NIs the frame DCT mode and the macroblock MB_U, MB_D, MB_LHowever, if both are in the field DCT mode, as in the case of FIG._N-1And adjacent block B to the right_N-2Has the same line structure, so the target block B_N-1For the right boundary pixel, the difference from the right adjacent pixel is taken. That is, for the right boundary pixel, the target macroblock MB_NBlock B in the upper right of_N-2Differences from the first pixels of the first to eighth lines are taken.
[0066]
In addition, attention block B_N-1For each lower boundary pixel, the adjacent pixel below it, that is, the target macroblock MB_NBlock B at the lower left of_N-3The difference from each of the pixels in the first line is taken.
[0067]
Furthermore, attention block B_N-1For the left boundary pixel, the boundary pixel of the first, third, fifth, and seventh lines is the target macroblock MB._NMacroblock MB next to the left_LBlock B in the upper right that constitutes_L-2The difference from each of the last pixels of the first to fourth lines is taken, and for each of the boundary pixels of the second, fourth, sixth, and eighth lines, the target macroblock MB_NMacroblock MB next to the left_LBlock B on the bottom right_L-4Differences from the first pixels of the first to fourth lines are taken.
[0068]
In addition, attention block B_N-1For each of the border pixels above, the target macroblock MB_NMacroblock MB adjacent on top of_UBlock B on the bottom left_U-3A difference from each of the pixels constituting the eighth line is taken.
[0069]
Next, for example, as shown in FIG._NAnd macroblock MB_U, MB_D, MB_LHowever, if both are in the field DCT mode, the target block B is the same as in FIG._N-1And adjacent block B to the right_N-2Has the same line structure, so the target block B_N-1For the right boundary pixel, the difference from the right adjacent pixel is taken. That is, for the right boundary pixel, the target macroblock MB_NBlock B in the upper right of_N-2Differences from the first pixels of the first to eighth lines are taken.
[0070]
In addition, attention block B_N-1For each of the boundary pixels above, the adjacent pixel above it, that is, the target macroblock MB_NMacroblock MB adjacent on top of_UBlock B at the lower left of_U-1A difference from each of the pixels in the eighth line is taken.
[0071]
Furthermore, attention block B_N-1For each left border pixel, the target macroblock MB_NMacroblock MB next to the left_LBlock B in the upper right that constitutes_L-2Differences from the last pixels of the first to eighth lines are taken.
[0072]
MB_U, MB_L, MB_OAre different DCT types, a combination of FIGS. 5 and 7 or FIGS.
[0073]
In addition, attention block B_N-1For each lower boundary pixel, the target macroblock MB_NMacroblock MB adjacent under_DBlock B in the upper left_D-1The difference from each of the pixels constituting the first line is taken.
[0074]
As described above, the block boundary step detection unit 21 recognizes the line structure of the target macroblock and the blocks constituting the macroblock adjacent to the target macroblock by referring to the DCT type supplied from the side information unit 13. For the boundary pixel of the block, the difference between the boundary pixel and the originally adjacent pixel is calculated to obtain difference information.
[0075]
Next, processing of the activity calculation unit 22 constituting the distortion correction value calculation unit 2 in FIG. 1 will be described.
[0076]
The activity calculation unit 22 calculates the activity of the pixel value block supplied from the MPEG code unit 11. That is, if the pixel value of the i-th row and j-th column of the block is expressed as p (i, j), the activity calculation unit 22 determines that the activity V of the block is_actIs calculated according to the following equation, for example.
[0077]

However, in Formula (1), Σ represents a summation when i and j are changed from 1 to I and J, and I and J are the numbers of lines and columns of the pixels constituting the block, respectively. To express. Therefore, in this embodiment, I and J are both 8.
[0078]
The activity calculation unit 22 calculates the activity for the block of interest, and also calculates the activities of blocks adjacent to the block of interest in the vertical and horizontal directions. However, the activity calculation unit 22 recognizes the line structure constituting the block by referring to the DCT type supplied from the side information extraction unit 13 with respect to the blocks adjacent to the upper, lower, left, and right sides of the target block. Activity is calculated for a block with the same structure as the block.
[0079]
That is, for example, as shown in FIG._NAnd macroblock MB_U, MB_D, MB_LHowever, if both are in the frame DCT mode, the activity calculating unit 22_N-1The activity of blocks adjacent to the top, bottom, left, and right of the block B_N-1Block B adjacent on top of_U-31st to 8th line, block B adjacent below_N-31st to 8th lines, block B adjacent to the left_L-21st to 8th lines, block B adjacent to the right_N-2From the first to eighth lines, the calculation is performed according to the equation (1).
[0080]
Also, for example, as shown in FIG._NIs the field DCT mode and the macroblock MB_U, MB_D, MB_LHowever, if both are in the frame DCT mode, the activity calculating unit 22_N-1The activity of blocks adjacent to the top, bottom, left, and right of the block B_N-1Macroblock MB adjacent on top of_UBlock B of_U-14 odd lines and block B_U-34 odd lines below, macroblock MB adjacent below_DBlock B of_D-14 odd lines and block B_D-34 odd lines, macroblock MB adjacent to the left_LBlock B of_L-24 odd lines and block B_L-44 odd lines, right adjacent block B_N-2From the first to eighth lines, the calculation is performed according to the equation (1).
[0081]
Further, for example, as shown in FIG._NIs the frame DCT mode and the macroblock MB_U, MB_D, MB_LHowever, if both are in the field DCT mode, the activity calculating unit 22_N-1The activity of blocks adjacent to the top, bottom, left, and right of the block B_N-1Macroblock MB adjacent on top of_UBlock B of_U-15th to 8th lines and block B_U-35th to 8th lines, block B adjacent below_N-31st to 8th lines, macroblock MB adjacent to the left_LBlock B of_L-2First to fourth lines and block B_L-4First to fourth lines, block B adjacent to the right_N-2From the first to eighth lines, the calculation is performed according to the equation (1).
[0082]
Also, for example, as shown in FIG._N, Macroblock MB_U, MB_D, MB_LHowever, if both are in the field DCT mode, the activity calculating unit 22_N-1The activity of blocks adjacent to the top, bottom, left, and right of the block B_N-1Macroblock MB adjacent on top of_UBlock B of_U-11st to 8th lines, macroblock MB adjacent below_DBlock B of_D-11st to 8th lines, block B adjacent to the left_L-21st to 8th lines, block B adjacent to the right_N-2From the first to eighth lines, the calculation is performed according to the equation (1).
[0083]
Next, the process of the correction value calculation unit 23 constituting the distortion correction value calculation unit 2 in FIG. 1 will be described.
[0084]
The correction value calculation unit 23 processes the difference information of the boundary of the target block supplied from the block boundary level detection unit 21 based on the target block from the activity calculation unit 22 and the activities of blocks adjacent to the upper, lower, left, and right sides. Thus, the correction value is calculated.
[0085]
That is, when the activity of the block of interest and the block adjacent thereto (hereinafter referred to as an adjacent block as appropriate) are substantially equal, the correction value calculation unit 23 determines the pixel value of the boundary pixel of the block of interest and the boundary pixel thereof. A correction value is calculated so as to bring the pixel value of the adjacent block adjacent to to the intermediate value of the two pixel values.
[0086]
Specifically, for example, if the difference between the pixel value of the boundary pixel of the target block and the pixel value of the adjacent block adjacent to the boundary pixel, which is currently used as difference information, is called the difference pixel value, For example, as shown in FIG. 9A, the value calculation unit 23 adjoins the boundary pixel of the pixel value of the boundary pixel of the target block by 7/16 of the difference pixel value in order to reduce block distortion. A correction value that approximates the pixel value of the adjacent block is calculated.
[0087]
As for the pixel value of the adjacent block adjacent to the boundary pixel, when the adjacent block becomes the target block, the pixel value is corrected by 7/16 of the difference pixel value in the same manner as described above. A correction value is obtained. Further, in this case, the pixel value of the boundary pixel of the target block and the pixel value of the adjacent block adjacent to the boundary pixel are corrected so as to approach each other by 7/16 of the difference pixel value. There is a difference of 1/8 (= 2/16) of the difference pixel value between the two corrected pixel values, which is a certain amount between the boundary pixels of adjacent blocks. This is because if there is a difference, the blocks are connected smoothly and look natural.
[0088]
On the other hand, when there is a certain difference in activity between the block of interest and the adjacent block, the correction value calculation unit 23, or the pixel value of the boundary pixel of the block of interest or the pixel of the adjacent block adjacent to the boundary pixel Among the values, a correction value is calculated so that the pixel value of the block with the larger activity is brought closer to the pixel value of the block with the smaller activity.
[0089]
That is, when there is a certain difference in activity between the block of interest and the adjacent block, the block with the larger activity is likely to contain mosquito noise, so the mosquito noise is It is necessary to prevent propagation (influence) to the block (the block with the smaller activity).
[0090]
Therefore, when there is a certain difference in the activity between the target block and the adjacent block and the activity of the target block is larger, the correction value calculation unit 23, for example, as shown in FIG. A correction value that brings the pixel value of the boundary pixel of the target block closer to the pixel value of the adjacent block adjacent to the boundary pixel by the difference pixel value, that is, the pixel value of the boundary pixel of the target block A correction value that matches the pixel value of an adjacent block adjacent to the pixel is obtained.
[0091]
The correction value calculation unit 23 uses the difference information that is the difference pixel value from the block boundary step detection unit 21 and the activity from the activity calculation unit 22 as follows, for example, as follows. Ask.
[0092]
That is, for example, as shown in FIG. 10, if the activity of the block of interest is represented by C and the activities of blocks adjacent to the top, bottom, left, and right are represented by A, E, B, and D, respectively, the correction value The calculation unit 23 calculates the weights Gu, Gd, Gl, and Gr assigned to the difference pixel values as the difference information of the upper, lower, left, and right boundaries of the block of interest according to the following expression.
[0093]

[0094]
Then, as shown in FIG. 11, the correction value calculation unit 23 multiplies the difference pixel values as the difference information of the upper, lower, left, and right boundaries of the block of interest by Gu, Gd, Gl, and Gr to obtain a correction value.
[0095]
In addition, the correction value calculation part 23 attaches | subjects the average value of weight Gu and Gl as a weight about the difference pixel value of the upper left among difference information, for example. Similarly, the correction value calculation unit 23 sets the average value of the weights Gu and Gr for the uppermost difference pixel value, and sets the average value of the weights Gl and Gd for the lowermost difference pixel value. For the lower difference pixel value, an average value of the weights Gr and Gd is assigned as a weight.
[0096]
Therefore, the correction value calculation unit 23 obtains the correction value of the 8 × 8 block in which the weighted difference pixel value is arranged in the boundary portion and 0 is arranged in the remaining portion.
[0097]
Further, in the above-described case, the correction value is obtained by assigning a weight based only on the activity to the difference information. However, for example, the adjacent value adjacent to the target block is the target of the correction value. When it belongs to a macroblock that is not a macroblock, and the quantization block is different from the target block and the adjacent block, the difference information is obtained by adding a weight based on the activity and the quantization scale, etc. Is possible.
[0098]
Next, the processing of the high frequency reduction unit 25 constituting the distortion correction value calculation unit 2 in FIG. 1 will be described.
[0099]
In the high frequency reduction unit 25, the correction value of the 8 × 8 block in the spatial region obtained by the correction value calculation unit 23 is subjected to DCT processing as shown in FIG. Eight frequency domain DCT coefficients are supplied.
[0100]
The high frequency reduction unit 25 reduces the higher-order DCT coefficients as correction values supplied thereto, that is, for example, as shown in FIG. The correction value is corrected by setting the fifth to eighth rows and the fifth to eighth columns of the x8 DCT coefficients to 0, and supplied to the correction value adding unit 31. .
[0101]
That is, if the correction value output by the DCT conversion unit 24 is used as it is for correcting the DCT coefficient of the block output by the DCT coefficient extraction / inverse quantization unit 12, only the pixel values near the boundary of the block are corrected. As a result, for example, the pixel value at the boundary portion of a block having a steep step as shown in FIG. 13A is corrected as shown in FIG. Therefore, although the block boundary becomes less conspicuous than before the correction, the block structure appears in the decoded image because the step at the boundary portion of the block is still steep.
[0102]
Therefore, the high frequency reduction unit 25 corrects the correction value by reducing the higher order DCT coefficient as the correction value. When such a corrected correction value is used for correcting the DCT coefficient of the block output by the DCT coefficient extraction / inverse quantization unit 12, not only the vicinity of the boundary of the block but also the internal pixel value thereof As a result, for example, the pixel value at the boundary portion of the block having a steep step as shown in FIG. 13A is corrected as shown in FIG. Therefore, since the level difference at the boundary portion of the block becomes smooth, the appearance of the block structure in the decoded image can be sufficiently reduced.
[0103]
Next, as a method of correcting the correction value so as to reduce the higher-order DCT coefficient as the correction value in the high-frequency reduction unit 25, as shown in FIG. A method other than the method of setting the DCT coefficient to 0 can be employed.
[0104]
That is, for example, as shown in FIG. 14A, the correction values are not corrected to 0 in the 5th to 8th rows and the 5th to 8th columns but to any i-th row. It is possible to correct the above and the i th column or more to zero. In this case, the variable i can be determined based on the activity of the block of interest, the quantization scale used for quantization of the block of interest, and the like. That is, for example, when the activity of the target block is large, only the higher-order DCT coefficient is corrected to 0, and when the activity is small, the lower-order DCT coefficient is also corrected to 0. can do. Further, for example, when the quantization scale of the block of interest is small, only higher order DCT coefficients are corrected to 0, and when the quantization scale is large, lower order DCT coefficients are also set to 0. It can be corrected.
[0105]
Note that when the order of the DCT coefficient to be corrected to 0 is determined based on the activity of the block of interest, the quantization scale, or the like, a lower limit value of the order can be set. That is, for example, the DCT coefficients in the fourth row and below and the fourth column and below can be set so as not to be corrected to zero. In this case, the DCT coefficient to be corrected to 0 changes in the ranges of the fifth to eighth rows and the fifth to eighth columns.
[0106]
Further, as shown in FIG. 14A, the correction value is obtained by correcting the same row and column or more of 8 × 8 DCT coefficients to 0, and correcting the row and column to be corrected to 0. It can also be determined independently.
[0107]
That is, for example, as shown in FIG. 14B, when the activity of blocks adjacent to the left and right of the target block is large, only the higher-order DCT coefficients in the horizontal direction are corrected to zero. If the activity of blocks adjacent to the left and right is small, the lower DCT coefficient in the horizontal direction can be corrected to zero. Also, for example, when the activity of adjacent blocks above and below the target block is large, only the higher-order DCT coefficient in the vertical direction is corrected to 0, and the activity of the vertically adjacent blocks is small. The lower DCT coefficient in the vertical direction can be corrected to zero.
[0108]
Further, for example, when the quantization scale of blocks adjacent to the left and right of the target block is small, only the higher-order DCT coefficients in the horizontal direction are corrected to 0, and the blocks adjacent to the left and right When the quantization scale is large, the lower-order DCT coefficients in the horizontal direction can be corrected to zero. Also, for example, when the quantization scale of blocks adjacent above and below the target block is small, only the higher-order DCT coefficients in the vertical direction are corrected to 0, and the quantum of the blocks adjacent to the top and bottom thereof is corrected. When the conversion scale is large, the lower-order DCT coefficient in the vertical direction can be corrected to zero.
[0109]
Further, in the case shown in FIGS. 14A and 14B, it is corrected to 0 based on the sum of activities of the block of interest and the blocks adjacent to the top and bottom and the blocks adjacent to the left and right. It is also possible to determine the order of the DCT coefficients.
[0110]
Further, the DCT coefficient as the correction value can be corrected by, for example, applying a predetermined weight to the DCT coefficient.
[0111]
That is, for example, as shown in FIG. 14C, out of the 8 × 8 DCT coefficients as correction values, the DCT coefficient in the v-th row and u-th column is expressed as F (u, v) and after correction. If the DCT coefficient as the correction value is expressed as F ′ (u, v), the DCT coefficient as the correction value can be corrected according to one of the following two equations, for example.
[0112]

However, in Expression (3), a, b, and c represent constants set based on the block of interest, the activities of blocks adjacent to the top, bottom, left, and right, the quantization scale, or the like.
[0113]
Further, the DCT coefficient as the correction value can be corrected by performing weighting based on the quantization table used for quantization of the block of interest, for example.
[0114]
That is, if the quantization table q used for quantization of the block of interest is, for example, as shown in FIG. 14D, the value of the vth row and uth column of the quantization table q is When expressed as q (u, v), the DCT coefficient as the correction value can be corrected according to the following equation, for example.
[0115]

However, in Expression (4), a represents a constant set based on the activity of the block of interest or the like, as in Expression (3).
[0116]
Next, the processing of the correction value adding unit 31 constituting the image reconstruction unit 3 in FIG. 1 will be described.
[0117]
As shown in FIG. 15A, the correction value adding unit 31 is supplied from the DCT coefficient extracting / inverse quantizing unit 12 from the DCT coefficient of the target block extracted from the video stream, from the high frequency reducing unit 25. By subtracting the supplied DCT coefficient as the correction value corresponding to the target block, the DCT coefficient of the target block is corrected and supplied to the inverse DCT transform unit 32. In the inverse DCT conversion unit 32, the DCT coefficient of the target block after correction supplied from the correction value addition unit 31 in this way is subjected to inverse DCT processing, whereby 8D in the spatial region as shown in FIG. A block of x8 pixels is obtained.
[0118]
For P and B pictures, the difference (prediction residual) between the original image and the predicted image is subjected to DCT processing and further quantized during MPEG encoding, so that the DCT obtained as a result of the quantization is used. The coefficients may all be 0, in which case no DCT coefficients are included in the video stream. In such a case, assuming that the DCT coefficients of the block are all 0, the correction value adding unit 31 subtracts the correction value from the 0 to obtain a corrected DCT coefficient.
[0119]
In the above-described case, the correction value adding unit 31 always corrects the DCT coefficient of the block of interest with the correction value. However, when sufficient image quality is obtained with the DCT coefficient of the block of interest (block distortion or the like) For example, when the image is not conspicuous, it is possible not to perform the correction.
[0120]
That is, the correction value adding unit 31 determines, for example, on a macroblock basis whether or not to correct the blocks constituting the macroblock based on the quantization scale supplied from the side information extracting unit 13 or the like. It is possible to Specifically, for example, when the quantization scale of a block is equal to or less than a predetermined threshold value (when the quantization is fine), the block is not corrected and the quantization scale of the block is larger than the predetermined threshold value. In the case of (above) (the quantization is coarse), it is possible to perform block correction.
[0121]
Next, FIG. 16 and FIG. 17 show the simulation results performed by the present inventors. In addition, the part enclosed by the rectangle on the left side of FIG. 16 and FIG. 17 is an enlarged part of each image.
[0122]
FIG. 16 shows a decoding result obtained by decoding an image encoded by the conventional MPEG method by the conventional MPEG method. As is clear from the enlarged portion shown on the left side of FIG. 16, noticeably block distortion appears.
[0123]
FIG. 17 shows a decoding result obtained by decoding an image encoded by the conventional MPEG method by the image processing apparatus of FIG. As is clear from the enlarged portion shown on the left side of FIG. 17, it can be seen that the block distortion appearing in a mosaic pattern in FIG. 16 is sufficiently reduced.
[0124]
Note that in MPEG, if a skip macroblock occurs in a P or B picture, the DCT type is not added to the skip macroblock, so the block boundary step detection unit 21 or the activity calculation unit 22 has a block line structure. Therefore, it is impossible to recognize whether the mode is the frame DCT mode or the field DCT mode. This is a problem when motion prediction is performed on the I picture obtained in the MPEG decoding unit 11 to obtain a predicted image, but as described in FIG. 1, the I picture obtained in the output image creating unit 33 is used. When a predicted image is obtained by performing motion compensation on the skip macroblock, there is no particular problem because it is only necessary to skip the skip macroblock without performing any particular processing.
[0125]
As described above, the correction value is obtained by adding the weight based on the activity to the difference information of the boundary portion of the block, and the DCT coefficient of the block is corrected by the correction value. It is possible to obtain a high-quality decoded image with sufficiently reduced block distortion that easily occurs.
[0126]
Furthermore, it is possible to prevent mosquito noise generated in a block including an edge (a block with high activity) from propagating to a flat block.
[0127]
Further, as described with reference to FIG. 9B, the pixel value of the block in which mosquito noise occurs (block with high activity) is corrected so as to approach the pixel value of a flat block (block with low activity). Therefore, the mosquito noise of the block in which the mosquito noise is generated can be made inconspicuous.
[0128]
Further, the image processing apparatus of FIG. 1 obtains a DCT coefficient as a correction value for removing block distortion and the like, and an MPEG decoding process for performing an inverse DCT process to correct the DCT coefficient of the block by the correction value. Can be incorporated into an MPEG decoder and processed in real time.
[0129]
Further, since the correction value is generated from the difference information of the block boundary portion, that is, the distortion at the block boundary itself, the correction effect by the correction value basically has the influence of the compression rate at the time of MPEG encoding. I hardly receive it.
[0130]
Further, in the correction of the correction value in the high frequency reducing unit 25 and the correction of the block in the correction value adding unit 31, adaptive distortion removal can be performed by using the quantization scale and other side information. That is, for example, as described above, the correction value adding unit 31 estimates the degree of image quality degradation due to the quantization error by referring to the quantization scale of the macroblock, and the block value is determined according to the degree of the degradation. Corrections can be made or not.
[0131]
Next, in the image processing apparatus of FIG. 1, the inverse DCT conversion unit 32 performs inverse DCT processing on the DCT coefficient to convert the DCT coefficient into a pixel value. For example, the conversion into can also be performed using, for example, the classification adaptation process previously proposed by the present applicant.
[0132]
Class classification adaptive processing consists of class classification processing and adaptive processing. Data is classified into classes based on their properties by class classification processing, and adaptive processing is performed for each class. It is of the technique like.
[0133]
That is, in the adaptive processing, for example, by obtaining a predicted value of the original pixel value corresponding to the DCT coefficient by linear combination of the DCT coefficient and a predetermined tap coefficient, the DCT coefficient is decoded into the original pixel value. Is done.
[0134]
Specifically, for example, a certain image is used as teacher data, and a DCT coefficient obtained by performing DCT processing on the image in units of blocks is used as student data. Consider that E [y] is obtained by a linear linear combination model defined by a linear combination of a set of several DCT coefficients x1, x2,... And predetermined tap coefficients w1, w2,. . In this case, the predicted value E [y] can be expressed by the following equation.
[0135]

[0136]
In order to generalize Equation (5), a matrix W composed of a set of tap coefficients wj, a matrix X composed of a set of student data xij, and a matrix Y ′ composed of a set of predicted values E [yj]
[Expression 1]

Then, the following observation equation holds.
[0137]

Here, the component xij of the matrix X means j-th student data in the i-th set of student data (a set of student data used for prediction of the i-th teacher data yi). The component wj represents a tap coefficient by which a product with the jth student data in the student data set is calculated. Yi represents the i-th teacher data, and therefore E [yi] represents the predicted value of the i-th teacher data. Note that y on the left side of the equation (5) is obtained by omitting the suffix i of the component yi of the matrix Y, and x1, x2,... On the right side of the equation (5) are also components xij of the matrix X. The suffix i is omitted.
[0138]
Then, it is considered to apply the least square method to this observation equation to obtain a predicted value E [y] close to the original pixel value y. In this case, a matrix Y composed of a set of true pixel values y serving as teacher data and a matrix E composed of a set of residuals e of predicted values E [y] for the pixel values y are
[Expression 2]

From the equation (6), the following residual equation is established.
[0139]

[0140]
In this case, the tap coefficient wj for obtaining the predicted value E [y] close to the original pixel value y is a square error.
[Equation 3]

Can be obtained by minimizing.
[0141]
Therefore, when the above-mentioned square error differentiated by the tap coefficient wj is 0, that is, the tap coefficient wj satisfying the following equation is called an optimum value to obtain the predicted value E [y] close to the original pixel value y. It will be.
[0142]
[Expression 4]

... (8)
[0143]
Therefore, first, the following equation is established by differentiating the equation (7) by the tap coefficient wj.
[0144]
[Equation 5]

... (9)
[0145]
From equations (8) and (9), equation (10) is obtained.
[0146]
[Formula 6]

(10)
[0147]
Further, considering the relationship among student data xij, tap coefficient wj, teacher data yi, and residual ei in the residual equation of equation (7), the following normal equation can be obtained from equation (10). .
[0148]
[Expression 7]

(11)
[0149]
In addition, the normal equation shown in Expression (11) has a matrix (covariance matrix) A and a vector v,
[Equation 8]

And the vector W is defined as shown in Equation 1,

Can be expressed as
[0150]
Each normal equation in equation (11) can be set to the same number as the number J of tap coefficients wj to be obtained by preparing a certain number of sets of student data xij and teacher data yi. By solving the equation (12) for the vector W (however, in order to solve the equation (12), the matrix A in the equation (12) needs to be regular), the optimal tap coefficient (here, the square error) The tap coefficient) wj that minimizes. In solving the equation (12), for example, a sweeping method (Gauss-Jordan elimination method) or the like can be used.
[0151]
As described above, the learning process for obtaining the optimum tap coefficient wj is performed, and further, using the tap coefficient wj, a predicted value E [y] close to the original pixel value y is obtained by Expression (5). The adaptive process performs the prediction process.
[0152]
FIG. 18 shows a configuration example of the inverse DCT transform unit 32 that decodes DCT coefficients into pixel values by the class classification adaptation process as described above.
[0153]
The DCT coefficients for each 8 × 8 block output from the correction value adding unit 31 are supplied to the prediction tap extraction circuit 41 and the class tap extraction circuit 42.
[0154]
The prediction tap extraction circuit 41 has a block of pixel values corresponding to a block of DCT coefficients (hereinafter referred to as a DCT block as appropriate) supplied thereto (this block of pixel values does not exist at this stage, but virtually (Hereinafter, referred to as “pixel block” as appropriate) is sequentially set as a target pixel block, and each pixel constituting the target pixel block is sequentially set as a target pixel in, for example, a so-called raster scan order. Further, the prediction tap extraction circuit 41 extracts a DCT coefficient used for predicting the pixel value of the target pixel and sets it as a prediction tap.
[0155]
That is, the prediction tap extraction circuit 41 extracts, for example, all DCT coefficients of the DCT block corresponding to the pixel block to which the pixel of interest belongs, that is, 64 × 8 × 8 DCT coefficients, as prediction taps. Therefore, in this embodiment, the same prediction tap is configured for all the pixels in a certain pixel block. However, the prediction tap can be configured with different DCT coefficients for each pixel of interest.
[0156]
The prediction taps obtained by the prediction tap extraction circuit 41 for each pixel constituting the pixel block, that is, 64 sets of prediction taps for each of the 64 pixels, are supplied to the product-sum operation circuit 45. However, in the present embodiment, as described above, since the same prediction tap is configured for all the pixels of the pixel block, in practice, one set of prediction taps is assigned to one pixel block. What is necessary is just to supply to the product-sum operation circuit 45.
[0157]
The class tap extraction circuit 42 extracts a DCT coefficient used for class classification for classifying the pixel of interest into one of several classes, and sets it as a class tap.
[0158]
In MPEG encoding, since an image is subjected to DCT processing for each pixel block, all pixels belonging to a certain pixel block are classified into the same class, for example. Accordingly, the class tap extraction circuit 42 configures the same class tap for each pixel of a certain pixel block. That is, the class tap extraction circuit 42 extracts all 8 × 8 DCT coefficients of the DCT block corresponding to the pixel block to which the pixel of interest belongs as a class tap, for example, as in the prediction tap extraction circuit 41. .
[0159]
Here, classifying all the pixels belonging to a pixel block into the same class is equivalent to classifying the pixel block. Therefore, the class tap extraction circuit 42 is configured to configure one set of class taps for classifying the pixel block of interest, instead of 64 sets of class taps for classifying each of the 64 pixels constituting the pixel block of interest. Therefore, for each pixel block, the class tap extraction circuit 42 extracts 64 DCT coefficients of the DCT block corresponding to the pixel block in order to classify the pixel block. It is supposed to be.
[0160]
In addition, the DCT coefficient which comprises a prediction tap and a class tap is not limited to the thing of the pattern mentioned above.
[0161]
The class tap of the pixel block of interest obtained in the class tap extraction circuit 42 is supplied to the class classification circuit 43, and the class classification circuit 43 receives the attention based on the class tap from the class tap extraction circuit 42. The pixel block is classified, and a class code corresponding to the resulting class is output.
[0162]
Here, as a method of classifying, for example, ADRC (Adaptive Dynamic Range Coding) or the like can be employed.
[0163]
In the method using ADRC, the DCT coefficients constituting the class tap are subjected to ADRC processing, and the class of the pixel block of interest is determined according to the ADRC code obtained as a result.
[0164]
In the K-bit ADRC, for example, the maximum value MAX and the minimum value MIN of the DCT coefficient constituting the class tap are detected, and DR = MAX−MIN is set as the local dynamic range of the set, and this dynamic range DR is included in this dynamic range DR. Based on this, the DCT coefficients constituting the class tap are requantized to K bits. That is, the minimum value MIN is subtracted from the DCT coefficients constituting the class tap, and the subtracted value is DR / 2.^KDivide by (quantize). A bit string obtained by arranging the K-bit DCT coefficients constituting the class tap in a predetermined order, which is obtained as described above, is output as an ADRC code. Therefore, when a class tap is subjected to, for example, 1-bit ADRC processing, each DCT coefficient constituting the class tap is an average value of the maximum value MAX and the minimum value MIN after the minimum value MIN is subtracted. By division, each DCT coefficient is made 1 bit (binarized). A bit string in which the 1-bit DCT coefficients are arranged in a predetermined order is output as an ADRC code.
[0165]
The class classification circuit 43 can also output, for example, the level distribution pattern of the DCT coefficient constituting the class tap as a class code as it is, but in this case, the class tap has N DCT coefficients. If the KT bit is assigned to each DCT coefficient, the number of class codes output by the class classification circuit 43 is (2^N)^KAs a result, the number is exponentially proportional to the number of bits K of the DCT coefficient.
[0166]
Therefore, the class classification circuit 43 preferably performs class classification after compressing the information amount of the class tap by the above-described ADRC processing or vector quantization.
[0167]
By the way, in this embodiment, the class tap is composed of 64 DCT coefficients as described above. Therefore, for example, even if class classification is performed by performing 1-bit ADRC processing on a class tap, the number of class codes is 2⁶⁴It becomes a big value of street.
[0168]
Therefore, in the present embodiment, the class classification circuit 43 extracts feature quantities having high importance from the DCT coefficients constituting the class tap, and classifies based on the feature quantities, thereby reducing the number of classes. It is supposed to be.
[0169]
That is, FIG. 19 shows a configuration example of the class classification circuit 43 of FIG.
[0170]
The class tap is supplied to the power calculation circuit 51. The power calculation circuit 51 divides the DCT coefficients constituting the class tap into those of several spatial frequency bands, and distributes the power of each frequency band. Calculate.
[0171]
That is, the power calculation circuit 51 divides the 8 × 8 DCT coefficients constituting the class tap into, for example, four spatial frequency bands S0, S1, S2, and S3 as shown in FIG.
[0172]
Further, the power calculation circuit 51 calculates the AC component power (sum of squares of AC components) P0, P1, P2, and P3 of the DCT coefficient for each of the spatial frequency bands S0, S1, S2, and S3, and generates a class code. Output to the circuit 52.
[0173]
The class code generation circuit 52 compares the powers P0, P1, P2, and P3 from the power calculation circuit 51 with the corresponding threshold values TH0, TH1, TH2, and TH3 stored in the threshold value table storage unit 53, respectively. The class code is output based on the magnitude relationship. That is, the class code generation circuit 52 compares the power P0 with the threshold value TH0, and obtains a 1-bit code representing the magnitude relationship. Similarly, the class code generation circuit 52 compares the power P1 and the threshold value TH1, the power P2 and the threshold value TH2, and the power P3 and the threshold value TH3, thereby obtaining a 1-bit code for each. Then, the class code generation circuit 52, for example, a 4-bit code obtained by arranging the four 1-bit codes obtained as described above in a predetermined order (accordingly, any one of 0 to 15). Is output as a class code representing the class of the pixel block of interest. Therefore, in this embodiment, the target pixel block is 2^FourIt is classified into one of (= 16) classes.
[0174]
The threshold value table storage unit 53 stores threshold values TH0 to TH3 to be compared with the powers P0 to P3 of the spatial frequency bands S0 to S3, respectively.
[0175]
In the above case, the DC component of the DCT coefficient is not used for the class classification process, but the class classification process can also be performed using this DC component.
[0176]
Returning to FIG. 18, the class code output from the class classification circuit 43 as described above is given to the coefficient table storage unit 44 as an address.
[0177]
The coefficient table storage unit 44 stores a coefficient table in which tap coefficients obtained by performing learning processing using teacher data and student data as described above are registered, and the class classification circuit 43 outputs the coefficient table. The tap coefficient stored at the address corresponding to the class code is output to the product-sum operation circuit 45.
[0178]
Here, in this embodiment, since the pixel block is classified, one class code is obtained for the pixel block of interest. On the other hand, since the pixel block is composed of 64 pixels of 8 × 8 pixels in the present embodiment, 64 sets of tap coefficients for decoding each of the 64 pixels constituting the pixel block of interest are required. is there. Accordingly, the coefficient table storage unit 44 stores 64 sets of tap coefficients for an address corresponding to one class code.
[0179]
The product-sum operation circuit 45 acquires the prediction tap output from the prediction tap extraction circuit 41 and the tap coefficient output from the coefficient table storage unit 44, and uses the prediction tap and the tap coefficient to obtain Equation (5). An output image creation unit 33 (the inverse DCT result) is obtained with the pixel value of 8 × 8 pixels of the target pixel block obtained as a result of performing the linear prediction calculation (product-sum calculation) shown in FIG. 1).
[0180]
Here, in the prediction tap extraction circuit 41, as described above, each pixel of the target pixel block is sequentially set as the target pixel. However, the product-sum operation circuit 45 becomes the target pixel of the target pixel block. Processing is performed in an operation mode corresponding to the position of the pixel (hereinafter, referred to as pixel position mode as appropriate).
[0181]
That is, for example, if the i-th pixel in the raster scan order is represented by pi and the pixel pi is the pixel of interest in the pixel block of interest, the product-sum operation circuit 45 uses the pixel position mode # Process i is performed.
[0182]
Specifically, as described above, the coefficient table storage unit 44 outputs 64 sets of tap coefficients for decoding each of the 64 pixels constituting the target pixel block, and for decoding the pixel pi among them. When the set of tap coefficients is represented as Wi, the product-sum operation circuit 45 uses the prediction tap and the set Wi among the 64 sets of tap coefficients when the operation mode is the pixel position mode #i, The product-sum operation of 5) is performed, and the product-sum operation result is set as the decoding result of the pixel pi.
[0183]
Next, processing of the inverse DCT transform unit 32 in FIG. 18 will be described with reference to the flowchart in FIG.
[0184]
The DCT coefficients for each block output by the correction value adding unit 31 (FIG. 1) are sequentially received by the prediction tap extraction circuit 41 and the class tap extraction circuit 42, and the prediction tap extraction circuit 41 receives the DCT coefficients supplied thereto. A pixel block corresponding to the block (DCT block) is sequentially set as a target pixel block.
[0185]
In step S 21, the class tap extraction circuit 42 extracts from the DCT coefficients received there, what is used to classify the pixel block of interest, forms a class tap, and supplies the class tap to the class classification circuit 43. To do.
[0186]
In step S22, the class classification circuit 43 classifies the target pixel block using the class tap from the class tap extraction circuit 42, and outputs the class code obtained as a result to the coefficient table storage unit 44.
[0187]
That is, in step S22, the power calculation circuit 51 of the class classification circuit 43 (FIG. 19) divides the 8 × 8 DCT coefficients constituting the class tap into the four spatial frequency bands S0 to S3 shown in FIG. Then, the respective powers P0 to P3 are calculated. The powers P0 to P3 are output from the power calculation circuit 51 to the class code generation circuit 52.
[0188]
The class code generation circuit 52 reads the thresholds TH0 to TH3 from the threshold table storage unit 53, compares each of the powers P0 to P3 from the power calculation circuit 51 with each of the thresholds TH0 to TH3, and based on the magnitude relationship of each. Generate class code.
[0189]
The class code obtained as described above is given as an address from the class classification circuit 43 to the coefficient table storage unit 44.
[0190]
When the coefficient table storage unit 44 receives the class code as the address from the class classification circuit 43, the coefficient table storage unit 44 reads out 64 sets of tap coefficients stored in the address and outputs them to the product-sum operation circuit 45 in step S23.
[0191]
Then, the process proceeds to step S24, and the prediction tap extraction circuit 41 predicts the pixel value of the target pixel by using, as the target pixel, a pixel that has not yet been set as the target pixel in the raster scan order among the pixels of the target pixel block. DCT coefficients used for the extraction are extracted and configured as prediction taps. This prediction tap is supplied from the prediction tap extraction circuit 41 to the product-sum operation circuit 45.
[0192]
Here, in the present embodiment, for each pixel block, the same prediction tap is configured for all the pixels of the pixel block, so in practice, the processing in step S24 is first performed for the target pixel block. If it is performed only for the pixel that is the target pixel, it is not necessary to perform the remaining 63 pixels.
[0193]
In step S25, the product-sum operation circuit 45 acquires a set of tap coefficients corresponding to the pixel position mode for the target pixel from among the 64 sets of tap coefficients output from the coefficient table storage unit 44 in step S23, and the tap coefficients. And the prediction tap supplied from the prediction tap extraction circuit 41 in step S24, the product-sum operation shown in Expression (5) is performed to obtain a decoded value of the pixel value of the target pixel.
[0194]
Then, the process proceeds to step S <b> 26, and the prediction tap extraction circuit 41 determines whether or not processing has been performed on all the pixels of the target pixel block as the target pixel. If it is determined in step S26 that all the pixels in the pixel block of interest have not been processed as pixels of interest, the process returns to step S24, and the prediction tap extraction circuit 41 selects the raster among the pixels of the pixel block of interest. In the scan order, a pixel that has not yet been set as the target pixel is newly set as the target pixel, and the same processing is repeated.
[0195]
If it is determined in step S26 that all the pixels of the target pixel block have been processed as the target pixel, that is, if the decoded values of all the pixels of the target pixel block are obtained, the product-sum operation circuit 45 Outputs the pixel block (decoded block) composed of the decoded values to the output image creation unit 33 (FIG. 1), and ends the process.
[0196]
In the coefficient table storage unit 44, an image is used as teacher data, and the output of the correction value addition unit 31 when the image is MPEG-encoded and input to the image processing apparatus in FIG. 1 is used as student data. It is necessary to store the tap coefficient obtained by performing the learning process.
[0197]
In the above case, the class classification is performed using only the DCT coefficient of the DCT block. The class classification includes, for example, the activity of the DCT block, the quantization scale, I, It is possible to carry out by using different P and B pictures. In this case, also in the learning process, it is necessary to classify using DCT coefficients, activities, quantization scales, I, P, and B pictures, etc., so that the tap coefficients are DCT coefficients, activities, quantization scales. , I, P, and B pictures. Therefore, when such a tap coefficient is used to convert a DCT coefficient into a pixel value, it is possible to improve the image quality of an image composed of the pixel value.
[0198]
Furthermore, the class classification adaptive processing can be used not only when converting DCT coefficients into pixel values, but also when converting pixel values into DCT coefficients. That is, for example, the DCT conversion unit 24 of FIG. 1 can also convert the output of the correction value calculation unit 23 into a DCT coefficient using the class classification adaptation process. However, in this case, the output of the correction value calculation unit 23 is used as student data, and learning processing is performed by using DCT coefficients obtained by DCT processing of the output as teacher data to obtain tap coefficients. .
[0199]
Next, FIG. 22 shows a configuration example of another embodiment of an image processing apparatus to which the present invention is applied. In the figure, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, in the image processing apparatus of FIG. 22, firstly, a computing unit 4 is newly provided, secondly, a DCT conversion unit 14 is newly provided in the input image analysis unit 1, and thirdly, a distortion correction value is provided. The block level detecting unit 21 or the activity calculating unit 22 of the calculating unit 2 is replaced with a block level detecting unit 26 or an activity calculating unit 27, respectively.
[0200]
In the image processing apparatus of FIG. 1, the difference information of the block boundary portion and the activity of the block are both obtained from the pixel values output from the MPEG decoding unit 11, but in the image processing apparatus of FIG. The difference information and the activity are obtained from the DCT coefficient.
[0201]
That is, the DCT conversion unit 14 performs DCT processing on the block of pixel values obtained by the MPEG decoding unit 11 and supplies the block to the computing unit 4 as a block of DCT coefficients.
[0202]
Note that the DCT conversion unit 14 can also convert pixel values into DCT coefficients using the above-described class classification adaptive processing.
[0203]
The computing unit 4 is supplied with a block of DCT coefficients output from the DCT coefficient extraction / inverse quantization unit 12 in addition to a block of DCT coefficients output from the DCT conversion unit 14. The blocks of these two DCT coefficients are added as necessary, and supplied to the block boundary step detection unit 26, the activity calculation unit 27, and the correction value addition unit 31.
[0204]
That is, for the I picture, the DCT coefficient block output from the DCT coefficient extraction / inverse quantization unit 12 is obtained by DCT processing the block of pixel values of the original image, and the DCT coefficient is subjected to inverse DCT processing. Thus, the pixel value of the original image can be obtained, so that the computing unit 4 directly uses the block of the DCT coefficient output from the DCT coefficient extraction / dequantization unit 12 as it is, the block boundary step detection unit 26, and the activity calculation unit. 27 and the correction value adding unit 31.
[0205]
On the other hand, for the P and B pictures, the DCT coefficient block output from the DCT coefficient extraction / inverse quantization unit 12 is obtained by DCT processing the difference (prediction residual) between the pixel value block of the original image and the predicted image. Therefore, if the DCT coefficient is subjected to inverse DCT processing, the pixel value of the original image cannot be obtained. That is, for the P and B pictures, the DCT coefficient obtained by adding the DCT coefficient block output from the DCT coefficient extraction / inverse quantization unit 12 and the DCT coefficient obtained by DCT processing of the predicted image is converted into the inverse DCT. The pixel value of the original image can be obtained by processing. Therefore, in this case, the DCT conversion unit 14 performs DCT processing on the prediction image obtained by the MPEG decoding unit 11 and supplies the DCT coefficient obtained as a result to the arithmetic unit 4. Then, the computing unit 4 adds the DCT coefficient block output from the DCT coefficient extraction / inverse quantization unit 12 and the DCT coefficient output from the DCT conversion unit 14, and obtains the resulting DCT coefficient block as a block. This is supplied to the boundary step detection unit 26, the activity calculation unit 27, and the correction value addition unit 31.
[0206]
Therefore, the block of DCT coefficients output from the computing unit 4 is a block of pixel values of the original image by performing inverse DCT processing regardless of the picture type (any of I, P and B pictures). Is obtained.
[0207]
Here, as described above, in FIG. 22, the MPEG decoding unit 11 does not have to perform the entire MPEG decoding process. That is, the MPEG decoding unit 11 in FIG. 22 only needs to be able to generate a predicted image by performing motion compensation using the reference image stored in the image memory 34.
[0208]
The block boundary level detector 26 obtains difference information for the block of the DCT coefficient output from the computing unit 4 in the same manner as in the block boundary level detector 21 of FIG. Therefore, the difference information obtained by the block boundary step detection unit 26 is not a difference in pixel values but a difference in DCT coefficients.
[0209]
The activity calculation unit 27 calculates the activity of the block from the block of the DCT coefficient output from the computing unit 4. That is, the activity of a block has a high correlation with the square sum of the AC components of the DCT coefficients of the block. Therefore, the activity calculation unit 27 obtains the square sum of the AC components of the block of the DCT coefficient output from the computing unit 4 and outputs the sum of the squares to the correction value calculation unit 23 and the high frequency reduction unit 25 as the activity of the block.
[0210]
Hereinafter, in the image processing apparatus of FIG. 22, an image with reduced block distortion and the like is decoded in the same manner as in FIG.
[0211]
However, as described above, the block of DCT coefficients output from the arithmetic unit 4 to the correction value adding unit 31 is obtained by performing inverse DCT processing regardless of the picture type, thereby obtaining a block of pixel values of the original image. Therefore, it is not necessary for the output image creation unit 33 to add a predicted image for not only an I picture but also a P picture and a B picture.
[0212]
Next, FIG. 23 shows a configuration example of an embodiment of a transmission system to which the present invention is applied.
[0213]
This transmission system includes a transmission device 61 and a reception device 62, and MPEG encoded image data is transmitted from the transmission device 61 via, for example, a network 63 such as a public network, the Internet, a CATV network, or a satellite line. The data is transmitted to the receiving device 62.
[0214]
Image data is input to the transmission device 61, and this image data is supplied to the MPEG encoding unit 71. The MPEG encoding unit 71 performs MPEG encoding on the image data supplied thereto, and supplies the encoded data obtained as a result to an input image analysis unit 72 and a MUX (multiplexer) 74.
[0215]
The input image analysis unit 72 performs, for example, the same processing as the input image analysis unit 1 of FIG. 1 on the encoded data from the MPEG encoding unit 71, and supplies the processing result to the distortion correction value calculation unit 73. To do. The distortion correction value calculation unit 73 uses the output of the input image analysis unit 72 to perform, for example, the same processing as that of the distortion correction value calculation unit 2 in FIG. 1, and is output to the image reconstruction unit 3 in FIG. The same correction value is obtained and supplied to the MUX 74.
[0216]
The MUX 74 multiplexes the encoded data from the MPEG encoding unit 71 and the correction value from the distortion correction value calculation unit 73, and supplies the multiplexed data obtained as a result to a communication I / F (Interface) 75. The communication I / F 75 transmits the multiplexed data from the MUX 74 to the receiving device 62 via the network 63.
[0217]
In the receiving device 62, the communication I / F 81 receives the multiplexed data transmitted from the transmitting device 61 via the network 63 as described above, and supplies it to the DMUX (demultiplexer) 82. The DMUX 82 separates the multiplexed data from the communication I / F 81 into encoded data and correction values, and supplies the encoded data to the input image analysis unit 83 and the correction values to the image reconstruction unit 84, respectively.
[0218]
The input image analysis unit 83 performs, for example, the same processing as the input image analysis unit 1 of FIG. 1 on the encoded data from the DMUX 82 and supplies the processing result to the image reconstruction unit 84.
[0219]
In accordance with the operation of the operation unit 85, the image reconstruction unit 84 corrects the output of the input image analysis unit 83 with the correction value from the DMUX 82 or processes it without correction, and obtains and outputs a decoded image.
[0220]
That is, when the user does not operate the operation unit 85 so as to request a high-quality image, the image reconstruction unit 84 corrects the output of the input image analysis unit 83 with the correction value from the DMUX 82. Process, and obtain and output a decoded image. Therefore, in this case, the user is provided with a decoded image in which block distortion or the like is conspicuous depending on the activity of the image or the compression rate of the image in the MPEG encoding unit 71 of the transmission device 61.
[0221]
On the other hand, when the user operates the operation unit 85 to request a high-quality image, the image reconstruction unit 84 corrects the output of the input image analysis unit 83 with the correction value from the DMUX 82. Process to obtain and output a decoded image. Therefore, in this case, the user is provided with a high-quality decoded image with reduced block distortion and the like.
[0222]
Further, in this case, the image reconstruction unit 84 supplies the communication I / F 81 with a message indicating that a high-quality image corrected by the correction value has been provided (hereinafter, referred to as a high-quality image provision message as appropriate). . The communication I / F 81 transmits the high-quality image providing message to the transmission device 61 via the network 63 together with the reception device ID (Identification) attached to the reception device 62 in advance.
[0223]
In the transmission device 61, the communication I / F 75 receives the high-quality image provision message and the reception device ID from the reception device 62 and supplies them to the charging processing unit 76. The charging processing unit 76 manages the receiving device ID and the charging information for the user corresponding to the receiving device ID in association with each other, and when receiving the high-quality image providing message and the receiving device ID, The accounting process for the corresponding user is performed, and the accounting information is updated. Based on this billing information, the user is billed as a price for providing a high-quality image.
[0224]
In the above-described case, a high-quality image corrected by the correction value is provided in response to the operation of the operation unit 85, and then the billing process is performed. It is also possible to collect a fee in advance, permit only the user who has paid the fee to use the correction value, and provide a high-quality image corrected by the correction value. Furthermore, it is possible to provide a high-quality image corrected by the correction value without collecting a fee. Further, the correction value can be generated not by the transmission device 61 but by the reception device 62.
[0225]
Next, the series of processes described above can be performed by hardware or software. When a series of processing is performed by software, a program constituting the software is installed in a general-purpose computer or the like.
[0226]
FIG. 24 shows an example of the configuration of an embodiment of a computer in which a program for executing the series of processes described above is installed.
[0227]
The program can be recorded in advance in a hard disk 105 or a ROM 103 as a recording medium built in the computer.
[0228]
Alternatively, the program is stored temporarily on a removable recording medium 111 such as a floppy disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disc, a DVD (Digital Versatile Disc), a magnetic disc, or a semiconductor memory. It can be stored permanently (recorded). Such a removable recording medium 111 can be provided as so-called package software.
[0229]
The program is installed in the computer from the removable recording medium 111 as described above, or transferred from the download site to the computer wirelessly via a digital satellite broadcasting artificial satellite, LAN (Local Area Network), The program can be transferred to a computer via a network such as the Internet, and the computer can receive the program transferred in this way by the communication unit 108 and install it in the built-in hard disk 105.
[0230]
The computer includes a CPU (Central Processing Unit) 102. An input / output interface 110 is connected to the CPU 102 via the bus 101, and the CPU 102 operates an input unit 107 including a keyboard, a mouse, a microphone, and the like by the user via the input / output interface 110. When a command is input as a result, the program stored in a ROM (Read Only Memory) 103 is executed accordingly. Alternatively, the CPU 102 also transfers from a program stored in the hard disk 105, a program transferred from a satellite or a network, received by the communication unit 108 and installed in the hard disk 105, or a removable recording medium 111 attached to the drive 109. The program read and installed in the hard disk 105 is loaded into a RAM (Random Access Memory) 104 and executed. Thus, the CPU 102 performs processing according to the above-described flowchart or processing performed by the configuration of the above-described block diagram. Then, the CPU 102 outputs the processing result from the output unit 106 configured with an LCD (Liquid Crystal Display), a speaker, or the like via the input / output interface 110, or from the communication unit 108 as necessary. Transmission and further recording on the hard disk 105 are performed.
[0231]
Here, in the present specification, the processing steps for describing a program for causing the computer to perform various processes do not necessarily have to be processed in time series in the order described in the flowcharts, but in parallel or individually. This includes processing to be executed (for example, parallel processing or processing by an object).
[0232]
Further, the program may be processed by one computer or may be distributedly processed by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.
[0233]
In this embodiment, the case of decoding an MPEG-encoded image is targeted. However, the present invention can be applied to the case of decoding an image encoded by performing orthogonal transform in units of blocks. .
[0234]
【The invention's effect】
  As above,According to the present invention,Block distortion and the like can be easily reduced, and a high-quality decoded image can be obtained.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration example of a first embodiment of an image processing apparatus to which the present invention has been applied.
FIG. 2 is a flowchart for explaining processing of the image processing apparatus in FIG. 1;
FIG. 3 is a diagram for explaining processing of a block boundary level detector 21;
FIG. 4 is a diagram illustrating macro blocks in a frame DCT mode and a field DCT mode in MPEG.
FIG. 5 is a diagram for explaining processing of a block boundary level detector 21;
FIG. 6 is a diagram for explaining processing of a block boundary level detector 21;
FIG. 7 is a diagram for explaining processing of a block boundary level detector 21;
FIG. 8 is a diagram for explaining processing of a block boundary level detector 21;
FIG. 9 is a diagram for explaining processing of a correction value calculation unit.
FIG. 10 is a diagram for explaining processing of a correction value calculation unit 23;
11 is a diagram for explaining processing of a correction value calculation unit 23. FIG.
FIG. 12 is a diagram for explaining processing of a high frequency reduction unit 25;
FIG. 13 is a diagram for explaining the effect of processing by the high frequency reduction unit 25;
FIG. 14 is a diagram for explaining processing of a high frequency reduction unit 25;
15 is a diagram for explaining processing of a correction value adding unit 31. FIG.
FIG. 16 is a half-tone photograph displayed on a display showing a simulation result by the present inventor.
FIG. 17 is a half-tone photograph displayed on a display showing a simulation result by the present inventor.
18 is a block diagram illustrating a configuration example of an inverse DCT conversion unit 32. FIG.
19 is a block diagram illustrating a configuration example of a class classification circuit 43. FIG.
FIG. 20 is a diagram for explaining processing of the power calculation circuit 51;
FIG. 21 is a flowchart for describing processing of the inverse DCT transform unit 32 in FIG. 18;
FIG. 22 is a block diagram illustrating a configuration example of a second embodiment of an image processing apparatus to which the present invention has been applied.
FIG. 23 is a block diagram illustrating a configuration example of an embodiment of a transmission system to which the present invention has been applied.
FIG. 24 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present invention has been applied.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Input image analysis part, 2 Distortion correction value calculation part, 3 Image reconstruction part, 4 Calculator, 11 MPEG decoding part, 12 DCT coefficient extraction / inverse quantization part, 13 Side information extraction part, 14 DCT conversion part, 21 Block boundary step detection unit, 22 Activity calculation unit, 23 Correction value calculation unit, 24 DCT conversion unit, 25 High frequency reduction unit, 26 Block boundary step detection unit, 27 Activity calculation unit, 31 Correction value addition unit, 32 Inverse DCT conversion Unit, 33 output image creation unit, 34 image memory, 35 picture selection unit, 41 prediction tap extraction circuit, 42 class tap extraction circuit, 43 class classification circuit, 44 coefficient table storage unit, 45 product-sum operation circuit, 51 power operation circuit , 52 Class code generation circuit, 53 Threshold table Memory unit, 61 transmitting device, 62 receiving device, 63 network, 71 MPEG encoding unit, 72 input image analyzing unit, 73 distortion correction value calculating unit, 74MUX, 75 communication I / F, 76 charge processing unit, 81 communication I / F , 82 DMUX, 83 input image analysis unit, 84 image reconstruction unit, 85 operation unit, 101 bus, 102 CPU, 103 ROM, 104 RAM, 105 hard disk, 106 output unit, 107 input unit, 108 communication unit, 109 drive, 110 I / O interface, 111 Removable recording medium

Claims (10)

A coefficient extraction unit that blocks an image constituting the input stream and extracts an orthogonal transform coefficient for each block of a predetermined size;
  Decoding means for decoding the stream;
  Extraction means for extracting the type of orthogonal transformation of each block included in the stream;
  The extraction unit extracts difference information corresponding to the difference between the pixel value at the boundary of the block constituting the image decoded by the decoding unit and the pixel value at the boundary of the adjacent block that is a block adjacent to the block. Difference information calculation means for generating the difference between adjacent pixels with reference to the orthogonal transformation type,
  Activity detecting means for detecting an activity of the block constituting the image decoded by the decoding means;
  Based on the difference information and the activity detected by the activity detection means, if the activity of the block and the activity of the adjacent block are equal, a boundary that is a pixel adjacent to the adjacent block among the pixels of the block A correction value is calculated such that the pixel value of the pixel and the pixel value of the pixel of the adjacent block adjacent to the boundary pixel approach an intermediate pixel value, and there is a predetermined difference between the activity of the block and the activity of the adjacent block. In some cases, the pixel value of the block with the higher activity among the pixel value of the boundary pixel of the block and the pixel value of the pixel of the adjacent block adjacent to the boundary pixel is set to the pixel value of the block with the lower activity. Correction value calculating means for calculating a correction value that approaches the pixel value;
  Orthogonal transformation means for performing orthogonal transformation processing on the correction value calculated by the correction value calculation means and expressed as a difference between pixel values;
  Correction means for correcting the orthogonal transformation coefficient obtained by performing orthogonal transformation processing by the orthogonal transformation means,
  Correction means for obtaining a corrected orthogonal transform coefficient by adding or subtracting the orthogonal transform coefficient corrected by the correction means to the orthogonal transform coefficient extracted by the coefficient extraction means;
  Inverse orthogonal transform means for transforming the corrected orthogonal transform coefficient into a pixel value by inverse orthogonal transform;
  Image creating means for creating an image using an inverse orthogonal transform result and a predicted image obtained by the decoding means by performing motion compensation on a predetermined reference image;
  With
  The image created by the image creating means is used by the decoding means as a reference image for decoding the stream.
  Image processing device. It said correction means, by reducing the high frequency components, modifies the orthogonal transformation coefficients
The image processing apparatus according to claim 1 . The correction means corrects the orthogonal transform coefficient by setting the higher order to zero.
The image processing apparatus according to claim 1 . The correction means determines an orthogonal transform coefficient to be 0 based on the activity.
The image processing apparatus according to claim 1 . In the case where the orthogonal transform coefficient of the block is quantized,
Said correcting means, an orthogonal transform coefficient of the block based on the quantization scale used when quantizing, among the orthogonal transformation coefficient, to determine what you 0
The image processing apparatus according to claim 1 . It said correction means, by performing a predetermined weighting modifies the orthogonal transformation coefficients
The image processing apparatus according to claim 1 . It said correction means, with respect to the orthogonal transform coefficients, performs weighting based on the order of the orthogonal transform coefficients
The image processing apparatus according to claim 1 . In the case where the orthogonal transform coefficient of the block is quantized,
Said correction means performs relative orthogonal transform coefficients, the weighting based on the quantization table used when quantizing the orthogonal transformation coefficients of the block
The image processing apparatus according to claim 1 .   Block the images that make up the input stream, extract the orthogonal transform coefficients for each block of a predetermined size,
  Decoding the stream by a decoding means;
  Extracting the type of orthogonal transform of each block contained in the stream;
  The difference information corresponding to the difference between the pixel value at the boundary of the block constituting the decoded image and the pixel value at the boundary of the adjacent block which is a block adjacent to the block is referred to the type of the orthogonal transform extracted. , Generate the difference between adjacent pixels,
  Detecting the activity of the blocks comprising the decoded image,
  Based on the difference information and the detected activity, when the activity of the block and the activity of the adjacent block are equal, the pixel value of the boundary pixel that is the pixel adjacent to the adjacent block among the pixels of the block; A correction value is calculated so as to bring the pixel value of the pixel of the adjacent block adjacent to the boundary pixel closer to an intermediate pixel value, and when there is a predetermined difference between the activity of the block and the activity of the adjacent block, Of the pixel value of the boundary pixel of the block and the pixel value of the pixel of the adjacent block adjacent to the boundary pixel, the pixel value of the block with the larger activity is brought closer to the pixel value with the smaller activity. Calculate the correction value,
  An orthogonal transformation process is performed on the calculated correction value represented as a difference between pixel values,
  Correct the orthogonal transformation coefficient obtained by performing the orthogonal transformation process,
  The corrected orthogonal transform coefficient is added to or subtracted from the extracted orthogonal transform coefficient to obtain a corrected orthogonal transform coefficient,
  The corrected orthogonal transform coefficient is subjected to inverse orthogonal transform and converted into a pixel value,
  An image is created using the inverse orthogonal transform result and the predicted image obtained by the decoding means by motion compensation of a predetermined reference image
  Including steps,
  The created image is used by the decoding means as a reference image for decoding the stream.
  Image processing method.   Block the images that make up the input stream, extract the orthogonal transform coefficients for each block of a predetermined size,
  Decoding the stream by decoding means;
  Extracting the type of orthogonal transform of each block contained in the stream;
  The difference information corresponding to the difference between the pixel value at the boundary of the block constituting the decoded image and the pixel value at the boundary of the adjacent block which is a block adjacent to the block is referred to the type of the orthogonal transform extracted. , Generate the difference between adjacent pixels,
  Detecting the activity of the blocks comprising the decoded image,
  Based on the difference information and the detected activity, when the activity of the block and the activity of the adjacent block are equal, the pixel value of the boundary pixel that is the pixel adjacent to the adjacent block among the pixels of the block; A correction value is calculated so as to bring the pixel value of the pixel of the adjacent block adjacent to the boundary pixel closer to an intermediate pixel value, and when there is a predetermined difference between the activity of the block and the activity of the adjacent block, Of the pixel value of the boundary pixel of the block and the pixel value of the pixel of the adjacent block adjacent to the boundary pixel, the pixel value of the block with the larger activity is brought closer to the pixel value with the smaller activity. Calculate the correction value,
  An orthogonal transformation process is performed on the calculated correction value represented as a difference between pixel values,
  Correct the orthogonal transformation coefficient obtained by performing the orthogonal transformation process,
  The corrected orthogonal transform coefficient is added to or subtracted from the extracted orthogonal transform coefficient to obtain a corrected orthogonal transform coefficient,
  The corrected orthogonal transform coefficient is subjected to inverse orthogonal transform and converted into a pixel value,
  An image is created using the inverse orthogonal transform result and the predicted image obtained by the decoding means by motion compensation of a predetermined reference image
  Including steps,
  The created image is used by the decoding means as a reference image for decoding the stream.
  A recording medium on which a program for causing a computer to execute processing is recorded.

Images