JP4334932B2 - Image processing apparatus and image processing method - Google Patents

Description

[式１]におけるＺは、歪中心（Ｘd，Yｄ）から今注目している点（Ｘ，Ｙ）までの距離である。[式１]によって、補正画像の点（Ｘ，Ｙ）に対して歪んでいる元画像の方の座標（Ｘ’，Ｙ’）が算出される。[式１]について補足説明すると、Ｍは光学系のデータを用いて理論的に補正した際、補正後に画像がはみ出したり、不足したりする現象を補正するための補正倍率である。Ｓx，Ｓyは間引き取込みなど、縦横で空間的なサンプリング間隔が異なる現象の補正をするためのサンプリング比である。Ｘoff，Ｙoffは歪補正処理によって、歪補正処理後に、被写体位置が撮影時の位置とはずれてしまう現象の補正をする中心ずれの値である。
【００５５】
本発明に係る[式１]では、高次項（具体的にはＺ４、Ｚ６、…）を考慮することで、より複雑な陣笠型の歪（図４参照）にも対応できるようにしている。また、中心が一致していても撮像素子の一部しかデータを取り込まない場合や、光軸中心と撮像素子中心がずれている場合など、歪中心（Ｘｄ，Ｙｄ）を考慮する必要がある。さらに、歪補正処理により被写体位置が撮影時の位置からずれる現象を補正するため、中心ずれ補正値（Ｘoff，Ｙoff）を考慮する必要がある。間引き取込みの場合や輝度データ（Ｙ）と色データ（Ｃb，Ｃr）からなるＹＣ画像を処理する場合は、Ｘ，Ｙで空間的なサンプリングが異なる。そこで係数としてサンプリング比（ＳＸ，Ｓｙ）を導入して、座標変換を補正するようにしている。
【００５６】
次に、図２５，図２６を参照して画像データの書き込み及び読み出しの処理及び順序について説明する。
図２５は、本実施の形態におけるフレームメモリからの画像データの読み出し順序を説明する図である。
【００５７】
画像データは、通常は、ライン方向、つまり行方向にスイープさせて書き込まれていて、読み出し時は行方向に読み出されるようになっており、１ラインの画像データを全て読み出して、次に隣接するラインの画像データを全て読み出すといった動作を繰り返して行うのが普通である。
【００５８】
これに対して、本発明に係る画像処理装置は、行方向にスイープさせて書き込まれた画像データを、列方向にある一定の長さを単位に、行方向の画像データを順番にイメージプロセス部７に入力し、以降、順次隣の列をイメージプロセス部７に入力して、画像の右端まで繰り返して得られる小領域（矩形状の画像データ）をブロックライン（ＢＬ）と呼ぶ。
【００５９】
こうした画像データの読み出しを可能にするための第１データ順序変換部６と、第２データ順序変換部９との構成を、図２６を参照して説明する。図２６は、第１，第２データ順序変換部の構成を示すブロック図である。
【００６０】
第１データ順序変換部６は、図２６(a)に示すように、ブロック単位の画像データを記憶可能なメモリを複数、ここでは２つ備えており、この２つのメモリ６ａとメモリ６ｂが書き込み側，読み出し側の各スイッチにて書き込み，読み出しが交互に切り換えられるようになっている。すなわち、フレームメモリ５は、書き込み側のスイッチにて、これらメモリ６ａとメモリ６ｂとに切換可能に接続されているとともに、イメージプロセス部７も読み出し側のスイッチにてこれらのモリ６ａとメモリ６ｂとに切換可能に接続されていて、フレームメモリ５がメモリ６ａとメモリ６ｂとの一方に接続されているときには、該メモリ６ａとメモリ６ｂとの他方がイメージプロセス部７に接続されるように切り換えられる。すなわち、メモリ６ａ，６ｂは、フレームメモリ５とイメージプロセス部７との両方に同時に接続されることがないようにスイッチングされ、交互に書き込み，読み出しが行われるようになっている。
【００６１】
フレームメモリ５に記憶されているフレーム画像の一部は、ブロック単位でライン方向に読み出され、一方のメモリ、ここでは例えばメモリ６ａに記憶される。
【００６２】
これと並行して、メモリ６ｂからは、既にフレームメモリ５から読み出されて記憶されているブロック単位の画像データが、列方向（縦方向）に順に読み出されて、イメージプロセス部７へ出力される。
【００６３】
フレームメモリ５からメモリ６ａへの書き込みと、メモリ６ｂからイメージプロセス部７への読み出しが終了すると、書き込み側のスイッチおよび読み出し側のスイッチが切り換えられて、次に、フレームメモリ５からメモリ６ｂへ次のブロック単位の画像データの書き込みが開始されるとともに、メモリ６ａからイメージプロセス部７へのブロック単位の画像データの読み出しが開始される。
【００６４】
第２データ順序変換部９も、図２６(b)に示すように、上記第１データ順序変換部６とほぼ同様に構成されていて、ほぼ同様に動作するようになっている。
【００６５】
すなわち、第２データ順序変換部９は、メモリ９ａ，メモリ９ｂと、書き込み側スイッチ，読み出し側スイッチとを有して構成されている。
【００６６】
そして、該第２データ順序変換部９の動作時には、歪補正処理部８からの書き込みがメモリ９ａとメモリ９ｂとの一方に対して列方向（縦方向）に行われ、メモリ９ａとメモリ９ｂとの他方からは行方向（横方向）に読み出しが行われて、ＪＰＥＧ処理部１０へ出力されるようになっている。
図６は、図２における歪補正処理部８の詳細な構成を示している。
図６の各部と図２の各部との対応関係を説明する。図６における補間位置算出回路２２が図２の補間位置生成部811に、セレクタ２４が図２のセレクタ813に、歪補正係数補正回路２１及び補間位置補正回路２３が図２の歪補正座標変換部812に、それぞれ対応している。また、図６における２ポートＳＲＡＭ２６が図２の内部メモリ部８２に対応し、書込みアドレス生成回路２８，バッファ空き容量監視回路２９，データ送信可否判定回路３０，バッファ開放量算出回路３１及び読出しアドレス生成回路２５が図２のメモリ制御部８３に、補間回路２７が図２の補間演算部８４に、それぞれ対応している。エラー検出回路３２は、図２には図示していないがメモリ制御部８３に接続して設けられている。エラー検出回路３２は、歪補正処理において歪量が大きくなり後述するｐｒｅＵＬＢ、ｐｏｓｔＵＬＢの設定値を越える歪が発生したときに、エラー（ERROR）としてＣＰＵ４に伝える機能を有するものである。
【００６７】
補間位置算出回路２２は、歪補正処理部８の後段回路からのリクエストに応じて後段回路へグラント（リクエスト受付）を返したときにそのグラントをトリガとして、１ユニットライン（以下、１ＵＬという）分の補間位置（Ｘ1，Ｙ1）を算出するものである。ここで、１ＵＬとは、前述したブロックライン処理において、メモリ部への書き込み，読み出しの際に、列方向に一列に並んだ一定数分の画像データを書き込んだり，読み出したりする際の一単位である。つまり、１ＵＬは、ブロックライン（ＢＬ）上の列方向に一列に並んだ一定画素数のデータを指している。
【００６８】
補間位置補正回路２３は、歪補正係数算出回路２１からの歪補正係数Ｆを補間位置（Ｘ1，Ｙ1）に乗算して、歪補正前の元データの座標位置（X’，Ｙ’）を計算する。セレクタ２４は、（Ｘ1，Ｙ1）と（X’，Ｙ’）を選択するもので、歪補正を行う場合は（X’，Ｙ’）を選択して出力し、拡大・縮小処理（リサイズ）のみを行う場合は（Ｘ1，Ｙ1）を選択して出力する。
【００６９】
２ポートＳＲＡＭ２６は、歪補正処理部８内におけるデータを格納するバッファである。
【００７０】
読出しアドレス生成回路２５は、補間位置に対応する２ポートＳＲＡＭ１６内のアドレス（ADR）を生成したり、２ポートＳＲＡＭ２６からの出力を整列するための制御信号を出力したり、出力画像データに同期して書き込み制御信号WE_Nを出力したり、図７及び[式２]に示すＤ0が２ポートＳＲＡＭ上のどこの位置かを知らせるデータ列制御信号を出力したりする。
【００７１】
書き込みアドレス生成回路２８は、書き込み制御信号WEに合わせて内部メモリである２ポートＳＲＡＭ２６のアドレス（ADDRESS）を生成したり、１ＵＬ分のデータ入力が完了したら、その内部カウンタ（BLC）をカウントアップする。
【００７２】
データ送信可否判定回路３０は、BLC値，本回路の動作状態，次のＵＬ先頭座標，歪補正処理部８の後段回路からのリクエスト（ＲＥＱ）状態とから、後段回路からのＲＥＱ信号に対してグラント（GRANT_Ｎ）を送信できる状態にあるかを判定し、グラント送信可能ならGRANT_Ｎ＝１とする機能を有する。
【００７３】
補間回路２７は、補間位置に対応する画像データにつき、１６点補間を行う。バッファ開放量算出回路３１は、現在処理中のＵＬ先頭座標と、次に処理予定のＵＬ先頭座標の整数部の差をバッファ開放量（図１７参照）として算出する。
【００７４】
バッファ空き容量監視回路２９は、内部バッファとしての２ポートＳＲＡＭ２６に格納されているデータ（ＵＬ）量を保持し、バッファに空きがあれば、歪補正処理部８の前段回路に対してリクエスト（REQ＝１）を送信する。
【００７５】
次に、図６の歪補正処理部８の回路動作を説明する。
まず、内部メモリ部（バッファ）である２ポートＳＲＡＭ２６に空きがある場合、バッファ空き容量監視回路２９から前段回路に対してデータ要求としてのリクエスト（ＲＥＱ）を送信すると、バッファ空き容量監視回路２９は前段回路からリクエスト受付としてのグラント（ＧＲＡＮＴ）を受信し、この受信と同時に２ポートＳＲＡＭ２６に格納可能なＵＬ数を記憶するカウンタ（回路２９内にある）を１つ減らす。１回のリクエストとグラントで１ＵＬを一動作単位としてデータ転送する。上記カウンタが０になったらリクエストを取り下げる。
【００７６】
そして、前段回路から書込みアドレス生成回路２８ヘデータが流入し、２ポートＳＲＡＭ２６への書込みが行われる。１ＵＬ入力毎に書込みアドレス生成回路１８の内部カウンタ（BLC）がアップする。
【００７７】
２ポートＳＲＡＭ２６は、補間回路２７にて例えば１６点補間を行えるように、図８の如く読出しと書込みが同時に行える２ポートＳＲＡＭが４×４の計１６個で構成されている。
【００７８】
ここで、図８を用いて、２ポートＳＲＡＭ２６を説明するが、メモリの数や各メモリの大きさは異なっていても構わない。例えば、図８では、ブロックラインの幅（ＵＬの長さ）を９６としているが、もっと長くしても短くしても構わない。回路規模と補正性能とのバランスで決めればよい。また、４×４（計１６個）も、４点補間であれば２×２（計４個）で構わない。補間方式（何点で補間するか）に合わせて決めればよい。
【００７９】
図８は、１６個の２ポートＳＲＡＭからなるメモリ空間で、横方向は入力されるＵＬ順であるが、縦方向は各２ポートＳＲＡＭごとに付したアドレスである。Ｎを整数とし、処理のある時点での状態を説明する。縦方向に並んだ４つの各２ポートＳＲＡＭであるＮｏ.０，４，８，１２に格納するデータは、４Ｎ，４Ｎ＋４，４Ｎ＋８，４Ｎ＋１２番目のＵＬのデータであり、Ｎｏ.１，５，９，１３に格納するデータは、４Ｎ＋１，４Ｎ＋５，４Ｎ＋９，４Ｎ＋１３番目のＵＬのデータであり、Ｎｏ.２，６，１０，１４に格納するデータは、４Ｎ＋２，４Ｎ＋６，４Ｎ＋１０，４Ｎ＋１４番目のＵＬのデータであり、Ｎｏ.３，７，１１，１５に格納するデータは、４Ｎ＋３，４Ｎ＋７，４Ｎ＋１１，４Ｎ＋１５番目のＵＬのデータである。
【００８０】
横方向に並んだ４つの各２ポートＳＲＡＭであるＮｏ.０，１，２，３及びＮｏ.４，５，６，７及びＮｏ.８，９，１０，１１及びＮｏ.１２，１３，１４，１５の各組についても同様に格納されている。
【００８１】
図９を使って、各２ポートＳＲＡＭにデータが書き込まれる様子を説明する。図９では、図９(a)に示すように、幅８ライン（画素）のブロックラインを想定しており、データは、ＵＬ単位に「１，２，…，ｉ，ｍ」、「５，６，…，ｊ，ｎ」、「９，１０，…，ｋ，ｏ」、…という順序で入力される。図９(b)は同図(a) に示した書込み順番のデータが4×4の１６個の２ポートＳＲＡＭNo.0〜No.15上のどこに書き込まれていくかを示している。例えばNo.0，No.4，No.8，No.12には、図９(a) の左側縦一列の１ＵＬ分の画像データ（順番1〜mで示されるデータ）が分散されて格納されていく。なお、図８に示される縦方向の２ポートＳＲＡＭNo.0，No.4，No.8，No.12の各縦ラインごとに付された表記0〜92，1〜93，2〜94，3〜95は、図９(a) で示される各ユニットライン（1〜m…、5〜n…、9〜ｏ…、・・・・）の表記とは異なっている。これは、図９(a)，(b)では、４×４の１６個の画像データにつき書き込み順が分かるように1〜16，a〜p，A〜P，…と１６個の塊ごとに符号を付してあるためである。図８の場合はその縦方向の２ポートＳＲＡＭNo.0，No.4，No.8，No.12の各縦ラインごとに付された表記は、縦方向の１つのＵＬにおけるデータ順に相当する。２ポートＳＲＡＭＮｏ．０，Ｎｏ．４，Ｎｏ．８，Ｎｏ．１２のそれぞれには１つのＵＬ画素データNo.0〜95が交互に分散して記載されている。従って、図８の縦方向に並んだ４つの各縦ラインには１つのユニットライン（ＵＬ）の画素順（0〜95）の何番目の画素であるかを示す表記が付されている。図８の２ポートＳＲＡＭのＮｏ.０の縦方向にはＵＬの４の倍数の画素（0，4，8…92）が格納してあり、２ポートＳＲＡＭのＮｏ.４の縦方向にはＵＬの（４の倍数）＋１の画素（1，5，9…93）が格納してあり、２ポートＳＲＡＭのＮｏ.８の縦方向にはＵＬの（４の倍数）＋２の画素（2，6，10…94）が格納してあり、２ポートＳＲＡＭのＮｏ.１２の縦方向にはＵＬの（４の倍数）＋３の画素（3，7，11…95）が格納してある。さらに、２ポートＳＲＡＭのＮｏ.１，２，３及びＮｏ．５，６，７及びＮｏ.９，１０，１１及びＮｏ.１３，１４，１５の各組についても同様に格納してある。
【００８２】
再び図８に戻って、１ＵＬ分のデータは縦方向に並ぶ４つの２ポートＳＲＡＭに１画素づつ順に格納されている。最初のＵＬデータはＮｏ．０，４，８，１２（図８の４Ｎのライン）に書き込まれる。次のＵＬデータはＮｏ．１，５，９，１３（図８の４Ｎ＋１のライン）に書き込まれる。図８のバッファ形態では、歪補正処理によって補間位置がどのように移動しても、バッファ内の任意の座標周辺１６点を１回のアクセスで同時に抽出することができる構成としてある。
【００８３】
データ送信可否判定回路３０が後段回路からリクエスト（ＲＥＱ_Ｎ）を受信し、次のＵＬデータが送れる状態にあればリクエスト受付（ＧＲＡＮＴ_Ｎ）を出力する。このリクエスト受付（ＧＲＡＮＴ_Ｎ）は自分自身へのトリガとなり、補間位置算出回路２２が動作を開始し、１ＵＬ分動作したら次のＵＬ先頭座標を算出して終了する。
【００８４】
補間位置算出回路２２が動作開始して補間位置（Ｘ１，Ｙ１）を出力するのと同期するようにデータ送信可否判定回路３０からは歪補正係数算出回路２１へ動作トリガ（ｔｒｉｇ）を送る。歪補正係数算出回路２１も補間位置算出回路２２と同様、１ＵＬ分動作したら次のＵＬ先頭座標を算出して終了する。
【００８５】
読み出しアドレス生成回路２５は、入力される補間座標から、２ポートＳＲＡＭ２６の１６個夫々に読み出しアドレスを発行する。
【００８６】
次に、読み出し及び補間の方法について説明する。
補間方法を図７に示す。図７は補間回路２７における補間演算のイメージ図である。補正座標位置Ｐの座標（Ｘ’，Ｙ’）は前出の［式１］で既に求めてある。この座標における画素値（輝度データ）を求めるのに、座標Ｐ（Ｘ’，Ｙ’）の周辺１６点の画素データＤ0〜Ｄ15から求める。その内、Ｄ0が１６個のメモリのどのメモリから来ているかが分かれば、Ｄ1，Ｄ2…，Ｄ15はＤ0に対する位置関係から分かる。後述するように、Ｄ0は補間位置の座標によって求まる。
【００８７】
図１０は１６個の２ポートＳＲＡＭからなるバッファからの読出し例を示している。今、補間位置（Ｘ’，Ｙ’）がＸ’＝１０．…，Ｙ’＝５０．…（…は小数点以下の数字）とすると、Ｘ’＝１０．…は（４×０＋１０）以上であるから、横方向には２ポートＳＲＡＭの４Ｎ＋１０番目のＵＬより若干右側に補間位置が来る。Ｙ’＝５０．…は（４×１２＋２）以上であり、４Ｎ＋１０番目のＵＬで、５０番目の画素が格納されているメモリ（Ｎｏ．１０）が図７における画素Ｄ5を出力するので、Ｄ0を出力するメモリは、その左上にあるＮｏ．５となる。
【００８８】
図７のＤ0〜Ｄ15に対応する画素データは図１０の○部であるので、それらが出力されるようにアドレスを生成する。
【００８９】
Ｎｏ．０から出力される画像データがＤ0には対応しない。図１０の例では、Ｎｏ．５からの出力がＤ0に対応する。このため、どのメモリからどのデータが出力されているかを識別するため、読出しアドレス生成回路２５からデータ列制御信号を出力し、これによって、補間演算を行う補間回路２７はＤ0が２ポートＳＲＡＭ２６のどこから出ているかを認識して１６点補間を行う。
【００９０】
画素データＤ0〜Ｄ15が分かれば、［式２］の補間式による補間処理を行うことによって、補正座標位置の画素データをＤoutとして求めることができる。
【００９１】
【式２】

最後のＵＬデータ出力の後、バッファ開放量算出回路３１は今処理し終わったＵＬ先頭座標と次のＵＬ先頭座標との差を計算し（図１７参照）、必要なくなったデータの蓄積されているバッファを開放するため、バッファ開放量をバッファ空き容量監視回路２９へ出力する。ただし、歪中心を跨ぐ時のバッファ開放量は、図１６(b)に示すようにｐｏｓｔＵＬＢ１，ｐｏｓｔＵＬＢ２の値変化も考慮した量を開放することが好ましい。 図１６(a)に示すｐｒｅＵＬＢ、ｐｏｓｔＵＬＢとは、ＵＬを形成する画素のうち、最初の画素の座標位置に対して、行方向に対して前側及び後側に所定幅の領域を設けたもので、これらをそれぞれｐｒｅＵＬＢ、ｐｏｓｔＵＬＢと定義している。歪中心を跨いだ時はバッファ開放量を通常のＵＬ先頭座標の差ではなく、参照値の変わるｐｏｓｔＵＬＢ（ＵＬＢ：Unit Line Bufferの略、ユニットラインバッファ）の変化量を参照してバッファ開放量を調節する（調整量は、ｐｏｓｔＵＬＢ１−ｐｏｓｔＵＬＢ２で、図１６(b)参照）。
【００９２】
（ＵＬ先頭座標の差）＋（調整量）がマイナスになる場合は、マイナス値を記憶しておく。処理が進んでバッファ開放量がマイナス値を超えるまでバッファの開放を行わない。
【００９３】
前述のバッファ開放量の算出と共に、バッファ開放量算出回路３１は次のＵＬ処理にあとどれだけ前段回路からデータが必要かをデータ送信可否判定回路３０へ送信する。
【００９４】
バッファ空き容量監視回路２９は前述のバッファ開放量の算出によってバッファに空きができたら、前述の前段回路へのリクエストを行う。
【００９５】
データ送信可否判定回路３０は書込みアドレス生成回路２８の内部カウンタ（ＢＬＣ）とバッファ開放量算出回路３１からの人力とｐｒｅＵＬＢ値とに基づいて、次のＵＬデータを送れるかどうかを判断する。データ送信可否判定回路３０は、後段回路のリクエストに対してリクエスト受付（GRANT_N）を返す。
【００９６】
エラー検出回路３２は、読出しアドレス生成回路２５に入力される座標がブロックライン（ＢＬ）の左端（図１１参照）や右端を逸脱したり、ブロックライン（ＢＬ）の上下端（図１２参照）を逸脱したり、歪量がｐｒｅＵＬＢ、ｐｏｓｔＵＬＢの設定値を逸脱（図１３参照）した場合にエラーを出力する。これらのエラーの出力は、座標を判別して出力することになる。図１１及び図１２の場合には、エラーを出力するが、処理は継続する。図１３の場合も、エラーを出力するが、処理は継続する。図１３のｐｒｅＵＬＢ、ｐｏｓｔＵＬＢの場合については、エラーを出力するだけではなく、逸脱した量を記憶し、レジスタに設定してＣＰＵ４がデータとして取得できるようにしておく。
【００９７】
このように、ブロックライン処理で、出力側の画像に対して入力側の画像を補間算出しているときに入力画像の範囲からはみ出すようなときがある。その場合、入力範囲にデータがない部分で補間データを生成することになり、エラー検出回路３２から補間ができない旨のエラーを出力することになる。
【００９８】
図１４にあるように、歪の中心座標に対して、画像の歪み方は対照的になる。また、歪みは一般的に中心に近いほど小さく、離れるほど大きくなる。そこで、ＢＬの両端にあるＵＬの変形を考慮し、バッファを確保すれば、全てのＵＬにおいて、補間処理でデータが不足することは通常はない。そこで、補間する最初の座標（図ではブロックラインＢＬ内の画像データの一番上の×印にて示す先頭座標）に対して前方（図示右側）に所定の領域ｐｒｅＵＬＢを、後方（図示左側）に所定の領域ｐｏｓｔＵＬＢを設ける。
【００９９】
サポート機能については、本願と同日に出願した特願２００３‐２０２４９３号において、「歪補正範囲算出部」と称する部分に該当する。図２７(a)において、注目領域がＢＬとなるが、歪補正の座標変換で同図(b)のように変形される。このとき、Ｘ’TL，Ｙ’TL，ＸLmin，ＸLmaxからｐｒｅＵＬＢ1／ｐｏｓｔＵＬＢ1が決められる（図１６(a)参照）。同様に、Ｘ’TR，Ｙ’TR，ＸRmin，ＸRmaxからｐｒｅＵＬＢ2／ｐｏｓｔＵＬＢ2が決められる。
【０１００】
さらに、ＵＬ処理において、歪の中心を通過すると、光学的歪の曲がり方が逆方向となる。すると、図１４のように、歪中心の左右ではｐｒｅＵＬＢ、ｐｏｓｔＵＬＢの値が異なる。この値を、左右どちらにも対応できるよう、図１５のように右側で最も大きいｐｏｓｔＵＬＢを採用し、左側で最も大きいｐｒｅＵＬＢを採用すると、左側又は右側の一方側の処理のときに必要もないのにもう一方側のデータを大きく（すなわち、バッファを広く）確保しておかなければならず、バッファの無駄である。
【０１０１】
そこで、ｐｒｅＵＬＢとｐｏｓｔＵＬＢで予め同じ大きな値で両方決めておくのではなくて、ｐｒｅＵＬＢとｐｏｓｔＵＬＢを変数としてそれぞれに値を入れるようにする。歪の中心座標を過ぎたら、これらの値を変えてやって確保量を変えるようにする。すなわち、歪中心前後でｐｒｅＵＬＢ、ｐｏｓｔＵＬＢを変化することにより、内部バッファの使い方に無駄をなくし、小さなバッファで比較的大きな歪補正を行える。
【０１０２】
ｐｒｅ，ｐｏｓｔＵＬＢをレジスタ８５に設定するが、それらの値を生成するためには、歪の変形を考慮して画像データの入力範囲を算出可能とするサポート機能が必要となる。このサポート機能は、歪補正処理部８の歪補正処理機能に付加するなどして設ければよい。メモリ制御部８３は、１ＵＬ処理中は他の処理によって前記領域ｐｒｅＵＬＢ，ｐｏｓｔＵＬＢが上書きされないように制御する。前記領域ｐｒｅＵＬＢ，ｐｏｓｔＵＬＢの設定は、ＣＰＵ４からレジスタ８５（図２参照）に設定しても構わないし、ＣＰＵで自動的に計算をして設定してもよい。
【０１０３】
歪補正処理において１６点補間すると前に述べたが、図１６(a)に示すように、歪補正によって補間位置は１ＵＬ処理の間にａ（ただし、ａ＞０）だけ動き得る。１６点補間する関係上、入力画像範囲としてはｐｒｅ側，ｐｏｓｔ側とも上記間隔ａに、補間に必要な画素分の間隔ｂL，ｂRを加えた範囲が必要となる。
【０１０４】
歪中心を越えると、ｐｒｅ，ｐｏｓｔＵＬＢの値を変えることで、バッファを必要最小限に絞る。また、前述の「歪補正算出部」で、補間分を考慮して結果を出力させてもよい。
【０１０５】
歪補正処理部８の持っている内部バッファ８２は、図８の場合、最大で縦９６画素の横に１６ライン（画素）である。これを使って歪補正していくことになる。ブロックライン処理のデータは少なくとも１ＵＬずつ図示右方向にスイープしていく。一気に数ＵＬ開放されることもあるので、必ずしも１ＵＬずつスイープするわけでない。内部バッファ８２は、はじめは空の状態からスタートするので、１６ライン分入ってきて、歪補正処理を行う。これらのユニットラインを処理していくと、左側のラインに不要なデータ（１から数ＵＬ分のデータ）が生じることになる。なお、処理したら必ず不要になる訳ではなく、拡大率が大きい時は不要になるまで数ＵＬかかることもある。不要なデータ分はバッファを開放（すなわち、上書き許可）していって新しいデータを入れていくことにする。スイープしていく画像は右側にずれていく。最大１６ライン読み込めるサイズしかないので、要らなくなったデータ領域は開放して新しいデータを順次上書きしていくようにする。不要となったバッファ領域は全て一気に開放してしまう。その開放量は１ラインであるかもしれないし、５ラインであるかもしれない。例えば、はじめの３ライン分のデータがバッファに入っていたものを以降の処理に不要であるので開放してしまって次のデータを受け付けるようにして、次のデータが上書きされる。
【０１０６】
ところで、１ＵＬ処理が終わった後に開放できるバッファの量は、次のＵＬ先頭座標が決まらなければ分からない（図１７参照）。内部バッファは小さいので、使わなくなったデータが格納されている部分をできるだけ早くに開放する為、次のＵＬ先頭座標を算出して開放量を取得し、バッファを開放することで内部バッファを有効に利用できるようにする。ところが、歪補正に対して後段の回路ブロックからリクエストが来てそれに対するグラントを返して１ＵＬのデータを後段へ出力し、また次のリクエストが来てからグラントを返した後、座標を求めてというように動作させると、バッファの開放が遅れ、これに伴って、新しいデータの取り込みも遅れ、結果として、パイプライン動作に長い空きができてしまうことが起こり得る。そこで、図１８に示すように先の歪補正処理（図示の座標1〜ｎの処理）を行っている段階で、後段の回路ブロックからリクエストに関係なく、座標ｎの処理が終わったら次の先頭座標ｎ＋１を演算して開放量を求め、バッファを早期に開放してしまう。これにより、前段回路ブロックから新たなデータ入力が比較的早く行なわれ、パイプラインの空きを詰めることができる。しかしながら、図１８の場合であると、座標ｎの処理をした後、画素ｎ＋１を座標を計算してから処理を終了するので、図１９のようにバッファ内に次のＵＬ処理に必要なデータが揃っているのに、後段からのリクエストが連続するとパイプラインに空きが生じる。依然としてパイプラインの空きは存在するので、リクエストが連続した時は、装置全体の動作速度に悪影響を及ぼす恐れがある。
【０１０７】
そこで、図１８の改善策として、図２０のように、なるべく早くに次の座標が決まってバッファの開放量を求める方法がある。図１８のように一番最後に次の先頭座標を求めるよりももっと早い段階で次のＵＬ先頭座標を取得しておき、必要なデータが揃っていると判断されたら、次のＵＬ処理を連続して行うことで処理の空きを小さくする。すなわち、１ＵＬ処理が終了する前にＵＬのデータをレジスタ等に保持しておいて、次のＵＬの先頭の座標位置を算出することで、予め次のＵＬ処理の開放量を求めておく。図２０では、先の歪補正処理（図示の座標1〜ｎの処理）の２画素目に次のＵＬの先頭座標を求めておいてバッファ開放量を早い段階で取得する。これによって、バッファの開放を早くすることができ、図２１に示すように図１９に比べてパイプラインの空きをかなり少なくできる。しかしながら、回路的に複雑となり、制御も困難となる問題点がある。
【０１０８】
そこで、図２０をさらに改善したのが、図２２である。図２２では、現在処理中のＵＬが終わった時に開放する量（開放量１）は前のＵＬ処理中に既に分かっている。座標ｎを算出した後、２つ先のＵＬ先頭座標を計算し、次のＵＬ処理が終わった時に開放できる量（開放量２）を予め求めておく。このようにすると、図２０の場合のようにＵＬ処理中に例外的な座標生成（座標２の生成）を行う必要がない。このように、次のＵＬの先頭ではなくて次の次の２つ先のＵＬの先頭座標を求めておけば、図２０のような例外的な処理がなくなるので、回路的には簡単になる。
【０１０９】
【発明の効果】
以上述べたように本発明によれば、バスの転送量やメモリの容量を大きく増大させることなく、歪補正を実現することができる。すなわち、小さなバッファ容量で比較的大きな歪補正処理を行うことが可能となる。
【図面の簡単な説明】
【図１】本発明の一実施の形態の画像処理装置の全体構成を示すブロック図。
【図２】歪補正処理部の構成を示すブロック図。
【図３】歪補正処理部における座標変換の概念図。
【図４】陣笠型歪を示す図。
【図５】入力画像データを示す図。
【図６】歪補正処理部の詳細な構成を示すブロック図。
【図７】補間回路における補間演算のイメージ図。
【図８】歪補正処理部における内部メモリ部を説明する図。
【図９】図８の補足説明をする、書き込み順番を示す図。
【図１０】図７の補正座標位置を算出するのに必要なＤ0を求めるための説明図。
【図１１】エラー処理例を示す図。
【図１２】他のエラー処理例を示す図。
【図１３】他のエラー処理例を示す図。
【図１４】歪補正処理に要するバッファ量について説明する図。
【図１５】歪補正処理に要するバッファ量について説明する図。
【図１６】歪補正処理に要するバッファ量について説明する図。
【図１７】ＵＬ処理に伴う開放量の算出方法について説明する図。
【図１８】ＵＬ処理に伴う開放量の算出方法について説明する図。
【図１９】図１８の処理に伴うパイプライン処理の空きについて説明する図。
【図２０】ＵＬ処理に伴う開放量の算出方法について説明する図。
【図２１】図２０の処理に伴うパイプライン処理の空きについて説明する図。
【図２２】ＵＬ処理に伴う開放量の算出方法について説明する図。
【図２３】格子状の被写体、樽型歪を生じた撮影画像、糸巻き型歪を生じた撮影画像を示す図。
【図２４】一般的なデジタルカメラの画像処理手順の概念、及び従来のデジタルカメラの画像処理装置におけるブロック構成を示す図。
【図２５】本発明に係る実施の形態におけるフレームメモリからの画像データの読み出し順序を説明する図。
【図２６】本発明に係る図１の第１，第２データ順序変換部の構成を示すブロック図。
【図２７】歪補正範囲算出部における、歪補正処理を行う際の入力画像範囲を算出する動作を説明する図。
【符号の説明】
１…ＣＣＤ
３…バス
４…ＣＰＵ
５…フレームメモリ
６…第１データ順序変換部
７…イメージプロセス回路
６…第１データ順序変換部
８…歪補正処理部（歪補正手段）
９…第２データ順序変換部
１０…ＪＰＥＧ処理部
１１…メモリカード（記録媒体）
８２…内部メモリ部（バッファ）
８３…メモリ制御部
８５…制御レジスタ[0001]
BACKGROUND OF THE INVENTION
The present invention mainly relates to an image processing apparatus and an image processing method used in an electronic imaging apparatus such as a digital camera, and more particularly to an image processing apparatus that realizes a distortion correction function without increasing a circuit scale or a data transfer amount. And an image processing method.
[0002]
[Prior art]
In general, distortion aberration appears in the optical system of a camera regardless of whether it is a digital camera or a silver salt camera. In addition, most cameras that are currently available are capable of optical zooming, and in this case, the state of distortion changes from the wide end to the tele end. There are many barrel distortions at the wide end, and there are many pincushion distortions at the tele end.
[0003]
The distortion is observed as barrel distortion or pincushion distortion when, for example, a grid-like subject is photographed.
[0004]
FIG. 23A shows a lattice-like object, FIG. 23B shows a photographed image with barrel distortion, and FIG. 23C shows a photographed image with pincushion distortion.
[0005]
By the way, in a digital camera, after various image processing is performed on data of an image sensor such as a CCD, data compressed by a compression method such as JPEG is recorded on a recording medium such as a memory card.
[0006]
FIG. 24A shows a concept of an image processing procedure performed by a general digital camera. The imaging signal captured from the CCD is subjected to processing such as pixel defect processing and A / D conversion in a preprocess, and is stored in a frame memory such as an SDRAM. Next, the image data stored in the frame memory is read out, and various image processes are performed by the image process 1, the image process 2,. The image data after the image processing is further compressed by a compression method such as JPEG and recorded on a memory card or the like as a recording medium.
[0007]
FIG. 24B is a block diagram of an image processing apparatus of a conventional digital camera (for example, a conventional example in Japanese Patent Laid-Open No. 2000-31327).
[0008]
In FIG. 24B, a conventional image processing apparatus includes a pre-processing circuit 102, a plurality of image processing circuits 106-1 to 106-n, a JPEG processing unit 107, a frame memory 105, and a recording medium as well as a CPU 104 on a bus 103. A memory card 108 is connected. Under the control of the CPU 104, the image pickup signal from the CCD 101 is subjected to pixel defect processing, A / D conversion, and the like in the preprocess circuit 102, and then temporarily stored in the frame memory 105 through the bus 103. Next, image data is read from the frame memory 105 and input to the image process circuit 106-1 through the bus 103 to perform predetermined image processing, and the image data is rewritten to the frame memory 105 through the bus 103 again. In the same manner, data is sequentially exchanged between the frame memory 105 and the image process circuits 106-2 to 106-n via the bus 103. Finally, the JPEG processing unit 107 performs JPEG compression processing, and the image The processed data is temporarily stored in the frame memory 105, and the processed data read from the frame memory 105 is recorded on the memory card 108 or the like. In the above image process and JPEG processing in each stage, image processing is performed in units of small areas (block lines).
[0009]
On the other hand, a prior art for correcting distortion as part of this image processing has been disclosed. For example, there are JP-A-6-181530 and JP-A-10-224695.
[0010]
Japanese Patent Laid-Open No. 6-181530 discloses an image generated by an imaging zoom lens when the detection unit detects that the imaging position of the imaging zoom lens at the time of shooting is within a position where distortion is large. Is corrected by reading out imaging data of a solid-state imaging device based on geometric deformation.
[0011]
In Japanese Patent Laid-Open No. 10-224695, data taken by a solid-state imaging device that receives light passing through an optical system is stored in a random access video memory, and a random read timing generation circuit corrects aberrations caused by the optical system. And correcting the distortion aberration of the optical system by reading out the signals generated by the solid-state imaging device in a predetermined order based on the aberration correction data and generating a video signal. Yes.
[0012]
[Patent Document 1]
JP 2000-31327 A (page 3, FIG. 19)
[0013]
[Patent Document 2]
Japanese Patent Laid-Open No. 6-181530 (first and second pages, FIG. 2)
[0014]
[Patent Document 3]
Japanese Patent Laid-Open No. 10-224695 (page 1-3, FIG. 1)
[0015]
[Problems to be solved by the invention]
However, in the conventional example of Patent Document 1, there are many data exchanges between the frame memory and each image process circuit, and the data transfer amount of the bus increases.
[0016]
In Patent Document 2, a large-capacity buffer memory is required to process one line. In addition, the image size that can be processed is limited by the capacity of the buffer memory.
[0017]
In Patent Document 3, a buffer memory is not required for processing by random access. However, generally, random access video memory such as SDRAM takes transfer time.
[0018]
In view of the above problems, an object of the present invention is to provide an image processing apparatus and an image processing method capable of realizing distortion correction processing without greatly increasing the transfer amount of a bus or the capacity of a memory. .
[0019]
[Means for Solving the Problems]
  The invention of claim 1Connected to image processing means so that pipeline processing is possible.In an image processing apparatus having a distortion correction unit that performs distortion correction processing on image data, the distortion correction unit includes:A coordinate position of a pixel in the corrected image after distortion correction, and an interpolation coordinate generation unit that generates an interpolation coordinate of the pixel in the image before distortion correction corresponding to the coordinate position;The image dataIn blocksStorebufferA memory unit;bufferFor the memory partimageA memory control unit that controls writing and reading of data;Based on the generated interpolated coordinates,SaidbufferRead from memoryBlock-wiseimage dataGenerate pixel data of the interpolation coordinates fromPerform interpolation calculationAn interpolation calculation unit, and the memory control unit includes a predetermined number of pixels arranged in a line in a column direction when writing image data before distortion correction from the image processing unit to the buffer memory unit. Control is performed so that unit lines as image data are sequentially written in the row direction, and when reading from the buffer memory unit, the image data after distortion correction processing is sequentially output in the row direction in units of the unit lines. Next, read control is performed on the image data stored in the buffer memory unit.It is characterized by that.
[0020]
  In this configuration, image data is stored in the memory unit as an internal buffer.In blocksIt is possible to store and perform interpolation calculation for distortion correction processing using the data.
  In addition, since data is written to and read from the memory unit in units of UL, the buffer capacity of the memory unit may have a capacity of several UL, that is, at least 1 UL according to the amount of optical distortion. Distortion correction can be realized without greatly increasing the amount or memory capacity.
[0023]
  Claim2The invention of claim1In the image processing apparatus, the memory control unitUnit lineThe coordinate position of the first pixel to be processedBased onCoordinate positionAs,This reference coordinate position andRow directionofFront coordinate positionPreULB is betweenwidthMemory that isProvide an area,Row-wiseRear coordinate positionBetween and postULBwidthMemory that isProvide an area,Unit lineDuring processing, other processingData byBy the aboveBothControl is performed so that the memory area is not overwritten.
[0024]
With this configuration, it is possible to ensure that data in the preULB and postULB areas of the memory unit as a buffer necessary for processing is not overwritten.
[0025]
  Claim3The invention of claim1Or2In the image processing apparatus, the memory control unit further includes theUnit lineDuring the processing ofcircuitA buffer that can be opened to input data fromOpenA buffer opening amount calculating circuit for calculating the amount, and a subsequent stage of the distortion correcting meanscircuitA data transmission availability determination circuit for determining whether data transmission is possible,HaveAboveThe buffer opening amount calculation circuitThe difference between the coordinates for the first pixel of the unit line currently being processed and the coordinates for the first pixel of the next unit line to be processedThe buffer opening amountAsCalculationAt the same time, the amount of data from the previous circuit required for processing of the next unit line is transmitted to the data transmission availability determination circuit.AndAboveData transmission availability judgment circuit, The amount of data transmitted from the buffer opening amount calculation circuit,The value of the preULBAnd based onIt is characterized by determining whether or not data transmission is possible.
[0026]
With this configuration, the memory unit as a buffer necessary for processing can be efficiently used with a small capacity.
[0027]
  Claim4The invention of claim2Or3In the image processing apparatus, the preULBofValue, postULBofThe values are characterized in that different values can be set on the front side and the rear side with respect to the distortion center.
[0028]
In this configuration, as shown in FIG. 14, based on the fact that the values of preULB and postULB are different on the left and right of the distortion center, if these values are made large as shown in FIG. Necessary buffers must be secured widely, which is a waste of buffers. By changing preULB and postULB before and after the distortion center, it is possible to eliminate wasteful use of the internal buffer and perform relatively large distortion correction with a small buffer capacity.
[0029]
  Claim5The invention of claim 1 to claim 13In any one of the image processing apparatuses, the image processing apparatus further includes:,SaidDepending on the calculated buffer release amountbufferInsideAnd a buffer free space monitoring circuit for detecting a free space in the buffer. When a free space in the buffer is detected by the buffer free space monitoring circuit, data can be written to the free space. .
[0030]
With this configuration, it is possible to perform a pipeline operation that can be input during data output.
[0031]
  Claim6The image processing apparatus according to claim 1, whereinbufferThe memory unit has a plurality of data read and write operations that can be performed simultaneously.bufferThe memory control unit further comprises thebufferA write address generation circuit for performing data write control to the memory unit;bufferA read address generation circuit for generating an address for simultaneously reading data necessary for the interpolation operation from the image data stored in the memory unit, and the data write control is different for the data read simultaneously.bufferIt is characterized by being written in.
[0032]
According to this configuration, it is possible to simultaneously read out information on a plurality of pixels (for example, 16 pixels) around an arbitrary coordinate as data necessary for the interpolation calculation from the image data in the buffer, and the calculation process can be quickly performed. Yes.
[0033]
  Claim7The invention of claim2~4In any one of the image processing apparatuses, the image processing apparatus further includes distortion correction processing.Interpolation obtained byThe coordinates areUnit lineProcessingBlock line image data to performWhen coordinates deviating from preULB and postULB are generated in the column direction with respect to the interpolation coordinates forInterpolation obtained by processingCoordinates from the previous stageIsEnteredGainError when there is no coordinateDo outputIt has an error detection circuit.
[0034]
In this configuration, the error detection circuit detects and notifies the CPU when a parameter is erroneously input or the estimation of the distortion correction amount is too small. In particular, regarding preULB and postULB setting over (see FIG. 13), there is a case where accurate estimation cannot be performed in advance, so that the setting value can be easily corrected by notifying the CPU of the over amount.
[0035]
  Claim8The invention of claim7In the image processing apparatus, when the error output is made, Input range of image data of the block lineResetDistortion correctionThe process can be re-executed.
[0036]
  Claim9The invention of claim1~5In any one of the image processing apparatuses,1 unit lineofDistortion correction for the last pixelAfter processing, the next 1Unit lineOr even the next oneOne unit lineAt firstPixels that perform distortion correction processingInterpolation coordinatesIs calculated,The buffer opening amount is obtained based on the interpolation coordinates.It is characterized by that.
[0037]
In this configuration, the amount of the buffer that can be released after the end of 1UL processing is used unless the next UL head coordinate is determined (see FIG. 17). Since the internal buffer is small, in order to release as soon as possible the part storing the data that is no longer used, the next UL head coordinate is calculated to obtain the release amount (see FIG. 18), and the buffer is released. The internal buffer can be used effectively.
[0038]
  Claim10The invention ofEnteredIn an image processing method for performing distortion correction processing on image data, the distortion correction processing is performed.UData writing and reading are controlledbufferIn the memory section,EnteredImage dataIn blocksStore andGenerate the coordinate position of the pixel in the corrected image after distortion correction, and the interpolation coordinate of the pixel in the image before distortion correction corresponding to this coordinate position, based on the generated interpolation coordinate,SaidbufferRead from memoryBlock-wiseimage dataGenerate pixel data of the interpolation coordinates fromPerform interpolation calculationThe bufferMemory partWhen writing to the buffer, control is performed so that unit lines, which are image data composed of a fixed number of pixels arranged in a line in the column direction, are sequentially written in the row direction as units.Memory partWhen reading from the buffer, the image data after distortion correction processing is sequentially output in the row direction with the unit line as a unit.Memory partRead control for image data stored inIt is characterized by that.
[0039]
  In this method, image data is stored in a memory unit as an internal buffer.In blocksIt is possible to store and perform interpolation calculation for distortion correction processing using the data.
[0040]
  Claim11The invention of claim2~4The image processing apparatus according to any one of the above, wherein the image processing apparatus further includes a distortion correction process performed by the distortion correction unit.Need forHas a distortion correction range calculator that calculates the input image range.The input image range is obtained as a region including a corresponding region of image data before distortion correction processing obtained by performing distortion correction coordinate transformation on a region of interest in the image data after distortion correction processing.It is characterized by that.
[0041]
  Claim12The invention of claim11In the image processing apparatus described above, the distortion correction range calculation unit includes:For a region of interest in the image data after the distortion correction processingApply the prescribed distortion correction formulaby doingCoordinate conversion is possible, distortion correctionprocessingLater in the output image rangeKick, Corresponding to each of the four sideseachPixelAgainstThe coordinate transformationTo goCorresponds to the maximum and minimum values and 4 vertices of the generated transformation coordinateseachPixelAgainstCoordinate transformationTo goAt least one of the generated conversion coordinates is calculated, and the preULB and the postULB are calculated by the distortion correction range calculation unit.CalculationresultOn the basis of theTo calculateTheFeatures.
[0042]
In these configurations, the above-described pre and postULB are set in the registers. However, in order to generate these values, a support function that enables calculation of the input range of image data in consideration of distortion deformation is required. Become. This support function is provided as a distortion correction range calculation unit for the distortion correction processing function of the distortion correction means.
[0043]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing the overall configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram showing the configuration of the distortion correction processing unit in FIG.
[0044]
In the image processing apparatus of FIG. 1, an image pickup signal from the CCD 1 is subjected to pixel defect processing, A / D conversion, and the like by the preprocess circuit 2 under the control of the CPU 4 that controls each unit connected to the bus 3. The obtained image data is temporarily stored in the frame memory 5 via the bus 3. The frame memory 5 is configured by an SDRAM or the like and stores data before image processing and data after image processing. Next, the image data read from the frame memory 5 is input to the first data order conversion unit 6 via the bus 3. As will be described later with reference to FIG. 26 (a), the first data order conversion unit 6 includes a plurality of, in this case, two memories capable of storing data in block units. The first data order conversion unit 6 reads and stores data from the frame memory 5 in the row direction, and then sequentially reads the data in the column direction and outputs the data to the image process circuit 7.
[0045]
The image process circuit 7 performs predetermined image processing on the input data and transfers it to the distortion correction processing unit 8 as the distortion correction means at the next stage. The distortion correction processing unit 8 performs distortion correction processing on the input data and transfers it to the second data order conversion unit 9 at the next stage. As will be described later with reference to FIG. 26 (b), the second data order conversion unit 9 includes a plurality of, in this case, two memories capable of storing data in block units. The second data order conversion unit 9 reads and stores data in the column direction from the distortion correction processing unit 8, then sequentially reads the data in the row direction, and transfers the data to the JPEG processing unit 10. The JPEG processing unit 10 performs JPEG compression processing, temporarily stores the processing data in the frame memory 5, and records the processing data read from the frame memory 5 in the memory card 11 or the like.
[0046]
As shown in FIG. 2, the distortion correction processing unit 8 positions the corrected image after distortion correction (interpolation positions, X and Y) and the corresponding original image position before distortion correction (X ′, Y). ') And an internal memory unit as a buffer memory (hereinafter simply referred to as a buffer) for temporarily storing part of the image data from the circuit of the preceding block of the distortion correction processing unit 8 82, a memory control unit 83 for controlling writing and reading with respect to the internal memory unit 82, and image processing is performed according to the converted coordinates (X ′, Y ′) of the image position before distortion correction to correct distortion of the data. And an interpolation calculation unit 84.
[0047]
As shown in FIG. 2, the interpolation coordinate generation unit 81 includes a coordinate generation unit 811 that generates interpolation coordinates (X, Y), and a predetermined distortion correction equation [formula for the generated interpolation coordinates (X, Y). 1] A distortion correction coordinate conversion unit 812 that outputs coordinates (X ′, Y ′) before correction converted by applying (described later), and interpolation coordinates (X, Y) from the coordinate generation unit 811 and distortion correction The selector 813 can selectively output the converted coordinates (X ′, Y ′) from the coordinate conversion unit 812. The coordinate generation unit 811, the distortion correction coordinate conversion unit 812, the selector 813, and the memory control unit 83 in the distortion correction processing unit 8 operate according to the set values for each block set in the control register 85 that stores control data. Further, the status of the processing result can be referred from the CPU.
[0048]
  In the image processing apparatus according to the present invention configured as described above, the first data order conversion unit 6 to the JPEG processing unit 10 are piped through an information transmission path different from the bus 3 without going through the bus 3. It is connected so that line processing is possible, and image data is transferred and processed in predetermined block units in a two-dimensional pixel array.. BaData transfer through the memory 3 is only transfer from the frame memory 5 to the first data order conversion unit 6, transfer from the JPEG processing unit 10 to the frame memory 5, and transfer from the frame memory 5 to the memory card 11. Therefore, compared with the conventional example (FIG. 24B) in which data is exchanged between the frame memory and each image process circuit, the amount of data transferred by the bus 3 can be greatly reduced. The load can be greatly reduced. Further, in the example shown in FIG. 1, only one image process circuit 7 for performing image processing is provided, but a plurality of image process circuits 7 may be provided. Furthermore, in the figure, the distortion correction processing unit 8 is provided at the subsequent stage of the image process circuit 7, but the configuration may be reversed.
[0049]
In the image process circuit unit configured by the first-stage image process circuit 7 and the distortion correction processing unit 8 which is the second-stage image process circuit, a pipeline register is not shown as a pipeline register before or inside each of the image process circuits 7 and 8. A small-capacity memory is arranged, and the image process circuits 7 and 8 are configured to perform pipeline processing operations via the small memory. These small-capacity memories are used to store peripheral data necessary for image processing when the image processing circuits 7 and 8 perform spatial image processing, and read out the image data in units of blocks and rearrange the data. It is provided because it is necessary to carry out processing and the like.
[0050]
When data transfer from the image process circuit 7 to the image process circuit 8 is free in the internal memory unit 82 of the distortion correction processing unit 8 which is a subsequent image process circuit, the image process circuit 7 in the previous stage is transferred from the distortion correction processing unit 8. In response to this, a request (REQ) is transmitted from the image process circuit 7 in the preceding stage, and at the same time, the free information in the internal memory unit 82 as a buffer is reduced by one. The distortion correction processing unit 8 in the subsequent block repeats this request and request reception, and when the space becomes 0, the request is withdrawn. The distortion correction processing unit 8, which is a subsequent block, receives data from the image process circuit 7 of the previous block simultaneously with the request reception (GRANT) from the previous block or after the request reception (GRANT), and writes the data to the internal memory unit 82. To do. When multiple stages of image process circuits are connected so that pipeline processing is possible, data is transferred from the previous stage to the subsequent stage as described above, and the data is written to the memory section of each image process circuit. After the image processing is performed, data is sequentially transferred while performing image processing from the preceding block to the subsequent stage based on a request from the subsequent stage.
[0051]
FIG. 3 shows a conceptual diagram of coordinate conversion in the distortion correction processing unit 8. FIG. 3A shows the original image data, FIG. 3B shows the corrected image, FIG. 3C shows the coordinate position (X, Y) of the corrected image in FIG. Data coordinates (X ′, Y ′) at the coordinate position P converted to the coordinates of the original data (the coordinates are not necessarily coincident with the positions of a plurality of pixels actually constituting the original data) ). The coordinate position (X ′, Y ′) at the P point is calculated using the coordinates of the 16 pixels around the P point, and the image data at the P point is the pixel data of the surrounding 16 points. Is used for interpolation calculation. The interpolation calculation unit 84 performs processing to interpolate the data at the position of the point P from the surrounding 16 pixel values (luminance data). The generation of the interpolation coordinates by the interpolation position generation unit 811 in FIG. 2 means which pixel position (X, Y) is indicated on the corrected image side in FIG.
[0052]
In [Formula 1], the pixel position (X ′, Y ′) before distortion correction with respect to the pixel position (X, Y) after distortion correction can be calculated. However, as described above, the pixel position (X ′, Y ′) before distortion correction is not always an integer value corresponding to the pixel position on the original image data.
[0053]
According to [Formula 1], the coordinates (X, Y) in FIG. 3B are converted into coordinates (X ′, Y ′) as shown in FIG. Thereby, it is possible to know where the coordinate position data in the original data should be created, and the interpolation equation [16-point interpolation processing described later] is performed on the data at the position (X ′, Y ′). It can be obtained by calculating from 16 known pixel values (input image data indicated by black circles in FIG. 5) using Equation (2).
[0054]
[Formula 1]

Z in [Formula 1] is the distance from the strain center (Xd, Yd) to the point (X, Y) of interest. According to [Expression 1], the coordinates (X ′, Y ′) of the original image that is distorted with respect to the point (X, Y) of the corrected image are calculated. Supplementally explaining [Formula 1], M is a correction magnification for correcting a phenomenon in which an image protrudes or becomes insufficient after correction when theoretically correcting using data of the optical system. Sx and Sy are sampling ratios for correcting a phenomenon in which vertical and horizontal spatial sampling intervals are different, such as thinning-in. Xoff and Yoff are values of center deviation for correcting a phenomenon in which the subject position deviates from the position at the time of photographing after the distortion correction process by the distortion correction process.
[0055]
In [Expression 1] according to the present invention, higher-order terms (specifically, Z4, Z6,...) Are taken into consideration, so that more complex Jinkasa type distortion (see FIG. 4) can be dealt with. In addition, it is necessary to consider the distortion center (Xd, Yd), for example, when only a part of the image sensor is captured even if the centers coincide, or when the center of the optical axis and the center of the image sensor are shifted. Furthermore, in order to correct a phenomenon in which the subject position deviates from the shooting position by the distortion correction process, it is necessary to consider the center deviation correction values (Xoff, Yoff). When sampling is performed or when processing a YC image consisting of luminance data (Y) and color data (Cb, Cr), spatial sampling differs between X and Y. Therefore, the sampling ratio (SX, Sy) is introduced as a coefficient to correct the coordinate conversion.
[0056]
Next, the processing and order of image data writing and reading will be described with reference to FIGS.
FIG. 25 is a diagram for explaining the reading order of image data from the frame memory according to the present embodiment.
[0057]
The image data is normally written by sweeping in the line direction, that is, in the row direction, and is read out in the row direction at the time of reading. All the image data of one line is read and then adjacent. It is common to repeat the operation of reading all the image data of the line.
[0058]
On the other hand, the image processing apparatus according to the present invention is configured so that image data written by sweeping in the row direction is sequentially processed in the row direction image data in units of a certain length in the column direction. A small area (rectangular image data) obtained by sequentially inputting the next column to the image processing unit 7 and repeatedly to the right end of the image is referred to as a block line (BL).
[0059]
The configuration of the first data order conversion unit 6 and the second data order conversion unit 9 for enabling reading of such image data will be described with reference to FIG. FIG. 26 is a block diagram illustrating a configuration of the first and second data order conversion units.
[0060]
As shown in FIG. 26 (a), the first data order conversion unit 6 includes a plurality of memories that can store image data in units of blocks, two in this case, and the two memories 6a and 6b write data. Writing and reading can be switched alternately by the switches on the side and reading side. That is, the frame memory 5 is connected to the memory 6a and the memory 6b by a switch on the writing side so as to be switchable, and the image processing unit 7 also has the memory 6a and the memory 6b in the reading side switch. When the frame memory 5 is connected to one of the memory 6a and the memory 6b, the other of the memory 6a and the memory 6b is switched to be connected to the image processing unit 7. . That is, the memories 6a and 6b are switched so as not to be simultaneously connected to both the frame memory 5 and the image processing unit 7, and are alternately written and read.
[0061]
A part of the frame image stored in the frame memory 5 is read in the line direction in units of blocks and stored in one memory, for example, the memory 6a in this case.
[0062]
In parallel with this, the block-unit image data that has already been read from the frame memory 5 and stored in the memory 6b is sequentially read in the column direction (vertical direction) and output to the image processing unit 7. Is done.
[0063]
When writing from the frame memory 5 to the memory 6a and reading from the memory 6b to the image processing unit 7 are finished, the switch on the writing side and the switch on the reading side are switched, and then the frame memory 5 transfers to the memory 6b. The writing of the block unit image data is started, and the reading of the block unit image data from the memory 6a to the image processing unit 7 is started.
[0064]
As shown in FIG. 26B, the second data order conversion unit 9 is configured in substantially the same manner as the first data order conversion unit 6 and operates in substantially the same manner.
[0065]
That is, the second data order conversion unit 9 includes a memory 9a and a memory 9b, a write side switch, and a read side switch.
[0066]
During the operation of the second data order conversion unit 9, writing from the distortion correction processing unit 8 is performed in the column direction (vertical direction) with respect to one of the memory 9a and the memory 9b, and the memory 9a and the memory 9b Reading from the other is performed in the row direction (horizontal direction) and output to the JPEG processing unit 10.
FIG. 6 shows a detailed configuration of the distortion correction processing unit 8 in FIG.
A correspondence relationship between each unit in FIG. 6 and each unit in FIG. 2 will be described. The interpolation position calculation circuit 22 in FIG. 6 is the interpolation position generation unit 811 of FIG. 2, the selector 24 is the selector 813 of FIG. 2, and the distortion correction coefficient correction circuit 21 and the interpolation position correction circuit 23 are the distortion correction coordinate conversion unit of FIG. 812, respectively. Further, the 2-port SRAM 26 in FIG. 6 corresponds to the internal memory unit 82 in FIG. 2, and a write address generation circuit 28, a buffer free capacity monitoring circuit 29, a data transmission availability determination circuit 30, a buffer release amount calculation circuit 31, and a read address generation. The circuit 25 corresponds to the memory control unit 83 in FIG. 2, and the interpolation circuit 27 corresponds to the interpolation calculation unit 84 in FIG. Although not shown in FIG. 2, the error detection circuit 32 is provided connected to the memory control unit 83. The error detection circuit 32 has a function of notifying the CPU 4 as an error (ERROR) when a distortion amount increases in the distortion correction process and a distortion exceeding a preset value of preULB and postULB described later occurs.
[0067]
When the interpolation position calculation circuit 22 returns a grant (request acceptance) to the subsequent circuit in response to a request from the subsequent circuit of the distortion correction processing unit 8, the grant is used as a trigger for one unit line (hereinafter referred to as 1UL). The interpolation position (X1, Y1) is calculated. Here, 1UL is a unit when writing or reading a certain number of image data arranged in a line in the column direction when writing to or reading from the memory unit in the block line processing described above. is there. That is, 1UL indicates data of a certain number of pixels arranged in a line in the column direction on the block line (BL).
[0068]
The interpolation position correction circuit 23 multiplies the interpolation position (X1, Y1) by the distortion correction coefficient F from the distortion correction coefficient calculation circuit 21, and calculates the coordinate position (X ′, Y ′) of the original data before distortion correction. To do. The selector 24 selects (X1, Y1) and (X ′, Y ′). When performing distortion correction, the selector 24 selects (X ′, Y ′) and outputs it, and enlargement / reduction processing (resizing) When performing only, select (X1, Y1) and output.
[0069]
The 2-port SRAM 26 is a buffer that stores data in the distortion correction processing unit 8.
[0070]
The read address generation circuit 25 generates an address (ADR) in the 2-port SRAM 16 corresponding to the interpolation position, outputs a control signal for aligning outputs from the 2-port SRAM 26, or synchronizes with output image data. Then, the write control signal WE_N is output, or the data string control signal that indicates where D0 shown in FIG. 7 and [Equation 2] is on the 2-port SRAM is output.
[0071]
The write address generation circuit 28 generates an address (ADDRESS) of the 2-port SRAM 26 that is an internal memory in accordance with the write control signal WE, or counts up the internal counter (BLC) when data input for 1 UL is completed. .
[0072]
The data transmission permission / inhibition determination circuit 30 responds to the REQ signal from the subsequent circuit based on the BLC value, the operation state of this circuit, the next UL head coordinate, and the request (REQ) state from the subsequent circuit of the distortion correction processing unit 8. It is determined whether or not the grant (GRANT_N) can be transmitted. If grant transmission is possible, GRANT_N = 1.
[0073]
The interpolation circuit 27 performs 16-point interpolation for the image data corresponding to the interpolation position. The buffer release amount calculation circuit 31 calculates the difference between the integer part of the UL start coordinate currently being processed and the UL start coordinate to be processed next as the buffer release amount (see FIG. 17).
[0074]
The buffer free capacity monitoring circuit 29 holds the amount of data (UL) stored in the 2-port SRAM 26 as an internal buffer, and if there is free space in the buffer, a request (REQ) is sent to the preceding circuit of the distortion correction processing unit 8. = 1) is transmitted.
[0075]
Next, the circuit operation of the distortion correction processing unit 8 in FIG. 6 will be described.
First, when the 2-port SRAM 26 that is an internal memory unit (buffer) has a space, the buffer space monitoring circuit 29 sends a request (REQ) as a data request from the buffer space capacity monitoring circuit 29 to the preceding circuit. A grant (GRANT) as a request reception is received from the preceding circuit, and simultaneously with this reception, the counter (in the circuit 29) that stores the number of ULs that can be stored in the 2-port SRAM 26 is decreased by one. Data is transferred with 1 UL as one operation unit in one request and grant. When the counter reaches 0, the request is withdrawn.
[0076]
Then, data flows from the preceding circuit to the write address generation circuit 28, and writing to the 2-port SRAM 26 is performed. The internal counter (BLC) of the write address generation circuit 18 is incremented for every 1 UL input.
[0077]
The 2-port SRAM 26 is composed of a total of 16 2 × 4 SRAMs that can simultaneously read and write as shown in FIG. 8 so that the interpolation circuit 27 can perform 16-point interpolation, for example.
[0078]
Here, the 2-port SRAM 26 will be described with reference to FIG. 8, but the number of memories and the size of each memory may be different. For example, in FIG. 8, the block line width (UL length) is 96, but it may be longer or shorter. It may be determined by a balance between the circuit scale and the correction performance. Also, 4 × 4 (16 in total) may be 2 × 2 (4 in total) as long as 4-point interpolation is performed. It may be determined according to the interpolation method (how many points are to be interpolated).
[0079]
FIG. 8 shows a memory space composed of 16 2-port SRAMs. The horizontal direction is the UL order input, but the vertical direction is an address assigned to each 2-port SRAM. Let N be an integer, and the state at a certain point in time will be described. The data stored in Nos. 0, 4, 8, and 12 that are four 2-port SRAMs arranged in the vertical direction are the 4N, 4N + 4, 4N + 8, and 4N + 12th UL data. , 13 is the 4N + 1, 4N + 5, 4N + 9, 4N + 13th UL data, and the data stored in Nos. 2, 6, 10, 14 is the 4N + 2, 4N + 6, 4N + 10, 4N + 14th UL data. The data stored in Nos. 3, 7, 11, and 15 are 4N + 3, 4N + 7, 4N + 11, 4N + 15th UL data.
[0080]
No. 0, 1, 2, 3 and No. 4, 5, 6, 7 and No. 8, 9, 10, 11 and No. 12, 13, 14 which are four 2-port SRAMs arranged in the horizontal direction. , 15 are also stored in the same manner.
[0081]
The manner in which data is written to each 2-port SRAM will be described with reference to FIG. In FIG. 9, a block line having a width of 8 lines (pixels) is assumed as shown in FIG. 9A, and data is “1, 2,..., I, m”, “5,” in UL units. 6,..., J, n ”,“ 9, 10,..., K, o ”,. FIG. 9B shows where the data in the write order shown in FIG. 9A is written on the 16 2 port SRAM No. 0 to No. 15 of 4 × 4. For example, in No. 0, No. 4, No. 8, No. 12, the image data for 1 UL (data shown in order 1 to m) in the left vertical column in FIG. 9A is distributed and stored. To go. Note that the notation 0 to 92, 1 to 93, 2 to 94, 3 attached to each vertical line of the vertical two-port SRAM No. 0, No. 4, No. 8, and No. 12 shown in FIG. ˜95 is different from the notation of each unit line (1˜m..., 5˜n..., 9˜o...) Shown in FIG. In FIGS. 9 (a) and 9 (b), 1 to 16, a to p, A to P,... And 16 blocks so that the writing order can be understood for 4 × 4 16 image data. This is because the reference numerals are attached. In the case of FIG. 8, the notation given to each vertical line of the 2-port SRAM No. 0, No. 4, No. 8, No. 12 in the vertical direction corresponds to the data order in one UL in the vertical direction. 2-port SRAM No. 0, No. 4, no. 8, no. Each of 12 includes one UL pixel data No. 0 to 95 distributed alternately. Accordingly, each of the four vertical lines arranged in the vertical direction in FIG. 8 is labeled with the number of pixels in the pixel order (0 to 95) of one unit line (UL). Pixels (0, 4, 8,... 92) that are multiples of UL are stored in the vertical direction of No. 0 of the 2-port SRAM in FIG. 8, and UL is stored in the vertical direction of No. 4 of the 2-port SRAM. (Multiple of 4) +1 pixels (1, 5, 9,... 93) are stored, and in the vertical direction of No. 8 of the 2-port SRAM, UL (multiple of 4) +2 pixels (2, 6 , 10... 94), and UL (multiple of 4) +3 pixels (3, 7, 11,... 95) are stored in the vertical direction of No. 12 of the 2-port SRAM. Further, No. 1, 2, 3 and No. 2 of 2-port SRAM. Each set of Nos. 5, 6, 7 and Nos. 9, 10, 11 and Nos. 13, 14, 15 is also stored in the same manner.
[0082]
Returning to FIG. 8 again, data for 1 UL is stored in order in one pixel at a time in four 2-port SRAMs arranged in the vertical direction. The first UL data is No. It is written in 0, 4, 8, 12 (line 4N in FIG. 8). The next UL data is No. 1, 5, 9, and 13 (4N + 1 line in FIG. 8). The buffer form of FIG. 8 has a configuration in which 16 points around arbitrary coordinates in the buffer can be simultaneously extracted by one access regardless of how the interpolation position is moved by the distortion correction processing.
[0083]
The data transmission availability determination circuit 30 receives a request (REQ_N) from the subsequent circuit, and outputs a request acceptance (GRANT_N) if the next UL data can be sent. This request reception (GRANT_N) becomes a trigger for itself, and the interpolation position calculation circuit 22 starts its operation. When it operates for 1 UL, the next UL head coordinate is calculated and ends.
[0084]
The data transmission availability determination circuit 30 sends an operation trigger (trig) to the distortion correction coefficient calculation circuit 21 so that the interpolation position calculation circuit 22 starts operating and outputs the interpolation position (X1, Y1). Similarly to the interpolation position calculation circuit 22, the distortion correction coefficient calculation circuit 21 calculates the next UL head coordinates after the operation for one UL, and ends.
[0085]
The read address generation circuit 25 issues a read address to each of the 16 2-port SRAMs 26 from the input interpolation coordinates.
[0086]
Next, reading and interpolation methods will be described.
An interpolation method is shown in FIG. FIG. 7 is an image diagram of the interpolation calculation in the interpolation circuit 27. The coordinates (X ′, Y ′) of the correction coordinate position P have already been obtained from the above [Equation 1]. The pixel values (luminance data) at these coordinates are obtained from the pixel data D0 to D15 at 16 points around the coordinates P (X ', Y'). If it is known from which of the 16 memories D0 comes, D1, D2,..., D15 can be known from the positional relationship with respect to D0. As will be described later, D0 is obtained from the coordinates of the interpolation position.
[0087]
FIG. 10 shows an example of reading from a buffer composed of 16 2-port SRAMs. Now, the interpolation position (X ′, Y ′) is X ′ = 10. ..., Y '= 50. ... (... is a number after the decimal point), X '= 10. ... is equal to or greater than (4 × 0 + 10), so the interpolation position comes slightly to the right of the 4N + 10th UL of the 2-port SRAM in the horizontal direction. Y '= 50. ... is (4 × 12 + 2) or more, and the memory (No. 10) in which the 50th pixel is stored in the 4N + 10th UL outputs the pixel D5 in FIG. No. in the upper left 5
[0088]
Since the pixel data corresponding to D0 to D15 in FIG. 7 are the circles in FIG. 10, addresses are generated so that they are output.
[0089]
No. The image data output from 0 does not correspond to D0. In the example of FIG. The output from 5 corresponds to D0. For this reason, in order to identify which data is output from which memory, a data string control signal is output from the read address generation circuit 25, whereby the interpolation circuit 27 for performing the interpolation operation has a D 0 from where in the 2-port SRAM 26. It recognizes whether it is coming out and performs 16-point interpolation.
[0090]
If the pixel data D0 to D15 are known, the pixel data at the corrected coordinate position can be obtained as Dout by performing an interpolation process according to the interpolation formula of [Formula 2].
[0091]
[Formula 2]

After the last UL data is output, the buffer release amount calculation circuit 31 calculates the difference between the UL start coordinate that has just been processed and the next UL start coordinate (see FIG. 17), and data that is no longer needed is accumulated. In order to release the buffer, the buffer release amount is output to the buffer free space monitoring circuit 29. However, it is preferable that the buffer release amount when straddling the strain center is opened in consideration of changes in values of postULB1 and postULB2, as shown in FIG. The preULB and postULB shown in FIG. 16A are obtained by providing areas of a predetermined width on the front side and the rear side in the row direction with respect to the coordinate position of the first pixel among the pixels forming the UL. These are defined as preULB and postULB, respectively. When straddling the distortion center, the buffer opening amount is not the difference between the normal UL start coordinates, but the buffer opening amount is referred to by the change amount of postULB (ULB: an abbreviation of Unit Line Buffer) where the reference value changes. (Adjustment amount is postULB1-postULB2, see FIG. 16 (b)).
[0092]
When (UL head coordinate difference) + (adjustment amount) is negative, a negative value is stored. The buffer is not released until the processing advances and the buffer release amount exceeds a negative value.
[0093]
Along with the above calculation of the buffer release amount, the buffer release amount calculation circuit 31 transmits to the data transmission availability determination circuit 30 how much data is required from the previous stage circuit for the next UL processing.
[0094]
The buffer free capacity monitoring circuit 29 makes a request to the preceding circuit when the buffer is freed by the above calculation of the buffer opening amount.
[0095]
The data transmission availability determination circuit 30 determines whether or not the next UL data can be sent based on the internal counter (BLC) of the write address generation circuit 28, the human power from the buffer opening amount calculation circuit 31, and the preULB value. The data transmission availability determination circuit 30 returns a request acceptance (GRANT_N) in response to a request from the subsequent circuit.
[0096]
In the error detection circuit 32, the coordinates input to the read address generation circuit 25 deviate from the left end (see FIG. 11) or right end of the block line (BL), or the upper and lower ends (see FIG. 12) of the block line (BL). An error is output when the value deviates or the distortion amount deviates from the set values of preULB and postULB (see FIG. 13). The output of these errors is performed by determining the coordinates. In the case of FIGS. 11 and 12, an error is output, but the process continues. In the case of FIG. 13 as well, an error is output, but the process continues. In the case of preULB and postULB in FIG. 13, not only an error is output, but also the amount of deviation is stored and set in a register so that the CPU 4 can acquire it as data.
[0097]
As described above, in the block line processing, when the input side image is interpolated and calculated with respect to the output side image, there is a case where the image is projected from the input image range. In this case, interpolation data is generated in a portion where there is no data in the input range, and an error indicating that interpolation cannot be performed is output from the error detection circuit 32.
[0098]
As shown in FIG. 14, the image distortion is contrasted with respect to the center coordinates of the distortion. Further, the distortion is generally smaller as it is closer to the center and larger as it is farther away. Therefore, if the buffer is secured in consideration of the deformation of the UL at both ends of the BL, there is usually no data shortage in the interpolation processing in all UL. Therefore, a predetermined area preULB is placed forward (right side in the figure) and rearward (left side in the figure) with respect to the first coordinate to be interpolated (the top coordinate indicated by the top X of the image data in the block line BL in the figure). Is provided with a predetermined area postULB.
[0099]
The support function corresponds to a part called “distortion correction range calculation unit” in Japanese Patent Application No. 2003-202493 filed on the same day as the present application. In FIG. 27 (a), the region of interest is BL, but is transformed as shown in FIG. 27 (b) by coordinate conversion for distortion correction. At this time, preULB1 / postULB1 is determined from X'TL, Y'TL, XLmin, and XLmax (see FIG. 16A). Similarly, preULB2 / postULB2 is determined from X'TR, Y'TR, XRmin, and XRmax.
[0100]
Further, in the UL processing, when the strain passes through the center of the distortion, the bending direction of the optical distortion is reversed. Then, as shown in FIG. 14, the values of preULB and postULB are different on the left and right of the distortion center. If the largest postULB is adopted on the right side and the largest preULB is adopted on the left side as shown in FIG. 15 so that this value can be applied to both the left and right sides, there is no need for processing on one side of the left side or the right side. On the other hand, the data on the other side must be kept large (that is, the buffer is wide), which is a waste of the buffer.
[0101]
Therefore, instead of pre-determining both preULB and postULB with the same large value in advance, preULB and postULB are respectively set as variables. After passing through the center coordinates of the distortion, change these values to change the amount secured. That is, by changing preULB and postULB before and after the distortion center, use of the internal buffer is eliminated, and relatively large distortion correction can be performed with a small buffer.
[0102]
Although pre and postULB are set in the register 85, in order to generate these values, a support function that enables calculation of the input range of image data in consideration of distortion deformation is required. This support function may be provided by being added to the distortion correction processing function of the distortion correction processing unit 8. The memory control unit 83 performs control so that the areas preULB and postULB are not overwritten by another process during the 1UL process. The areas preULB and postULB may be set from the CPU 4 to the register 85 (see FIG. 2), or may be automatically calculated and set by the CPU.
[0103]
As described above, 16 points are interpolated in the distortion correction process, but as shown in FIG. 16A, the interpolation position can move by a (where a> 0) during the 1 UL process by the distortion correction. Because of the 16-point interpolation, the input image range needs to be a range obtained by adding intervals bL and bR for pixels necessary for interpolation to the interval a on both the pre side and the post side.
[0104]
When the distortion center is exceeded, the buffer is narrowed to the minimum necessary by changing the values of pre and postULB. Further, the above-described “distortion correction calculation unit” may output the result in consideration of the interpolation.
[0105]
In the case of FIG. 8, the internal buffer 82 included in the distortion correction processing unit 8 has 16 lines (pixels) in the horizontal direction of 96 pixels at the maximum. This will be used for distortion correction. The block line processing data is swept in the right direction in the figure by at least 1 UL. Since several UL may be released at once, it is not always necessary to sweep 1 UL at a time. Since the internal buffer 82 starts from an empty state at first, it enters 16 lines and performs distortion correction processing. When these unit lines are processed, unnecessary data (1 to several UL data) is generated on the left line. Note that the processing is not necessarily unnecessary, and when the enlargement ratio is large, it may take several UL until it is unnecessary. For unnecessary data, the buffer is released (that is, overwriting is permitted) and new data is inserted. The image you are sweeping shifts to the right. Since there is only a size that can read a maximum of 16 lines, the data area that is no longer needed is released and new data is overwritten sequentially. All unnecessary buffer areas are released at once. The opening amount may be 1 line or 5 lines. For example, since the first three lines of data stored in the buffer are unnecessary for the subsequent processing, the next data is overwritten by releasing it and accepting the next data.
[0106]
By the way, the amount of the buffer that can be released after the end of 1UL processing is not known unless the next UL head coordinate is determined (see FIG. 17). Since the internal buffer is small, in order to release the part where the data that is no longer used is released as soon as possible, the next UL head coordinate is calculated to obtain the release amount, and the internal buffer is made effective by releasing the buffer. Make it available. However, a request for distortion correction comes from a circuit block at the rear stage, a grant is returned, 1 UL data is output to the rear stage, and a grant is returned after the next request comes, and then the coordinates are obtained. If the operation is performed as described above, the release of the buffer is delayed, and accordingly, the fetching of new data is also delayed. As a result, a long space may be created in the pipeline operation. Therefore, as shown in FIG. 18, at the stage where the previous distortion correction processing (processing of the coordinates 1 to n shown in the figure) is being performed, regardless of the request from the subsequent circuit block, the processing of the next coordinate n is completed. The coordinate n + 1 is calculated to obtain the release amount, and the buffer is released early. As a result, new data is input from the preceding circuit block relatively quickly, and the pipeline space can be reduced. However, in the case of FIG. 18, after processing the coordinate n, the processing is terminated after calculating the coordinates of the pixel n + 1, so that the data necessary for the next UL processing is stored in the buffer as shown in FIG. Even if the requests are complete, if the requests from the subsequent stage continue, the pipeline becomes empty. Since there is still a pipeline vacancy, there is a possibility that the operation speed of the entire apparatus will be adversely affected when requests continue.
[0107]
Therefore, as an improvement measure of FIG. 18, there is a method of obtaining the buffer opening amount by determining the next coordinates as soon as possible as shown in FIG. As shown in FIG. 18, the next UL head coordinate is acquired at a stage earlier than the last head coordinate obtained at the end, and if it is determined that the necessary data is available, the next UL processing is continued. By doing so, the processing space is reduced. That is, before the end of 1UL processing, the UL data is held in a register or the like, and the opening position of the next UL processing is obtained in advance by calculating the top coordinate position of the next UL. In FIG. 20, the leading coordinates of the next UL are obtained for the second pixel of the previous distortion correction processing (processing of the coordinates 1 to n shown in the figure), and the buffer opening amount is acquired at an early stage. As a result, the buffer can be released quickly, and the pipeline space can be considerably reduced as compared with FIG. 19, as shown in FIG. However, there is a problem that the circuit becomes complicated and control becomes difficult.
[0108]
Therefore, FIG. 22 is a further improvement of FIG. In FIG. 22, the amount to be released when the currently processed UL is over (open amount 1) is already known during the previous UL processing. After calculating the coordinate n, the second UL head coordinate is calculated, and the amount that can be released when the next UL processing is completed (open amount 2) is obtained in advance. In this way, it is not necessary to perform exceptional coordinate generation (coordinate 2 generation) during UL processing as in the case of FIG. Thus, if the start coordinates of the next two next ULs are obtained instead of the start of the next UL, the exceptional processing as shown in FIG. .
[0109]
【The invention's effect】
As described above, according to the present invention, distortion correction can be realized without greatly increasing the transfer amount of the bus or the capacity of the memory. That is, it is possible to perform a relatively large distortion correction process with a small buffer capacity.
[Brief description of the drawings]
FIG. 1 is a block diagram showing the overall configuration of an image processing apparatus according to an embodiment of the present invention.
FIG. 2 is a block diagram showing a configuration of a distortion correction processing unit.
FIG. 3 is a conceptual diagram of coordinate conversion in a distortion correction processing unit.
FIG. 4 is a diagram showing a Jinkasa type strain.
FIG. 5 is a diagram showing input image data.
FIG. 6 is a block diagram showing a detailed configuration of a distortion correction processing unit.
FIG. 7 is an image diagram of interpolation calculation in an interpolation circuit.
FIG. 8 is a diagram illustrating an internal memory unit in a distortion correction processing unit.
FIG. 9 is a diagram showing a writing order for supplementary explanation of FIG. 8;
10 is an explanatory diagram for obtaining D0 necessary for calculating the corrected coordinate position of FIG. 7; FIG.
FIG. 11 is a diagram showing an example of error processing.
FIG. 12 is a diagram showing another example of error processing.
FIG. 13 is a diagram showing another example of error processing.
FIG. 14 is a diagram for explaining a buffer amount required for distortion correction processing;
FIG. 15 is a diagram for explaining a buffer amount required for distortion correction processing;
FIG. 16 is a diagram for explaining a buffer amount required for distortion correction processing;
FIG. 17 is a diagram for explaining a method for calculating an opening amount associated with UL processing.
FIG. 18 is a diagram for explaining a method for calculating an opening amount associated with UL processing.
FIG. 19 is a diagram for explaining a vacancy in pipeline processing accompanying the processing in FIG. 18;
FIG. 20 is a diagram for explaining a method for calculating an opening amount associated with UL processing.
FIG. 21 is a diagram for explaining a vacancy in pipeline processing associated with the processing in FIG. 20;
FIG. 22 is a diagram for explaining a method for calculating an opening amount associated with UL processing.
FIG. 23 is a diagram showing a lattice-shaped subject, a captured image with barrel distortion, and a captured image with pincushion distortion.
FIG. 24 is a diagram showing a concept of a general digital camera image processing procedure and a block configuration of a conventional digital camera image processing apparatus.
FIG. 25 is a view for explaining the reading order of image data from the frame memory according to the embodiment of the present invention.
26 is a block diagram showing the configuration of the first and second data order conversion units in FIG. 1 according to the present invention.
FIG. 27 is a diagram for explaining an operation for calculating an input image range when performing distortion correction processing in a distortion correction range calculation unit;
[Explanation of symbols]
1 ... CCD
3 ... Bus
4 ... CPU
5 ... Frame memory
6 ... 1st data order conversion part
7. Image process circuit
6 ... 1st data order conversion part
8: Distortion correction processing unit (distortion correction means)
9: Second data order conversion unit
10 ... JPEG processing unit
11 ... Memory card (recording medium)
82 ... Internal memory section (buffer)
83. Memory control unit
85: Control register

Claims (12)

In an image processing apparatus having a distortion correction unit that is connected to an image processing unit so as to be pipeline- processed and that performs distortion correction processing on image data after image processing.
The distortion correction means includes
A coordinate position of a pixel in the corrected image after distortion correction, and an interpolation coordinate generation unit that generates an interpolation coordinate of the pixel in the image before distortion correction corresponding to the coordinate position;
A buffer memory unit for storing the image data in units of blocks ;
A memory control unit that controls writing and reading of image data to and from the buffer memory unit;
Based on the generated interpolation coordinates, an interpolation calculation unit that performs an interpolation calculation to generate pixel data of the interpolation coordinates from the block-unit image data read from the buffer memory unit ;
Have
The memory control unit
When the image data before distortion correction is written from the image processing means to the buffer memory unit, it is sequentially written in the row direction in units of unit lines, which are image data composed of a fixed number of pixels arranged in a line in the column direction. When the image data stored in the buffer memory unit is read out from the buffer memory unit, the image data after distortion correction processing is sequentially output in the row direction with the unit line as a unit. An image processing apparatus that performs read control . The memory control unit provides a memory area having a preULB width between the reference coordinate position and the front coordinate position in the row direction, with the coordinate position of the pixel processed first in the unit line as a reference coordinate position. And a memory area having a width of postULB between the rear coordinate position in the row direction,
The image processing apparatus according to claim 1, wherein during the processing of the unit line, control is performed so that both the memory areas are not overwritten by data by other processing. The memory control unit further includes:
A buffer opening amount calculation circuit that calculates a buffer opening amount that can be opened to input data from a preceding circuit of the distortion correction unit during the processing of the unit line;
A data transmission availability determination circuit that determines whether data transmission is possible with respect to a subsequent circuit of the distortion correction unit;
The buffer opening amount calculation circuit calculates the difference between the coordinates for the first pixel of the unit line currently being processed and the coordinates for the first pixel of the unit line to be processed next as the buffer opening amount, and Send the data amount from the previous circuit required for processing the unit line to the data transmission availability determination circuit,
3. The data transmission availability determination circuit according to claim 1, wherein the data transmission availability determination circuit determines whether data transmission is possible based on a data amount transmitted from the buffer opening amount calculation circuit and a value of the preULB . Image processing device. The image processing apparatus according to claim 2, wherein the preULB value and the postULB value can be set to different values on the front side and the rear side, respectively, with respect to the distortion center . The image processing apparatus further includes a buffer free capacity monitoring circuit that detects a free area in the buffer based on the calculated buffer opening amount,
4. The image processing according to claim 1, wherein when a free area in the buffer is detected by the buffer free capacity monitoring circuit, data can be written to the free area. apparatus. The buffer memory unit includes a plurality of buffers capable of simultaneously reading and writing data,
The memory control unit further includes:
A write address generation circuit for performing data write control to the buffer memory unit;
A read address generation circuit for generating an address for simultaneously reading data necessary for the interpolation calculation from the image data stored in the buffer memory unit;
Have
The image processing apparatus according to claim 1, wherein the data writing control is to write simultaneously read data to different buffers . The image processing apparatus further includes coordinates that deviate from the preULB and the postULB in the column direction with respect to the interpolation coordinates for the image data of the block line for which the interpolation coordinates obtained by the distortion correction processing perform the unit line processing. 5. An error detection circuit that outputs an error when generated or when an interpolated coordinate obtained by distortion correction processing becomes a coordinate that cannot be input from the previous stage. the image processing apparatus according to one or. The image processing apparatus according to claim 7, wherein when the error is output, the distortion correction processing can be re-executed by resetting an input range of the image data of the block line . After the distortion correction processing for the last pixel of one unit line is completed, the interpolation coordinates of the pixel for which distortion correction processing is performed at the beginning of the next one unit line or the next one unit line are calculated, and the interpolation coordinates are calculated. The image processing apparatus according to claim 1, wherein the buffer opening amount is obtained based on the image processing apparatus. In an image processing method for performing distortion correction processing on input image data,
When performing the distortion correction process,
The input image data is stored in block units in a buffer memory unit in which data writing and reading are controlled,
Generate the coordinate position of the pixel in the corrected image after distortion correction and the interpolation coordinate of the pixel in the image before distortion correction corresponding to this coordinate position,
Based on the generated interpolation coordinates, an interpolation calculation is performed to generate pixel data of the interpolation coordinates from the block-unit image data read from the buffer memory unit,
When writing to the buffer memory unit, control is performed so that unit lines, which are image data composed of a certain number of pixels arranged in a line in the column direction, are sequentially written in the row direction as units. When reading, the image data stored in the buffer memory unit is subjected to read control so that the image data after distortion correction processing is sequentially output in the row direction with the unit line as a unit. Image processing method. The image processing apparatus further includes a distortion correction range calculation unit that calculates an input image range necessary for the distortion correction unit to perform distortion correction processing,
The input image range is obtained as a region including a corresponding region of image data before distortion correction processing obtained by performing coordinate correction for distortion correction on a region of interest in the image data after distortion correction processing. The image processing apparatus according to any one of claims 2 to 4. The distortion correction range calculation unit can perform coordinate conversion by applying a predetermined distortion correction formula to a region of interest in the image data after the distortion correction processing, and in the output image range after the distortion correction processing ,
The maximum and minimum values of the converted coordinates generated by performing the coordinate conversion on each pixel corresponding to each of the four surrounding sides, and the conversion generated by performing the coordinate conversion on each pixel corresponding to the four vertices Calculate at least one of the coordinates,
The image processing apparatus according to claim 11, wherein the preULB and the postULB are calculated based on a calculation result of the distortion correction range calculation unit .

Images