JP4155875B2 - Imaging device - Google Patents

Description

の式で求められる(ただし、Ｐ(ｉ,ｊ)は、画像中の(ｉ,ｊ)座標における画素値、ｗは画像の横幅ピクセル数、ｈは画像の縦幅ピクセル数)。
【００７２】
通常、画像の明度が低すぎる場合、あるいは極端に高すぎる場合でも、認識精度は低下する。したがって、上記算出した明度Ｌについて、
Ｌ0 ≦ Ｌ かつ Ｌ ≦ Ｌ1
ならば、明度の値は適切であるという条件式、および、下限値Ｌ0,上限値Ｌ1の値が保存条件情報の１つとなっている。
【００７３】
保存条件判断部１０２は、明度Ｌが保存条件情報を満たすか否か、すなわち、
Ｌ0 ≦ Ｌ かつ Ｌ ≦ Ｌ1
を満たすか否かを判定する。この条件に適していなければ、保存条件は満たさないとして終了する。このとき、満たさない原因がわかっており、さらに、撮影条件をどのように変更すれば保存条件に適するかも簡単に求められる。例えば、Ｌ ＜ Ｌ0 である場合、明るさが不足しているため、より明るい撮影状態にする必要がある。この状態を報知している例が図６, 図７, 図８である。利用者は、表示された報知情報により、現在の撮影環境が文字画像を撮影保存するためには暗い環境であるということを知り、室内を明るくするなどの改善を行う。再度、ステップＳ1003に進んだときに明るさに関する条件は満たされたならば、次はステップＳ1004に進むことになる。
【００７４】
ステップＳ1004では、対象の大きさが適切であるか否かの判定を行われ、大きさが適切である場合、ステップＳ1005へ進み、適切でない場合、撮像状態は認識に適さないと判定される。
【００７５】
一般に画像認識装置では、認識可能な対象の最小の大きさが存在する。文字認識装置の場合、認識可能な最小文字より小さい文字は、認識不可能、あるいは、認識精度が著しく低い。従って、認識に適した画像を保存するためには、あらかじめ、認識可能な認識対象の最小の大きさ ChMin を実験等で定めておき、認識対象の大きさが、ChMin 以上になるようにすることが必要条件である。
【００７６】
図１５は画像入力部１０１内の画像中の認識対象の大きさを計算する一例を示すフローチャートであり、図１５のフローチャートと図１６をもとに説明する。本処理は、ステップＳ1002にて求めた２値画像を利用する。
【００７７】
最初にステップＳ1101において、画像入力部１０１内の画像データ(２値画像)に対して、一般にラベリングと呼ばれる画素成分連結成分抽出手法により、連結した文字成分を抽出することができる。図１６(a)に、「プ」の文字についての連結した文字成分の外接枠を点線で囲んで示している。
【００７８】
次に、ステップＳ1102で、各連結成分の外接枠統合処理を行う。外接枠枠統合処理とは、全ての画像連結成分について外接枠を求め、異なる２つの外接枠が一部分でも重なっていた場合、２つの外接枠を統合する。この統合処理を、全ての外接枠が重ならなくなるまで続ける処理である。図１６(a)の場合、半濁点からなる連結成分と半濁点以外(「フ」の部分)の連結成分の２つの外接枠が存在する。さらにこの２つの外接枠は、図１６(a)のように一部重なっている。したがって、この２つの外接枠を統合するという処理が行われる。統合後の外接枠を図で示したものが、図１６(b)である。文字によっては、必ずしも統合後の外接枠が文字の外接枠にならない場合もある。その例を図１６(c)にて図示している。図１６(c)の「川」のような分離文字の場合には、文字連結成分が重ならず、図１６(c)のように２つの外接枠は統合されずに残ってしまう。ステップＳ1102の処理が終わった後の例を、図１６(d)に挙げている。図１６(d)では、「早川太郎」の文字について、最終の外接枠を点線で図示している。「早」、「太」は外接枠で囲まれているが、「川」は３つ、「郎」は２つの外接枠が統合できずに残っている。
【００７９】
次にステップＳ1103の処理にて、最終の統合枠の長い方の辺(図中の縦方向)の長さを求め、その平均の長さを求める。文字の大きさの平均を求めるのに、このように長辺を使うのは、上記、「川」といった分離文字のためであり、長辺を使うことにより、分離文字の影響なく、画像入力部１０１内の画像中の文字の平均の大きさ Ch を求めることが可能である。
【００８０】
この文字の平均の大きさ Ch が、あらかじめ実験等で求めた文字の大きさの閾値ChMin 以上であるか否かを判定すればよい。
【００８１】
撮像条件判断部１０２は、文字の大きさの平均 Ch が撮像条件情報を満たすか否か、すなわち、
ChMin ≦ Ch
を満たすか否かを判定する。この条件に適していなければ、撮像条件は満たさないとして終了する。このとき、認識対象が小さいという旨の報知が行われる。この状態を報知している例が図９, 図１０である。この報知をもとに利用者は、装置を被写体に近づけるなどして文字が大きく撮影されるようにするなどの改善を行う。
【００８２】
再度、図１４に示すステップＳ1004に進んだときに大きさに関する条件が満たされたならば、次はステップＳ1005に進む。認識可能な文字の大きさの最大値の判定も必要な場合も上記最小値と同様に判定することが可能である。
【００８３】
次に、ステップＳ1005, S1006, S1007 では、最適状態存在条件による判定を行い、ステップＳ1005ではあおりの有無の判定、ステップＳ1006では傾きの有無の判定、ステップＳ1007では手ぶれの有無の判定を行う。
【００８４】
ステップＳ1005では、画像にあおりがあるか否かの判定を行われ、あおりがないと判断された場合、ステップＳ1006へ進み、あおりがあった場合、撮像状態は認識に適さないと判定される。
【００８５】
画像のあおりの有無の判定に関しては、例えば、次のような方法により可能である。最初に、画像入力部１０１内の画像を、上下、左右等分に２分割し、均等に４分割する。４分割の例を図１７(a)に図示している。次に、４分割された各領域で、各領域における画像の鮮鋭度を求める。
【００８６】
画像の鮮鋭度の求め方は、どのような方法でもよいが、例えば、次のような方法により可能である。
【００８７】
白黒多値画像の座標(i,j)の点P(i,j) について、
DC(P(i,j)) ＝ max {abs(P(i,j)-P(i+a,j+b));a＝-1,0,1,b＝-1,0,1}
で、点P(i,j)の微分値 DC(P(i,j))を定める(abs( x )は、ｘの絶対値を表す)。
【００８８】
図１７(b)は、画像入力部１０１内の画像データ(白黒多値画像)のうち、点P(i,j)を中心とした縦横３画素からなる領域である。上記点P(i,j)の微分値DC(P(i,j))は、点P(i,j)の周りの８個のうち、点P(i,j)の画素値と最も差が大きい画素値を持つ座標を選び、その差の絶対値をとった値である。
【００８９】
あらかじめ、実験等により、点P(i,j)が鮮鋭な点であるか否かを判定するための閾値RDCTH を求めておく。
【００９０】
さらに、２値画像から、文字画素成分の個数 PBNUM を求める。そして、鮮鋭であると判定された画素数を、
{ P(i,j) ; DC(P(i,j) ＞ RDCTH ) }
で表して、以下の式で、鮮鋭であると判定された画素数を文字成分の個数PBNUMで割った値 PB を求める。
【００９１】
PB = { P(i,j) ; DC(P(i,j) ＞ RDCTH ) }／ PBNUM
PB は、鮮鋭であると判定された画素の文字成分における比率を表している。
【００９２】
４つの領域で求められた鮮鋭度のうち、最大の値であるものをThMax , 最小の値であるものをThMin とする。あおりがない状態であるとは、４つの領域の鮮鋭度に差がない、すなわち、ThMax , ThMin に差がないことであると言える。従って、あらかじめ、実験等で、適切な閾値ThDisを求めておき、
ThMax − ThMin ＜ ThDis
を満たすならば、４つの領域で鮮鋭度に差がなく、従って画像にあおりがないと判定する一方、満たさないならば、４つの領域で鮮鋭度に差があり、従って画像にあおりがあると判定することができる。
【００９３】
画像にあおりが存在する場合、利用者は、「あおりがあります。被写体との接写面を垂直にしてください」などと表示された報知情報(図示せず)により、画像入力部１０１が被写体に対して水平になるように調整して撮像を行う。その結果、再度、ステップＳ1005へ進んだときに画像のあおりに関する条件が満たされたならば、次はステップＳ1006へ進むことになる。
【００９４】
ステップＳ1006では、名刺４０１(図４に示す)の傾きを計測し、傾きが許容範囲内か否かが測定される。傾きがない(許容範囲内)と判断された場合、ステップＳ1007へ進む一方、傾きがあった場合、撮像状態は認識に適さないと判定される。
【００９５】
画像の傾きを測定する処理はさまざまな手法が考えられるが、例えば画像中の文字列を抽出し、その後、各文字列の傾きの平均を求めて、求めた平均値を画像の傾きの値とするといった手法で容易に実現できる。
【００９６】
最初に、文字列(行)抽出処理を行う。文字列の抽出に関してはさまざまな方法が存在するが、例えば、特許第３０５８４８９号「文字列抽出方法」に文書画像から文字列を抽出する方法が記載されており、この第１実施形態でも、この方法を用いることにより、文字列を抽出することが容易に実現できる。
【００９７】
次に、各文字列の傾きの計測を行う。文字列の傾き計測に関しても種々の方法が存在するが、例えば次のような方法で求められる。以下、図１８を参照して説明する。行抽出処理にて、行数と各行の外接枠の座標が抽出できる。Ｎ行抽出された(この実施形態では、Ｎ＝６)として、各i行(ｉ＝１〜Ｎ)の左上座標を{ＳＴＸ(i), ＳＴＹ(i)}とし、右下座標を{ＥＮＸ(i), ＥＮＹ(i)}とする。この各行を、Ｘ座標値が、(ＳＴＸ(i)+ＥＮＸ(i))／２ の箇所で、左右に分割する。ここで左の領域をＡ0, 右の領域をＡ1とする。Ａ0,Ａ1 のそれぞれの領域について、それぞれについてＹ方向の重心 Ｙ0,Ｙ1 を次のように求める。
【００９８】
【数２】

ただし、Ｍ０(y) は、Ａ0 領域内でＹ座標が y である直線上に存在する文字画素数である。また、重心Ｙ1 に関しても同様の式により求められる。
【００９９】
以上の重心Ｙ0,Ｙ1 を用いて、各i行(ｉ＝１〜Ｎ)の行の傾きθ(i)[度]を、
tanθ(i) ＝ (Ｙ1−Ｙ0)／{(ＥＮＸ(i)−ＳＴＸ(i))／２}
＝ ２×(Ｙ1−Ｙ0)／{(ＥＮＸ(i)−ＳＴＸ(i))
で表せるので、傾きθ(i)は、
θ(i) ＝ arctan{２×(Ｙ1−Ｙ0)／{(ＥＮＸ(i)−ＳＴＸ(i))}
により求められる。最後に、傾きθ(i)の平均値θを求め、その値を傾きとする。
【０１００】
以上、画像の傾きの計測方法に関して述べたが、その他さまざまな計測方法があり、上記の方法に制限されるものではない。
【０１０１】
次に、以上で求めた傾きが認識対象として問題ないか否かが判定される。
【０１０２】
ここでは、傾き許容範囲は、認識実験等からα[度]以内と定められたとする。
すなわち、
−α≦θ かつ θ≦α
ならば、傾きは許容範囲であるという条件が保存条件情報の一構成要素となっている。
【０１０３】
上記で求めた傾きθが上記条件を満たさないとき、すなわち、
θ＜−α または α＜θ
のとき、保存条件は満たさないとして終了する。このときの傾きに関する画像の状態の報知例を図１１の８０１に示しており、「文書が左に傾いています」と表示される。
【０１０４】
その他の報知例として、傾きに関する画像の状態と改善に関する情報の報知例を、図１２の８０２に示しており、「文書が左に傾いています。文書を水平にしてください」と表示される。
【０１０５】
画像に傾きがあった場合、利用者は、傾きを修正して撮像を行う。その結果、再度、ステップＳ1006へ進んだときにぶれに関する条件が満たされたならば、次はステップＳ1007へ進むことになる。
【０１０６】
ステップＳ1007では、画像入力部１０１内の画像に手ぶれなどよる装置のぶれの有無が判定される。手ぶれがないと判断された場合、撮影状態は認識に適していると判定される一方、手ぶれがあった場合、撮像状態は認識に適さないと判定される。
【０１０７】
手ぶれの判定のために、画像入力部１０１の画像メモリ５０４は、複数の画像を保存することが可能な例えば図１９に示す画像保存領域１５０１からなる。この画像保存領域１５０１は、カウンタ領域１５０２と、Ｋ個からなる複数の一時記憶領域１５０３とから構成されており、画像データは、カウンタ領域１５０２内のカウンタが指す値の番号の画像保存領域に保存されるものとする。
【０１０８】
この画像データの保存処理を図２０のフローチャートを用いて説明する。数値Ｋの設定に関しては、後述する。
【０１０９】
最初に、カウンタ領域１５０２を初期化(初期値は１)する(ステップＳ1601)。次に、撮像が行われる(ステップＳ1602)が、撮像された画像は、カウンタ領域１５０２内の数値が示す図１９の領域に一時記憶される(ステップＳ1603)。
【０１１０】
次に、カウンタ領域１５０２の値ｉが１つ増加し(ステップＳ1604)、ｉ≦ＫであればステップＳ1601へ戻り撮影、一時記憶処理を続ける一方、ｉ＞Ｋであれば判定処理へ進む(ステップＳ1605)。
【０１１１】
なお、以上の構成により、図１９に示す一時記憶領域１５０３の領域１,領域２,…,領域Ｋには、画像入力部１０１にて撮像された画像が連続して一時記憶されている。
【０１１２】
ステップＳ1606で、画像がぶれのない画像であるか否かを判定が行われる。判定は、Ｋ個の一時記憶領域に一時記憶された各画像の変化量を調べることにより可能である。最初に次式で表される Dmax を求める。
【０１１３】
【数３】

ただし、M(i)(i=1,…,K)は、領域iに一時記憶された画像を意味するものとする。
【０１１４】
dis(M(i), M(j)) は、画像 M(i)とM(j)の距離を表す。ここで、距離はどんな距離でも良く、例えば、次式で示されるようなユークリッド距離でよい。
【０１１５】
【数４】

ただし、X, Y は、横幅w、縦幅h の白黒多値画像であり、X(i,j)、Y(i,j)は、それぞれ、画像X, Y の(i,j)成分の値である。
【０１１６】
判定は、あらかじめ実験等でぶれがないと判断して定めた値Dth と上記の方法で算出した Dmax との比較により行われる。Dmax ＜ Dth ならば、ぶれの判定時点において一時記憶された画像の変化量は小さく、従ってぶれはないと判断できる。
【０１１７】
一方、Dmax ≧ Dth の場合、画像の変化量は大きく、従って、ぶれがあるため、撮像条件は満たさないとして終了する。なお、一時領域の個数Ｋの値に関しては、上記の判定方法に適した個数となるように、あらかじめ実験などで求めておけばよい。
【０１１８】
画像にぶれがあった場合、利用者は、表示された報知情報により、現在の画像撮像装置を静止させる必要があることを知り、手ぶれなどに注意することにより、装置を静止させて撮像を行う。その結果、再度、ステップＳ1007へ進んだときにぶれに関する条件が満たされたならば、撮像条件は成立していると判定される。
【０１１９】
(第２実施形態)
図２１はこの発明の第２実施形態の画像撮像装置のブロック図を示しており、この画像撮像装置は、画像認識手段を備え、画像認識に適している画像を認識するように構成している。
【０１２０】
図２１に示すように、この第２の実施形態の画像撮像装置は、画像入力手段としての画像入力部１０１と、撮像条件判断手段としての撮像条件判断部１０２と、報知手段としての報知部１０３と、制御手段としての制御部１０６と、画像認識手段としての画像認識部１０７とを備えている。
【０１２１】
図２２は上記第２実施形態の画像撮像装置の動作を示すフローチャートであり、以下、図２２のフローチャートにしたがって画像撮像装置の動作を説明する。
【０１２２】
最初に、ステップＳ1801において、画像入力部１０１により被写体(名刺４０１)の撮像が開始される。
【０１２３】
次に、ステップＳ1802において、画像入力部１０１内の画像データに対して、撮像条件判断部１０２が撮像条件を満たすか否かを判断する。
【０１２４】
次に、ステップＳ1803で、ステップＳ1802の結果により撮像条件判断部１０２により画像データが撮像条件を満たさないと判断した場合は、ステップＳ1804に進み、撮像条件を満たさない画像データの状態の情報、または、撮像状態を改善するための情報、または、その両方を、報知部１０３により報知する。その後、ステップＳ1806に進む。
【０１２５】
一方、ステップＳ1803で撮像条件判断部１０２により撮像条件を満たすと判断した場合は、ステップＳ1805に進み、画像認識部１０７により画像入力部１０１内の画像データの認識を行った後、ステップＳ1806に進む。
【０１２６】
ステップＳ1806において処理継続か否かすなわち終了操作が行われたか否かを判断して、終了の操作がない場合、ステップＳ1801に戻る一方、終了の操作があれば、処理を終了する。
【０１２７】
図２３は、撮像条件判断部１０２により撮像条件を満たすと判断され、画像認識部１０７により画像データが認識された例である。１９０１は認識対象枠であり、この認識対象枠１９０１内の文字列が画像認識部１０７により認識され、その結果が認識結果表示窓１９０２に表示されている。
【０１２８】
上記上記画像撮像装置によれば、認識に適した画像が得られるので、確実な画像認識が可能となる。
【０１２９】
なお、上記画像撮像装置は、認識指示を行う認識指示部を備え、利用者が画像認識部１０７に認識指示を与えるような構成にすることも容易に可能である。
【０１３０】
上記第２実施形態では、画像認識として文字認識機能を有する画像撮像装置について説明したが、文字認識に限らず、バーコード認識、指紋認識、顔画像認識、図面認識などの一般の画像認識機能を有する画像撮像装置にこの発明を適用してもよい。
【０１３１】
この発明の画像撮像装置は、上記第１,第２実施形態に限定されるものではなく、例えば、第１,第２実施形態を組み合わせて、画像の保存と画像認識とを行う画像撮像装置に適用してもよい。
【０１３２】
【発明の効果】
以上より明らかなように、この発明の画像撮像装置によれば、画像入力手段により被写体を撮像する毎に、撮像条件判断手段の判断結果が報知手段により利用者に報知されるため、利用者はその都度報知される報知情報に基づいて、容易に装置の位置や、向き等を調整することができる。従って、利用者にとって使い勝手のよい、画像認識に適した状態の画像を撮像する撮像装置を実現することができる。
【０１３３】
また、上記画像撮像装置によれば、報知手段は、撮像条件判断手段の複数の撮像条件を、利用者が設定を行う明るさ、認識対象の大きさといった利用者設定条件を、利用者が容易に最適な状態を想定できるあおり、傾き、手ぶれの有無といった最適状態存在条件に優先して報知することにより、利用者は、おおよその見当で設定し、不適な度合いが大きい可能性がある明るさ、認識対象の大きさといった利用者設定条件の適不適を最初に知ることができる。
【０１３４】
そのため、利用者設定条件が不適であるために、あおり、傾き、手ぶれの有無といった最適状態存在条件の調整が全く無駄になるといった可能性をなくすことができる。
【０１３５】
また、上記画像撮像装置によれば、画像認識に適した画像の保存が可能であり、保存した画像の画像認識手段による認識結果(例えば、文字認識結果のテキスト情報など)を、いつでも作成することが可能である。画像認識の際、画像を保存しているため、被写体がその場になくても、認識結果の誤りを保存した画像を見ることで、利用者が修正することが可能である。
【０１３６】
また、画像認識に適した状態のよい画像をそのまま画像認識手段に入力し、すぐに認識結果を利用することも可能である。
【図面の簡単な説明】
【図１】 図１はこの発明の第１実施形態の画像撮像装置のブロック図である。
【図２】 図２は上記画像撮像装置のフローチャートである。
【図３】 図３(a)はカメラ付き携帯端末型の画像撮像装置を示す正面図であり、図３(b)は上記画像撮像装置の裏面図である。
【図４】 図４は認識対象とする文書(名刺)を示す図である。
【図５】 図５は上記画像撮像装置の画像入力部の構成を示すブロック図である。
【図６】 図６は明るさに関する撮像状態を報知する例を示す図である。
【図７】 図７は明るさに関する撮像状態を改善する情報を報知する例を示す図である。
【図８】 図８は明るさに関する撮像状態と撮像状態を改善する情報の両方を報知する例を示す図である。
【図９】 図９は認識対象の大きさに関する撮像状態を報知する例を示す図である。
【図１０】 図１０は認識対象の大きさに関する撮像状態を改善する情報を報知する例を示す図である。
【図１１】 図１１は傾きに関する撮像状態を報知する例を示す図である。
【図１２】 図１２は傾きに関する撮像状態と傾きに関する撮像状態を改善する情報の両方を報知する例を示す図である。
【図１３】 図１３は撮像状態が画像認識に適していることを報知する例を示す図である。
【図１４】 図１４は上記第１実施形態の画像認識装置の画像の状態を判断する処理を示すフローチャートである。
【図１５】 図１５は認識対象の大きさを計測する処理を示すフローチャートである。
【図１６】 図１６(a)は文字連結成分とその外接枠の例を示す図であり、図１６(b)は外接枠を統合した例を示す図であり、図１６(c)は１文字が１つの外接枠とならない例を示す図であり、図１６(d)は外接枠統合終了の例を示す図である。
【図１７】 図１７(a)は鮮鋭度を調べるために画像を４分割した例を示す図であり、図１７(b)は鮮鋭度を調べるために画素ごとの微分値を求めるための例を示す図である。
【図１８】 図１８は文字列の傾き検出例を示す図である。
【図１９】 図１９は手ぶれの有無を検出するための画像入力部内のメモリの構成を示すブロック図である。
【図２０】 図２０は手ぶれの有無を検出する処理を示すフローチャートである。
【図２１】 図２１はこの発明の第２実施形態の画像撮像装置のブロック図である。
【図２２】 図２２は上記画像撮像装置のフローチャートである。
【図２３】 図２３は上記画像撮像装置の認識例を示す図である。
【符号の説明】
１０１…画像入力部、
１０２…撮像条件判断部、
１０３…報知部、
１０４…画像保存指示部、
１０５…画像保存部、
１０６…制御部、
１０７…画像認識部、
３０１…本体、
３０２…ディスプレイ、
３０３…カメラ、
３０４…撮影ボタン、
４０１…名刺、
５０１…撮像装置、
５０２…ＣＣＤ、
５０３…Ａ／Ｄコンバータ、
５０４…画像メモリ、
５０５…表示装置、
５０６…ＬＣＤ、
５０７…バックライト、
６０１…撮像状態(明るさに関する)の情報の報知例、
６０２…撮像状態(明るさに関する)を改善する情報の報知例、
６０３…撮像状態(明るさに関する)の情報と撮像状態を改善する情報の報知例、
７０１…撮像状態(大きさに関する)の情報の報知例、
７０２…撮像状態(大きさに関する)を改善する情報の報知例、
８０１…撮像状態(傾きに関する)の情報の報知例、
８０２…撮像状態(傾きに関する)の情報と撮像状態を改善する情報の報知例、
９０１…撮像状態(認識に適している状態)の報知例、
１５０１…画像保存領域、
１５０２…カウンタ領域、
１５０３…一時記憶領域、
１９０１…認識対象枠、
１９０２…認識結果表示窓。[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to an image pickup apparatus, and more particularly to an image pickup apparatus having an image recognition function such as a portable information terminal with a camera function or a mobile phone.
[0002]
[Prior art]
  Conventionally, OCR (optical character recognition) technology as image recognition has been used in various devices such as a document reading device, a business card reading device, and a form reading device as a technology for extracting character information from a document image. . As such OCR image input means, a scanner is generally used, but in recent years, due to advances in CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor) sensors, Digital cameras using these devices are used as OCR image input means.
[0003]
  As such an image pickup device, a device that can capture an image after correcting the tilt by displaying the tilt correction information on the display surface so that the imaging plane of the camera is parallel to the shooting target. Yes (see, for example, Patent Document 1). Here, “aori” is a term representing a state in which the subject to be imaged and the imaging surface are not parallel.
[0004]
[Patent Document 1]
          JP-A-8-186757
[0005]
[Problems to be solved by the invention]
  By the way, the image capturing apparatus can adjust the photographer to be in the correct photographing state by notifying the photographer of the tilt state, but is not referred to regarding other photographing states. In order to capture an image suitable for recognition, there are various shooting conditions in addition to tilting. For example, there are conditions such as brightness, size of a recognition target, image inclination, presence / absence of camera shake, and the like. However, there may be a problem if the conditions regarding the plurality of image states are determined in an appropriate order and are not notified to the user. The problem will be described below.
[0006]
  In the present specification, “image inclination” is used to mean a state in which a photographing object and an imaging surface are parallel, but a horizontally written document image cannot be photographed horizontally, for example. In addition, “tilting” means that the subject to be imaged and the imaging surface are not parallel as described above. In this specification, “tilt of image” and “tilting” are used as different states. Yes.
[0007]
  First, the following can be said when the imaging conditions are considered.
[0008]
  When the user adjusts the state related to “brightness”, the state that is too dark or too bright is not suitable for recognition, and the user can assume that. However, since there is no optimal state related to brightness common to all recognition means, it is impossible for the user to adjust the brightness to be optimal for image recognition. It can be said that the user can only make adjustments with an approximate register.
[0009]
  Regarding the “recognition target size”, for example, there is no optimum value of the character size common to all the character recognition means for the character recognition means. Regarding character recognition devices, the size of recognizable characters is generally from a minimum number of pixels to a maximum number of pixels for each device. The user adjusts the size of the character imaging target with an approximate register by selecting the imaging range of the target to be imaged and adjusting the distance between the imaging device and the subject.
[0010]
  On the other hand, “presence of tilt”, “presence of tilt”, “presence of camera shake”
  ・ The subject and the object are horizontal (no tilt),
  ・ The tilt of the imaging target in the subject is 0 (no tilt).
  The imaging device is stationary (no camera shake)
There exists an optimal state. These states are common conditions for image recognition and can be easily assumed by the user.
[0011]
  That is, from this,
  -Adjustments related to brightness and the size of the recognition target have a lot of room depending on the user's registration. Therefore, the initial setting may be greatly deviated from the appropriate state. Further, when compared with “presence / absence of tilt”, “presence / absence of tilt”, and “presence / absence of camera shake”, it is difficult for the user to determine whether it is suitable for image recognition, and it is an imaging condition that is difficult to adjust.
[0012]
  -The “presence / absence of tilt”, “presence / absence of tilt”, and “presence / absence of camera shake” can be easily assumed by the user. Therefore, the imaging conditions are easy for the user to adjust.
I can say that.
[0013]
  Therefore, it is not possible to determine whether or not the user himself / herself has made adjustments suitable for image recognition regarding the “brightness” and “recognition target size” determined by the user with approximate registration. It can be said that it is a desirable configuration to notify the user by giving priority to informing whether or not.
[0014]
  In addition, since adjustments relating to “brightness” and “size of recognition target” are difficult for the user to assume an optimum state, there may be the following problems when the order of notification is followed.
[0015]
  For example, when the user adjusts the imaging state so that the subject is suitable for recognition based on the notification information with a single imaging start instruction by holding the imaging device over the subject, it is bright in the current shooting environment. If there is not enough light, there is a possibility that the shooting location must be changed because the “brightness” is insufficient. Also, if you are trying to shoot a document image with small characters written on it, if the user prioritizes shooting a large range of recognition target, no matter how the distance is adjusted, May not be appropriate. At this time, if the “presence / absence of tilt”, “presence / absence of tilt”, and “presence / absence of camera shake” are determined at first, even if the user performs the adjustment, it is completely useless.
[0016]
  SUMMARY OF THE INVENTION An object of the present invention is to provide an image pickup apparatus that allows a user to easily adjust a shooting state suitable for image recognition and easily obtain a good image recognizable image. It is in.
[0017]
[Means for Solving the Problems]
  In order to solve the above-described problem, an image capturing apparatus according to the present invention provides:
  Image input means for imaging a subject;
  An imaging condition determining unit comprising a plurality of imaging conditions for determining whether or not the state of the image captured by the image input unit is an image suitable for image recognition;
  Each time the subject is picked up by the image input means, when the picked-up image determination means determines that the state of the picked-up image is not suitable for image recognition, the predetermined priority order of the plurality of image pickup conditions is set. And an informing means for informing at least one of imaging state information or information for improving the imaging state based on,
  The plurality of imaging conditions included in the imaging condition determining means are:
  Optimal state existence conditions that allow the user to easily assume the optimal state,
  Including user setting conditions for setting with an approximate register because the above user cannot assume an optimal state,
  The notification means gives priority to notification regarding determination of the user setting condition over notification regarding determination of the optimum state existence condition.It is characterized by that.
[0018]
  According to the image pickup apparatus having the above-described configuration, the subject is picked up by the image input unit based on the image pickup conditions for picking up the subject by the image input unit and capturing an image suitable for image recognition. Whether or not the state of the captured image is an image suitable for image recognition is determined by a plurality of imaging condition determination means for each captured image, and determined not to be suitable for recognition. In such a case, the imaging state is notified each time. Therefore, every time the subject is imaged by the image input unit, the notification unit notifies the imaging state based on the determination result of the imaging condition determination unit, so that the user can easily shoot based on the notification information. Can be adjusted.
[0019]
  For example, the imaging device performs imaging and imaging status notifications sequentially at regular time intervals, so that the user can detect the position of the device based on the notification information that is notified each time (every regular time interval). Or the orientation can be adjusted. Accordingly, it is possible to realize an image pickup apparatus that is convenient for a user, can easily adjust a shooting state suitable for image recognition, and can easily obtain a good image recognizable image. be able to.
[0020]
  Here, the “image recognition” may be any of character recognition, barcode recognition, fingerprint recognition, face image recognition, drawing recognition, and the like, or two or more image recognitions may be combined.
  The order of notification of imaging status cannot be assumed by the user in terms of brightness, size of recognition target, etc., so the user setting conditions determined by the approximate register are the presence or absence of tilt, presence or absence of tilt, camera shake By giving priority over the optimum state existence condition that allows the user to easily assume the optimum condition such as the presence or absence of the user, it is possible to prioritize the suitability of the user setting condition determined by the approximate registration.
  In addition, when priority is given to notification of the optimal state existence condition, the user setting condition is inappropriate, so by changing the condition greatly, adjustments such as presence or absence of tilt, presence or absence of tilt, and presence or absence of camera shake have been made. Although there is a possibility of being wasted, priority is given to the notification of the user setting condition, so that the possibility of wasting can be eliminated.
[0021]
  Further, the image capturing apparatus according to the embodiment notifies the image capturing state when it is determined that the image captured by the image capturing condition determining unit is suitable for recognition.
[0022]
  Further, in the image capturing apparatus according to an embodiment, the notification unit includes information on an imaging state or information for improving an imaging state regarding an imaging condition having the highest priority among a plurality of imaging conditions included in the imaging condition determination unit. By informing at least one of the above, it is possible to clearly indicate the photographing condition to be corrected with the highest priority for the user.
[0023]
[0024]
[0025]
[0026]
  Further, in the image capturing apparatus of an embodiment, when the state of the image captured by the image input unit is determined to be suitable for image recognition in all of the plurality of image capturing conditions by the image capturing condition determining unit, the image capturing state Is notified every time an image is captured. Thereby, the image of the imaging state optimal for image recognition can be obtained.
[0027]
  In the image capturing apparatus according to an embodiment, the user setting conditions include conditions relating to image brightness and the size of a recognition target.
[0028]
  In the image capturing apparatus of an embodiment, the optimum state existence condition includes presence / absence of tilt, presence / absence of tilt, and presence / absence of camera shake.
[0029]
  The image capturing apparatus according to an embodiment includes an image storing unit for storing an image captured by the image input unit, and an image storing instruction unit for instructing an image to be stored in the image storing unit. Thus, an image suitable for recognition is stored. Because images suitable for recognition can be saved, the recognition results by the image recognition means can be used at any time after the image is saved, for example, the recognition result by the character recognition means of the saved business card image is used for the address book. Can do. Further, by saving the image, the user can correct the result of the image recognition at any time while viewing the saved image even when the subject is not on the spot.
[0030]
  In the image capturing apparatus of one embodiment, when the state of the image captured by the image input unit is determined to be suitable for image recognition by the image capturing condition determining unit, the image captured by the image input unit By recognizing image recognition means, an image suitable for recognition is recognized.
[0031]
DETAILED DESCRIPTION OF THE INVENTION
  DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image pickup apparatus of the present invention will be described in detail with reference to embodiments shown in the drawings.
[0032]
  (First embodiment)
  FIG. 1 is a block diagram showing the main functions of the image pickup apparatus according to the first embodiment of the present invention. As shown in FIG. 1, the image capturing apparatus according to the first embodiment includes an image input unit 101 as an image input unit, an imaging condition determination unit 102 as an imaging condition determination unit, and a notification unit 103 as a notification unit. , An image storage instruction unit 104 as an image storage instruction unit, an image storage unit 105 as an image storage unit, and a control unit 106 as a control unit.
[0033]
  FIG. 5 shows an imaging device 501 and a display device 505 as main configuration examples of the image input unit 101. As shown in FIG. 5, the imaging apparatus 501 includes a CCD 502, an A / D (analog / digital) converter 503, and an image memory 504. The CCD 502 converts the subject image into an analog signal, and the A / D converter 503 converts the analog signal from the CCD 502 into digital image data, and stores the converted digital image data in the image memory 504. Then, the digital image data stored in the image memory 504 is displayed on the display device 505. The display device 505 includes an LCD (liquid crystal display) 506 and a backlight 507.
[0034]
  The imaging condition determination unit 102 shown in FIG. 1 holds a plurality of information on imaging conditions for determining whether the image data in the image input unit 101 (more specifically, in the image memory 504 shown in FIG. 5) is suitable for image recognition. And determining whether or not the image data satisfies the imaging condition.
[0035]
  Here, the “imaging condition” is a condition such as an optimum state existence condition that allows the user to easily assume an optimum state, a user setting condition that is set by the user, and the like.
[0036]
  As described above, the “optimum state existence conditions” are conditions such as “presence / absence of tilt”, “presence / absence of tilt”, “presence / absence of camera shake”, and the like (states without tilt, tilt Is a condition that can be easily assumed by the user.
[0037]
  Similarly, “user setting conditions” are conditions related to “brightness”, “size of recognition target”, etc., and there is a large room for the user's registration, and it is determined whether the user is suitable for image recognition. This is an imaging condition that is difficult to adjust.
[0038]
  In addition, the notification unit 103 is a user for performing image capturing and appropriate image capturing based on a determination result under at least one of a plurality of image capturing conditions determined by the image capturing condition determining unit 102. This is a means for notifying an instruction or the like by display or voice.
[0039]
  The image storage instruction unit 104 is a unit for instructing the image storage unit 105 to store the image data in the image input unit 101, and specifically includes a button, a switch, and the like.
[0040]
  The image storage unit 105 is a means for storing image data in the image input unit 101 when a user instructs the image input instruction unit 104, and is a flash that can hold data without supplying power. This includes memory and magnetic disks.
[0041]
  The control unit 106 controls the image input unit 101, the imaging condition determination unit 102, the notification unit 103, the image storage instruction unit 104, and the image storage unit 105, thereby inputting an image by the image input unit 101. The determination by the imaging condition determination unit 102 whether the image state is an image suitable for image recognition, and the notification by the notification unit 103TheThe image storage instruction by the image storage instruction unit 104 of the user and the image storage by the image storage unit 105 are performed sequentially.TheDo.
[0042]
  FIGS. 3A and 3B show a camera-equipped mobile terminal type image pickup apparatus which is a specific configuration example of the first embodiment. FIG. 3A is a front view of the image pickup apparatus. FIG. 3B is a rear view of the image pickup apparatus.
[0043]
  As shown in FIGS. 3A and 3B, the image pickup apparatus includes a main body 301, a display 302, a camera 303, and a shooting button 304. The camera 303 shown in FIG. 3B corresponds to the image input unit 101 (shown in FIG. 1), and the shooting button 304 shown in FIG. 3A corresponds to the image storage instruction unit 104 (shown in FIG. 1).
[0044]
  The business card 401 in FIG. 4 is an example of a subject. In the first embodiment, the user is suitable for character recognition in order to use the business card 401, which is a recognition target document, for an address book after character recognition. Suppose you try to capture and save in state.
[0045]
  Next, the operation of the image pickup apparatus of the first embodiment will be described with reference to the flowchart of FIG.
[0046]
  First, by operating a power switch (not shown) and the like, in step S201 of FIG. 2, the image input unit 101 (camera 303 shown in FIG. 3B) starts imaging the subject (business card 401). Is done.
[0047]
  Next, state determination is performed in step S202. That is, for the image data in the image input unit 101, the imaging condition determination unit 102 determines whether or not the imaging condition is satisfied. Detailed description of the imaging conditions and the determination of the imaging condition determination unit 102 will be described later.
[0048]
  Next, in step S203, when the imaging condition determination unit 102 determines that the image data does not satisfy the imaging condition based on the result of the state determination in step S202, the process proceeds to step S204, and the state of the image data that does not satisfy the imaging condition is determined. Information, information for improving the imaging state, or both are notified by the notification unit 103.
[0049]
  For example, when the imaging condition determining unit 102 determines that the brightness is insufficient in step S202, information on the imaging state that “the brightness of the imaging environment is insufficient” as indicated by 601 in FIG. Is notified. At this time, “Please brighten the shooting environment” as shown in FIG. 7 is notified of information for improving the shooting state, or “Brightness of the shooting environment is insufficient.” "Please brighten the image" ", both the information on the imaging state and the information for improving the imaging state may be notified.
[0050]
  If it is determined that the size of the target is too small, information on the imaging state that “the character is too small” is notified as indicated by reference numeral 701 in FIG. At this time, as shown in FIG. 10, information that improves the imaging state may be notified, such as “Please move the device closer to the subject”. In addition, although not shown in the drawing, both the information on the imaging state and the information for improving the imaging state may be notified that “the character is too small. Please move the device closer to the subject”.
[0051]
  If it is determined that the image is tilted, information “the document is tilted to the left” is notified as indicated by reference numeral 801 in FIG. At this time, although not shown in the drawing, information for improving the imaging state may be notified as “Please keep the document level”. Further, as shown in FIG. 12, both the information on the imaging state and the information for improving the imaging state may be notified that “the document is tilted to the left. Please make the document horizontal”.
[0052]
  On the other hand, in step S203, when the imaging condition determination unit 102 determines that the image data satisfies the imaging condition based on the result of step S202, the process proceeds to step S205, and the state of the image data satisfying the imaging condition is notified. 103. For example, as in 901 in FIG. 13, information indicating that “the photographing state is suitable for recognition” is notified.
[0053]
  Next, in step S206, the process branches depending on whether or not the user has issued an image storage instruction by operating the image storage instruction unit 104 (whether or not the shooting button 304 in FIG. 3A has been pressed). If there is no save instruction, the process proceeds to step S208. On the other hand, if there is a storage instruction, the image data in the image input unit 101 is stored in the image storage unit 105 in step S207, and the process proceeds to step S208.
[0054]
  In step S208, it is determined whether or not the process is to be continued. That is, input of an image by the image input unit 101, determination of whether or not the state of the image by the imaging condition determination unit 102 is an image suitable for image recognition, determination of whether or not notification processing by the notification unit 103 continues. If there is no end operation, the process returns to step S201, and if there is an end operation, the process ends.
[0055]
  That is, in the first embodiment, during operation of the apparatus, it is determined whether the image input by the image input unit 101 and the image state by the imaging condition determination unit 102 are images suitable for image recognition. The notification process by the notification unit 103 is repeated. Based on the information about the state of the image notified by the notification unit 103, the user can change the imaging state such as “brightness”, “size of recognition target”, “aori”, “tilt”, “camera shake”, and the like. Adjust.
[0056]
  When the user adjusts the imaging state and the imaging condition determination unit 102 determines that the imaging state satisfies the imaging condition, the notification unit 103 notifies that the imaging is suitable. At this time, the user can store an image suitable for image recognition in the image storage unit 105 by operating the image storage instruction unit 104.
[0057]
  In step S205, even if the image capturing apparatus is configured not to notify when the photographing condition of step S205 is satisfied, the convenience of the apparatus is not reduced. This is because when the imaging condition determining unit 102 determines that the imaging state satisfies the imaging condition and the process of determining the input image proceeds to step S205, the notification unit 103 notifies that the input image is not suitable for image recognition. Not done. In this case, the user can determine that the input image is suitable for image recognition because there is no notification that the user is not suitable. For this reason, it is possible to configure the image capturing apparatus so as not to notify when the photographing condition is satisfied.
[0058]
  Hereinafter, details regarding the determination of the image state in step S202 will be described with reference to the flowchart of FIG. 14 and FIGS.
[0059]
  The imaging condition determination unit 102 determines user setting conditions such as “brightness” and “recognition target size” and optimum states such as “tilt”, “tilt”, and “camera shake” as conditions for determining the imaging state. Presence condition is retained.
[0060]
  In this way, notification of information on the imaging state of the image capturing apparatus (or information for improving the imaging state, or both) is notified in a predetermined priority order. Furthermore, the notification order is a notification regarding judgment of user setting conditions such as “brightness” and “size of recognition target”, and a notification regarding optimum condition existence conditions such as “aori”, “tilt”, and “shake”. Give priority.
[0061]
  Therefore, the user setting condition is first determined, and then the optimum state existence condition is determined, and the imaging state information (or information for improving the imaging state, or both) is determined for the condition that the imaging state is determined to be inappropriate. May be notified.
[0062]
  For all of the imaging conditions that are determined to be inappropriate for notification, information on the imaging state (or information that improves the imaging state, or both) may be notified. By notifying only one of the imaging state information (or information for improving the imaging state, or both), it is possible to clearly indicate the imaging condition to be corrected with the highest priority for the user.
[0063]
  In addition, if there is even one condition that is not appropriate for the determination processing of a plurality of imaging conditions, information on the imaging state (or information for improving the imaging state, or both) regarding the condition is notified, and from the conditions If the imaging conditions in the subsequent order are not performed on the captured image at the time of the imaging, the overall time taken to determine the imaging conditions can be reduced, and as a result, the responsiveness of the apparatus is improved. There is an advantage.
[0064]
  In addition, when the notification method is displayed in text, there is an advantage that the notification means becomes simple, for example, the display screen does not have to be enlarged.
[0065]
  Hereinafter, in the processing described, the imaging conditions are determined in descending order of notification priority, and if there is even one inappropriate condition, information on the imaging state (or information for improving the imaging state, or Both). An imaging condition having a priority after that condition is not performed on the captured image at the time of the imaging.
[0066]
  Hereinafter, description will be made step by step according to the flowchart shown in FIG.
[0067]
  First, in step S1001, when the image data in the image input unit 101 is a color image, in order to simplify the subsequent processing, a process of converting a color image into a monochrome multi-value image is performed (from the beginning, a monochrome multi-value image). This processing is not necessary for images). This conversion process may be any process. For example, when a color image is output in R (red), G (green), and B (blue), for each pixel.
      Pixel value after conversion = (R + G + B) / 3
It is possible to convert into a black and white multi-value image by converting using the equation (1). Hereinafter, the image data in the image input unit 101 is assumed to be a monochrome multi-value image.
[0068]
  Next, in step S1002, the image data in the image input unit 101 is binarized for use in subsequent state determination. The binarization method may be any method such as a discriminant analysis method called a so-called Otsu algorithm.
[0069]
  Next, in steps S1003 and S1004, determination based on user setting conditions is performed. In step S1003, the brightness of the captured image is determined. In step S1004, the size of the recognition target is determined.
[0070]
  In step S1003, the brightness (brightness) of the image data in the image input unit 101 is calculated, and it is determined whether the calculated numerical value is an appropriate numerical value. If the brightness is appropriate, the process proceeds to step S1004. If not appropriate, it is determined that the imaging state is not suitable for recognition.
[0071]
  The calculation of the brightness will be described for a monochrome multi-valued image by obtaining a monochrome multi-valued image in step S1001. The brightness is
[Expression 1]

Where P (i, j) is the pixel value at the (i, j) coordinate in the image, w is the number of horizontal pixels of the image, and h is the number of vertical pixels of the image.
[0072]
  Usually, even when the brightness of an image is too low or extremely high, the recognition accuracy is lowered. Therefore, for the calculated lightness L,
      L0 ≤ L and L ≤ L1
Then, the conditional expression that the value of brightness is appropriate, and the values of the lower limit value L0 and the upper limit value L1 are one of the storage condition information.
[0073]
  The storage condition determination unit 102 determines whether the lightness L satisfies the storage condition information, that is,
      L0 ≤ L and L ≤ L1
It is determined whether or not the above is satisfied. If it is not suitable for this condition, the storage condition is not satisfied and the process ends. At this time, the cause of not being satisfied is known, and furthermore, how to change the photographing condition to suit the storage condition can be easily obtained. For example, when L <L0, since the brightness is insufficient, it is necessary to obtain a brighter shooting state. Examples of notifying this state are FIGS. 6, 7, and 8. FIG. The user learns from the displayed notification information that the current shooting environment is a dark environment for shooting and storing character images, and makes improvements such as brightening the room. If the condition regarding the brightness is satisfied when the process proceeds to step S1003 again, the process proceeds to step S1004.
[0074]
  In step S1004, it is determined whether or not the size of the target is appropriate. If the size is appropriate, the process proceeds to step S1005. If the size is not appropriate, it is determined that the imaging state is not suitable for recognition.
[0075]
  In general, an image recognition apparatus has a minimum size of a recognizable object. In the case of a character recognition device, a character smaller than the smallest recognizable character is not recognizable or has a remarkably low recognition accuracy. Therefore, in order to save an image suitable for recognition, the minimum size of the recognition target that can be recognized, ChMin, should be determined in advance through experiments, etc., so that the size of the recognition target is greater than or equal to ChMin. Is a necessary condition.
[0076]
  FIG. 15 is a flowchart showing an example of calculating the size of the recognition target in the image in the image input unit 101, and will be described based on the flowchart of FIG. 15 and FIG. This process uses the binary image obtained in step S1002.
[0077]
  First, in step S1101, connected character components can be extracted from image data (binary image) in the image input unit 101 by a pixel component connected component extraction method generally called labeling. In FIG. 16 (a), the circumscribed frame of the connected character component for the character “P” is surrounded by a dotted line.
[0078]
  Next, in step S1102, circumscribed frame integration processing for each connected component is performed. In the circumscribing frame integration process, circumscribing frames are obtained for all image connected components, and if two different circumscribing frames overlap even partially, the two circumscribing frames are integrated. This integration process is a process that continues until all the circumscribed frames do not overlap. In the case of FIG. 16 (a), there are two circumscribed frames, that is, a connected component composed of a semi-turbid point and a connected component other than the semi-turbid point ("F" portion). Further, the two circumscribed frames partially overlap as shown in FIG. Therefore, a process of integrating the two circumscribed frames is performed. FIG. 16B shows a circumscribed frame after the integration. For some characters, the bounding frame after integration may not necessarily be the bounding frame for the character. An example of this is shown in FIG. In the case of a separated character such as “river” in FIG. 16 (c), the character connected components do not overlap, and the two circumscribed frames remain unintegrated as shown in FIG. 16 (c). An example after the processing of step S1102 is illustrated in FIG. In FIG. 16D, the final circumscribing frame of the character “Taro Hayakawa” is illustrated by a dotted line. Although “early” and “thick” are surrounded by circumscribing frames, three “rivers” and two “ro” remain unintegrated.
[0079]
  Next, in step S1103, the length of the longer side (vertical direction in the figure) of the final integrated frame is obtained, and the average length is obtained. The reason for using the long side in this way to calculate the average character size is because of the separation character such as “river” described above. By using the long side, the image input unit is not affected by the separation character. It is possible to obtain the average size Ch of the characters in the image in 101.
[0080]
  It may be determined whether or not the average character size Ch is equal to or larger than a character size threshold value ChMin obtained in advance through experiments or the like.
[0081]
  The imaging condition determination unit 102 determines whether or not the average character size Ch satisfies the imaging condition information, that is,
      ChMin ≤ Ch
It is determined whether or not the above is satisfied. If it is not suitable for this condition, it will be determined that the imaging condition is not satisfied. At this time, notification that the recognition target is small is performed. Examples of notifying this state are shown in FIGS. Based on this notification, the user makes improvements such as bringing the device closer to the subject so that characters are photographed larger.
[0082]
  If the size condition is satisfied again when the process proceeds to step S1004 shown in FIG. 14, the process proceeds to step S1005. When it is necessary to determine the maximum value of the recognizable character size, it can be determined in the same manner as the minimum value.
[0083]
  Next, in steps S1005, S1006, and S1007, a determination is made based on the optimum state existence condition. In step S1005, the presence / absence of a tilt is determined. In step S1006, the presence / absence of tilt is determined.
[0084]
  In step S1005, it is determined whether or not there is an error in the image. If it is determined that there is no error, the process proceeds to step S1006. If there is an error, it is determined that the imaging state is not suitable for recognition.
[0085]
  For example, the following method can be used to determine whether the image is tilted. First, the image in the image input unit 101 is divided into two parts, up and down and left and right, and equally divided into four parts. An example of four divisions is shown in FIG. Next, the sharpness of the image in each area is obtained from each of the four divided areas.
[0086]
  Any method may be used to obtain the sharpness of the image. For example, the following method may be used.
[0087]
  For the point P (i, j) of the coordinates (i, j) of the black and white multivalued image,
      DC (P (i, j)) = max (abs (P (i, j) -P (i + a, j + b)); a = -1,0,1, b = -1,0,1 }
Then, the differential value DC (P (i, j)) of the point P (i, j) is determined (abs (x) represents the absolute value of x).
[0088]
  FIG. 17B shows an area composed of three vertical and horizontal pixels centered on the point P (i, j) in the image data (monochrome multi-valued image) in the image input unit 101. The differential value DC (P (i, j)) of the point P (i, j) is the most different from the pixel value of the point P (i, j) among the eight around the point P (i, j). Is a value obtained by selecting a coordinate having a large pixel value and taking the absolute value of the difference.
[0089]
  A threshold value RDCTH for determining whether or not the point P (i, j) is a sharp point is obtained in advance by experiments or the like.
[0090]
  Furthermore, the number PBNUM of character pixel components is obtained from the binary image. And the number of pixels determined to be sharp,
      {P (i, j); DC (P (i, j)> RDCTH)}
The value PB obtained by dividing the number of pixels determined to be sharp by the number PBNUM of character components is obtained by the following equation.
[0091]
      PB = {P (i, j); DC (P (i, j)> RDCTH)} / PBNUM
PB represents a ratio of character components of pixels determined to be sharp.
[0092]
  Of the sharpness values obtained in the four areas, the maximum value is ThMax, and the minimum value is ThMin. It can be said that there is no difference between the sharpness of the four areas, that is, there is no difference between ThMax and ThMin. Therefore, an appropriate threshold value ThDis is obtained in advance through experiments, etc.
      ThMax − ThMin <ThDis
If satisfied, it is determined that there is no difference in sharpness in the four areas, and therefore there is no blur in the image, while if not satisfied, there is a difference in sharpness in the four areas, and therefore there is a shift in the image. Can be determined.
[0093]
  When there is a sway in the image, the user uses the notification information (not shown) such as “There is a sway. Please make the close-up surface with the subject vertical” to indicate that the image input unit 101 And adjust the image so that it is level. As a result, if the condition regarding the tilt of the image is satisfied when the process proceeds to step S1005 again, the process proceeds to step S1006.
[0094]
  In step S1006, the inclination of the business card 401 (shown in FIG. 4) is measured to determine whether the inclination is within an allowable range. If it is determined that there is no inclination (within an allowable range), the process proceeds to step S1007. On the other hand, if there is an inclination, it is determined that the imaging state is not suitable for recognition.
[0095]
  Various methods can be considered for measuring the inclination of the image.For example, a character string in the image is extracted, then the average of the inclination of each character string is obtained, and the obtained average value is used as the value of the inclination of the image. It can be easily realized by such a method.
[0096]
  First, a character string (row) extraction process is performed. There are various methods for extracting a character string. For example, Japanese Patent No. 3058889 “Character String Extraction Method” describes a method of extracting a character string from a document image. By using the method, it is possible to easily extract a character string.
[0097]
  Next, the inclination of each character string is measured. There are various methods for measuring the inclination of a character string. For example, the following method is used. Hereinafter, a description will be given with reference to FIG. In the line extraction process, the number of lines and the coordinates of the circumscribed frame of each line can be extracted. Assuming that N rows are extracted (N = 6 in this embodiment), the upper left coordinates of each i row (i = 1 to N) are {STX (i), STY (i)}, and the lower right coordinates are {ENX (i), ENY (i)}. Each row is divided into left and right portions at an X coordinate value of (STX (i) + ENX (i)) / 2. Here, the left area is A0 and the right area is A1. For each of the areas A0 and A1, the center of gravity Y0 and Y1 in the Y direction is obtained as follows.
[0098]
[Expression 2]

However, M0 (y) is the number of character pixels existing on a straight line whose Y coordinate is y in the A0 area. Also, the center of gravity Y1 can be obtained by the same equation.
[0099]
  Using the above centroids Y0 and Y1, the inclination θ (i) [degree] of the row of each i row (i = 1 to N) is
      tanθ (i) = (Y1-Y0) / {(ENX (i) -STX (i)) / 2}
               = 2 × (Y1−Y0) / {(ENX (i) −STX (i))
The slope θ (i) can be expressed as
      θ (i) = arctan {2 × (Y1−Y0) / {(ENX (i) −STX (i))}
Is required. Finally, an average value θ of the inclination θ (i) is obtained, and the value is set as the inclination.
[0100]
  The method for measuring the tilt of the image has been described above, but there are various other measuring methods, and the method is not limited to the above method.
[0101]
  Next, it is determined whether or not the inclination obtained above is a problem to be recognized.
[0102]
  Here, it is assumed that the allowable tilt range is set within α [degrees] from a recognition experiment or the like.
That is,
      −α ≦ θ and θ ≦ α
Then, the condition that the inclination is within an allowable range is a component of the storage condition information.
[0103]
  When the slope θ obtained above does not satisfy the above condition, that is,
      θ <−α or α <θ
In this case, the process is terminated assuming that the storage condition is not satisfied. A notification example of the image state relating to the tilt at this time is shown in 801 of FIG. 11, and “document is tilted to the left” is displayed.
[0104]
  As another notification example, a notification example of an image state related to tilt and information related to improvement is shown in 802 of FIG. 12, and “Document is tilted to the left. Please level the document” is displayed.
[0105]
  When the image has a tilt, the user corrects the tilt and takes an image. As a result, if the condition regarding blurring is satisfied again when the process proceeds to step S1006, the process proceeds to step S1007.
[0106]
  In step S1007, it is determined whether the image in the image input unit 101 is shaken by a device such as camera shake. When it is determined that there is no camera shake, it is determined that the shooting state is suitable for recognition. On the other hand, when there is a camera shake, it is determined that the imaging state is not suitable for recognition.
[0107]
  For determination of camera shake, the image memory 504 of the image input unit 101 includes, for example, an image storage area 1501 shown in FIG. 19 in which a plurality of images can be stored. The image storage area 1501 includes a counter area 1502 and a plurality of K temporary storage areas 1503, and the image data is stored in the image storage area having the number indicated by the counter in the counter area 1502. Shall be.
[0108]
  The image data saving process will be described with reference to the flowchart of FIG. The setting of the numerical value K will be described later.
[0109]
  First, the counter area 1502 is initialized (initial value is 1) (step S1601). Next, imaging is performed (step S1602), but the captured image is temporarily stored in the area of FIG. 19 indicated by the numerical value in the counter area 1502 (step S1603).
[0110]
  Next, the value i in the counter area 1502 is incremented by 1 (step S1604). If i ≦ K, the process returns to step S1601, and the shooting and temporary storage process is continued. S1605).
[0111]
  With the above configuration, images taken by the image input unit 101 are temporarily and continuously stored in the region 1, region 2,..., Region K of the temporary storage region 1503 shown in FIG.
[0112]
  In step S1606, it is determined whether or not the image is a blur-free image. The determination can be made by examining the amount of change of each image temporarily stored in the K temporary storage areas. First, obtain Dmax expressed by the following equation.
[0113]
[Equation 3]

However, M (i) (i = 1,..., K) means an image temporarily stored in the area i.
[0114]
  dis (M (i), M (j)) represents the distance between images M (i) and M (j). Here, the distance may be any distance, for example, a Euclidean distance represented by the following equation.
[0115]
[Expression 4]

However, X and Y are monochrome multi-valued images with horizontal width w and vertical width h, and X (i, j) and Y (i, j) are the (i, j) component of images X and Y, respectively. Value.
[0116]
  The determination is made by comparing the value Dth determined in advance by experimentation or the like and the value Dth calculated by the above method. If Dmax <Dth, the amount of change in the temporarily stored image at the time of blur determination is small, and therefore it can be determined that there is no blur.
[0117]
  On the other hand, if Dmax ≧ Dth, the amount of change in the image is large and, therefore, there is blurring, and the process ends with the imaging condition not being satisfied. It should be noted that the value of the number K of temporary areas may be obtained in advance by experiments or the like so as to be a number suitable for the above determination method.
[0118]
  When there is a shake in the image, the user knows that the current image pickup device needs to be stopped by the displayed notification information, and takes an image with the device stopped by paying attention to camera shake. . As a result, if the condition regarding blurring is satisfied again when the process proceeds to step S1007, it is determined that the imaging condition is satisfied.
[0119]
  (Second embodiment)
  FIG. 21 is a block diagram of an image pickup apparatus according to the second embodiment of the present invention. This image pickup apparatus includes image recognition means and is configured to recognize an image suitable for image recognition. .
[0120]
  As shown in FIG. 21, an image capturing apparatus according to the second embodiment includes an image input unit 101 as an image input unit, an imaging condition determination unit 102 as an imaging condition determination unit, and a notification unit 103 as a notification unit. And a control unit 106 as a control unit and an image recognition unit 107 as an image recognition unit.
[0121]
  FIG. 22 is a flowchart showing the operation of the image pickup apparatus of the second embodiment. Hereinafter, the operation of the image pickup apparatus will be described with reference to the flowchart of FIG.
[0122]
  First, in step S1801, imaging of the subject (business card 401) is started by the image input unit 101.
[0123]
  In step S1802, the image capturing condition determining unit 102 determines whether the image capturing condition is satisfied for the image data in the image input unit 101.
[0124]
  Next, in step S1803, when the imaging condition determination unit 102 determines that the image data does not satisfy the imaging condition based on the result of step S1802, the process proceeds to step S1804, or information on the state of the image data that does not satisfy the imaging condition, or Information for improving the imaging state or both are notified by the notification unit 103. Thereafter, the process proceeds to step S1806.
[0125]
  On the other hand, if the imaging condition determining unit 102 determines that the imaging condition is satisfied in step S1803, the process proceeds to step S1805, the image recognition unit 107 recognizes the image data in the image input unit 101, and then the process proceeds to step S1806. .
[0126]
  In step S1806, it is determined whether or not the process is continued, that is, whether or not an end operation has been performed. If there is no end operation, the process returns to step S1801, while if there is an end operation, the process ends.
[0127]
  FIG. 23 shows an example in which the imaging condition determining unit 102 determines that the imaging condition is satisfied, and the image recognition unit 107 recognizes the image data. Reference numeral 1901 denotes a recognition target frame. A character string in the recognition target frame 1901 is recognized by the image recognition unit 107, and the result is displayed in the recognition result display window 1902.
[0128]
  According to the above image pickup apparatus, an image suitable for recognition can be obtained, so that reliable image recognition is possible.
[0129]
  Note that the image capturing apparatus includes a recognition instruction unit that issues a recognition instruction, and can be easily configured so that the user gives a recognition instruction to the image recognition unit 107.
[0130]
  In the second embodiment, the image pickup apparatus having the character recognition function as the image recognition has been described. However, not only character recognition but also general image recognition functions such as barcode recognition, fingerprint recognition, face image recognition, and drawing recognition are provided. You may apply this invention to the image pick-up device which has.
[0131]
  The image pickup apparatus according to the present invention is not limited to the first and second embodiments described above. For example, the image pickup apparatus can be combined with the first and second embodiments to perform image storage and image recognition. You may apply.
[0132]
【The invention's effect】
  As is clear from the above, according to the image capturing apparatus of the present invention, every time a subject is imaged by the image input unit, the determination result of the imaging condition determination unit is notified to the user by the notification unit. The position, orientation, etc. of the device can be easily adjusted based on the notification information notified each time. Accordingly, it is possible to realize an imaging apparatus that captures an image that is convenient for the user and suitable for image recognition.
[0133]
  Further, according to the above-described image capturing apparatus, the notification unit allows the user to easily set the user setting conditions such as the brightness and the size of the recognition target set by the user for the plurality of image capturing conditions of the image capturing condition determining unit. It is possible to assume an optimal state for the camera, and by giving priority to the optimal state existence conditions such as tilt and presence / absence of camera shake, the user can set the brightness with an approximate register and the brightness that may be inappropriate. It is possible to first know whether or not the user setting conditions such as the size of the recognition target are appropriate.
[0134]
  For this reason, it is possible to eliminate the possibility that the adjustment of the optimum state existence conditions such as tilt, tilt, and presence / absence of camera shake is completely wasted because the user setting conditions are inappropriate.
[0135]
  Further, according to the above-described image pickup apparatus, it is possible to save an image suitable for image recognition, and to create a recognition result (for example, text information of a character recognition result) of the saved image by an image recognition unit at any time. Is possible. Since the image is stored at the time of image recognition, the user can correct the error by viewing the stored image of the recognition result even if the subject is not on the spot.
[0136]
  It is also possible to input an image in a state suitable for image recognition as it is to the image recognition means and use the recognition result immediately.
[Brief description of the drawings]
FIG. 1 is a block diagram of an image pickup apparatus according to a first embodiment of the present invention.
FIG. 2 is a flowchart of the image capturing apparatus.
FIG. 3A is a front view showing a portable terminal type image pickup apparatus with a camera, and FIG. 3B is a rear view of the image pickup apparatus.
FIG. 4 is a diagram illustrating a document (business card) to be recognized.
FIG. 5 is a block diagram illustrating a configuration of an image input unit of the image capturing apparatus.
FIG. 6 is a diagram illustrating an example in which an imaging state related to brightness is notified.
FIG. 7 is a diagram illustrating an example in which information for improving an imaging state related to brightness is notified.
FIG. 8 is a diagram illustrating an example in which both an imaging state relating to brightness and information for improving the imaging state are notified.
FIG. 9 is a diagram illustrating an example in which an imaging state relating to the size of a recognition target is notified.
FIG. 10 is a diagram illustrating an example of informing information for improving an imaging state related to the size of a recognition target.
FIG. 11 is a diagram illustrating an example of informing an imaging state related to tilt.
FIG. 12 is a diagram illustrating an example in which both an imaging state related to tilt and information for improving the imaging state related to tilt are notified.
FIG. 13 is a diagram illustrating an example of notifying that the imaging state is suitable for image recognition.
FIG. 14 is a flowchart illustrating a process of determining an image state of the image recognition apparatus according to the first embodiment.
FIG. 15 is a flowchart illustrating processing for measuring the size of a recognition target.
FIG. 16 (a) is a diagram showing an example of a character connected component and its circumscribing frame, FIG. 16 (b) is a diagram showing an example in which the circumscribing frame is integrated, and FIG. FIG. 16 is a diagram illustrating an example in which a character is not one circumscribed frame, and FIG. 16D is a diagram illustrating an example of the circumscribed frame integration end.
FIG. 17A is a diagram showing an example in which an image is divided into four parts for examining the sharpness, and FIG. 17B is an example for obtaining a differential value for each pixel for examining the sharpness. FIG.
FIG. 18 is a diagram illustrating an example of detecting the inclination of a character string.
FIG. 19 is a block diagram illustrating a configuration of a memory in an image input unit for detecting presence or absence of camera shake.
FIG. 20 is a flowchart illustrating a process for detecting the presence or absence of camera shake.
FIG. 21 is a block diagram of an image pickup apparatus according to a second embodiment of the present invention.
FIG. 22 is a flowchart of the image capturing apparatus.
FIG. 23 is a diagram illustrating a recognition example of the image capturing apparatus.
[Explanation of symbols]
  101: Image input unit,
  102 ... Imaging condition determination unit,
  103 ... notification section,
  104 ... Image saving instruction section,
  105: Image storage unit,
  106 ... control unit,
  107 ... an image recognition unit,
  301 ... main body,
  302 ... display,
  303 ... Camera,
  304 ... Shooting button,
  401 ... Business card,
  501 ... Imaging device,
  502 ... CCD,
  503 ... A / D converter,
  504: Image memory,
  505 ... Display device,
  506 ... LCD,
  507 ... Backlight,
  601 ... Notification example of information on imaging state (brightness),
  602 ... Information notification example for improving the imaging state (related to brightness),
  603 ... Notification example of information on imaging state (related to brightness) and information for improving imaging state,
  701 ... Notification example of information on imaging state (size),
  702 ... Notification example of information for improving the imaging state (related to size),
  801 ... Notification example of information on imaging state (inclination),
  802 ... Notification example of information on imaging state (related to tilt) and information for improving imaging state,
  901 ... Notification example of imaging state (state suitable for recognition),
  1501 ... Image storage area,
  1502 ... counter area,
  1503 ... Temporary storage area,
  1901 ... recognition target frame,
  1902: Recognition result display window.

Claims (7)

Image input means for imaging a subject;
An imaging condition determining unit comprising a plurality of imaging conditions for determining whether or not the state of the image captured by the image input unit is an image suitable for image recognition;
Each time the subject is picked up by the image input means, when the picked-up image determination means determines that the state of the picked-up image is not suitable for image recognition, the predetermined priority order of the plurality of image pickup conditions is set. based an informing means for informing at least one of information for improving the information or imaging state of the imaging condition,
The plurality of imaging conditions included in the imaging condition determining means are:
Optimal state existence conditions that allow the user to easily assume the optimal state,
Including user setting conditions for setting with an approximate register because the above user cannot assume an optimal state,
The image capturing apparatus according to claim 1, wherein the notification unit performs notification regarding determination of the user setting condition in preference to notification regarding determination of the optimum state existence condition . In the imaging device according to claim 1,
The informing means informs at least one of imaging state information or information for improving the imaging state for an imaging condition having the highest priority among a plurality of imaging conditions provided in the imaging condition determining unit. Imaging device. The image capturing apparatus according to claim 1 or 2 ,
When the imaging condition determination unit determines that the state of the image captured by the image input unit is suitable for image recognition under all of the plurality of imaging conditions, the imaging state is notified every time the image is captured. An image pickup apparatus. In the image pick-up device according to any one of claims 1 to 3 ,
The user setting condition includes at least one of a condition for determining whether the brightness of the image is appropriate and a condition for determining whether the size of the image recognition target is appropriate. An image pickup apparatus characterized by that. In the image pick-up device according to any one of claims 1 to 4 ,
The image pickup apparatus according to claim 1, wherein the optimum state existence condition includes at least one condition of presence / absence of tilt, presence / absence of tilt, and presence / absence of camera shake. In the image pick-up device according to any one of claims 1 to 5 ,
Image storage means for storing an image captured by the image input means;
An image storage instruction means for instructing an image to be stored in the image storage means,
An image capturing apparatus, wherein an image captured by the image input unit is stored in the image storage unit when a user instructs the image storage instructing unit. The image capturing apparatus according to any one of claims 1 to 6,
An image recognition means for recognizing an image picked up by the image input means when the state of the image picked up by the image input means is determined to be suitable for image recognition by the image pickup condition determination means. An image pickup apparatus.

Images