JP2004005384A - Image processing method, image processing device, program, recording medium, automatic trimming device and picture-taking arrangement - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To rapidly and accurately extract a face area by combining a face extracting process by flesh color and a simple discriminating process. <P>SOLUTION: The image processing device comprises an image input part 11 outputting an inputted color image as digital data, a flesh color area extracting part 12 extracting a flesh color area from the color image, a parietal region detecting part 13 detecting a position of a parietal region of a person from the color image and the flesh color area, a mouth detecting part 14 detecting a mouth of the person from the flesh color area and the position of the parietal region, an eye detecting part 15 detecting eyes of the person from the color image, the flesh color area and the position of the parietal region, and a jaw detecting part 16 calculating a jaw position of the person from positions of the eyes and mouth. In addition, it is composed of a center line detecting part 17 detecting a center line of a face of the person from the color image, and positions of the mouth and eyes, an area correcting part 18 correcting the face area on the basis of the jaw position and the facial center line, and a determining part 19 inputted with the color image, the flesh color area, the positions of the eyes and mouth and a corrected rectangular area, and determining whether or not the extracted flesh color area is a face of the person. <P>COPYRIGHT: (C)2004,JPO

Description

【００５６】
ここで、Ｒ、Ｇ、Ｂは、各画素の３原色の値を示す。また、抽出すべき肌色の範囲は、例えば、ｒ及びｇの各値に対する夫々下限値ｒｍｉｎ及びｇｍｉｎと、夫々上限値ｒｍａｘ及びｇｍａｘとを定めた下記式（２）により設定することができる。
【００５７】
【数２】

【００５８】
これにより、上記式（２）を満たす画素の集合として肌色領域が抽出される。但し、図２に示すように、抽出された肌色画素４１が、画像４０上において、例えば塊４２，４３等の複数の塊を構成している場合には、この画素の塊４２，４３毎に領域４０を分割し、各塊を異なる領域として区別する。画像４０の分割には、例えば、連結する画素の集合を１つの塊とし、この塊に対応する領域を抽出するラベリング処理等を使用することができる。
【００５９】
抽出された各肌色領域は、固有の番号が割り当てられ、２次元配列である領域マップＡ（ｘ，ｙ）上に多値画像として表現される。即ち、座標（ｘ，ｙ）の色が肌色領域ｎに含まれている場合、領域マップＡは、下記式（３）として示される。
【００６０】
【数３】

【００６１】
また、各肌色領域に対しては、それを取り囲む閉領域が算出される。閉領域としては、例えば、図３に示すように、肌色領域５１，５２を夫々取り囲む長方形領域５３，５４とすることができる。長方形領域は、図４に示すように、対角線上で相対する２つの頂点座標で示されるが、１つの肌色領域に対応する頂点座標｛（ｓｔｘ、ｓｔｙ），（ｅｄｘ、ｅｄｙ）｝は、例えば、次のように算出することができる。即ち、先ず、肌色領域に含まれる画素のｘ座標の平均値ｘａｖｅ、ｙ座標の平均値ｙａｖｅ、ｘ座標の標準偏差ｘｓｄｖ、及びｙ座標の標準偏差ｙｓｄｖを夫々下記式（４）により算出する。
【００６２】
【数４】

【００６３】
ここで、ｎは現在注目している肌色領域を識別する上記式（３）に示す番号である。これらの値を用いて、長方形領域の頂点座標は下記式（５）により算出することができる。
【００６４】
【数５】

【００６５】
ここで、ａは予め設定された係数である。肌色領域ｎに対して算出された長方形領域の頂点座標は、１次元配列である頂点リストＶに下記式（６）として格納される。
【００６６】
【数６】

【００６７】
肌色領域を表す領域マップＡは、口検出部１４及び判定部１９へ送られ、頂点リストＶは、頭頂部検出部１３、口検出部１４及び眼検出部１５へ送られる。
【００６８】
（２）特徴点検出
特徴点検出部では、肌色領域抽出部１２によって抽出された各肌色領域を顔領域と仮定し、この肌色領域に対応する頂点座標Ｖ（ｎ）が示す長方形領域に基づき、各特徴点が検出される。特徴点検出部は、人物の頭頂部の位置を検出する頭頂部検出部１３と、肌色領域内の赤みの強さに基づき、人物の口の位置を検出する口検出部１４と、頭頂部及び口の位置に基づき検索範囲を設定して眼を検出する眼検出部１５と、眼及び口の位置から顎の位置を算出する顎検出部１６と、口の位置から口領域を設定し、この口領域内の赤み強度に基づいて顔の中心線を検出する中心線検出部１７と、頭頂部、顎及び顔中心線の位置から、肌色抽出部１２にて算出された頂点座標Ｖ（ｎ）を修正する領域修正部１８とから構成される。以下、各検出部について更に詳細に説明する。
【００６９】
（２−１）人物の頭頂部の検出
頭頂部検出部１３は、肌色領域を顔として有する人物の頭頂部を検出する。頭頂部の検出は、例えば人物以外の背景領域は単一色であること及び人物の上方、即ち、垂直座標が小さい側には背景領域のみが存在し得ることを仮定し、背景色とは異なる色を有する画素の中で垂直座標が最も小さい位置を検出する。以下、頭頂部の位置における垂直方向の座標を頭頂部の高さという。
【００７０】
具体的には、図５に示すように、画像入力部１１から送られる入力カラー画像６０において、注目する肌色領域６１に対応する長方形領域６２の図中上方の領域、即ち、長方形領域６２よりも垂直座標が小さい領域であって、Ｖ（ｎ）．ｓｔｘ≦水平座標（ｘ座標）≦Ｖ（ｎ）．ｅｄｘの範囲に設定した頭頂部探索範囲６３を図中上方から走査し、各画素の値と背景領域６４の背景色との差ｄを下記式（７）によって算出する。
【００７１】
【数７】

【００７２】
ここで、Ｒ（ｘ，ｙ）、Ｇ（ｘ，ｙ）、Ｂ（ｘ，ｙ）はカラー画像上の座標（ｘ，ｙ）における画素のＲ、Ｇ、Ｂの値であり、Ｒｂｇ、Ｇｂｇ、Ｂｂｇは背景色のＲ、Ｇ、Ｂの値である。この背景色としては、例えば下記式（８）に示すように、現在の注目画素よりも上方、即ち、垂直座標（ｙ座標）が小さい領域における画素の平均値を使用することができる。
【００７３】
【数８】

【００７４】
ここで、Ｖ（ｎ）は肌色領域抽出部１２において得られた肌色領域ｎに対応する長方形領域の頂点座標であり、従って、上記式（８）は、探索範囲６３の内部において、現在の注目画素の座標位置（ｘ，ｙ）の上方のｍライン分の平均値を背景色とすることを示している。この場合、画像６０の最上部からｍラインにおいては上記式（８）による背景色の算出が不可能となる。即ち、画像６０最上部のｙ座標をｙ＝ｙ０としたとき背景を算出する際に使用されるのは、最上部（ｙ＝ｙ０）からｙ＝ｙ１＝ｙ０＋ｍまでであり、従って、この領域（ｙ０≦ｙ≦ｙ１）の背景を利用する最小の注目画素のｙ座標＝ｙ２＝ｙ１＋１＝ｙ０＋（ｍ＋１）であり、最上部（ｙ＝ｙ０）からｍ＋１ライン目となる。この場合は、利用できるラインのみを使用して上記式（８）と同様に背景色を算出したり、又は探索範囲６３の開始を、画像最上部からｍ＋１ライン目から始める等の方法をとることができる。
【００７５】
上記式（８）によって順次更新される背景色に対し、上記式（７）の色の差ｄを算出し、この値が所定の閾値Ｔよりも大きい画素が出現した時点で、その垂直座標ｙを頭頂部の高さＴＯＨとする。検出された頭頂部の高さＴＯＨは眼検出部１５及び領域修正部１８に送られる。
【００７６】
なお、検索範囲の各画素の値と背景色との差ｄに対する閾値は、予め設定された固定値を使用することもできるが、次に述べるように適応的に変化させることもできる。即ち、例えば背景色の更新に応じて、注目画素（ｘ，ｙ）の上方ｍラインにおける画素値の分散から下記式（９）により閾値Ｔを算出することができる。このように、背景色の更新に応じて差ｄに対する閾値を変化させることにより、より正確に頭頂部の検出を行うことができる。
【００７７】
【数９】

【００７８】
ここで、ｗは予め設定された係数である。またｖａｒ（Ｒ（ｘ，ｙ））、ｖａｒ（Ｇ（ｘ，ｙ））、ｖａｒ（Ｂ（ｘ，ｙ））はそれぞれＲ、Ｇ、Ｂの値の分散であり下記式（１０）によって算出される。
【００７９】
【数１０】

【００８０】
（２−２）人物の口検出
次に、口検出部１４は、肌色領域抽出部１２により抽出された各肌色領域に対し、口の位置（高さ）を検出する。先ず、頂点リストＶ（ｎ）によって表される長方形領域内において、肌色領域としては抽出されていない各画素（ｘ，ｙ）に対して、赤みの強さを示す下記式（１１）の値ｒｄｓｈ（ｘ，ｙ）を算出する。
【００８１】
【数１１】

【００８２】
ここで、Ｒ（ｘ，ｙ）、Ｇ（ｘ，ｙ）、Ｂ（ｘ，ｙ）はカラー画像の座標（ｘ，ｙ）における各値を示す。算出された値ｒｄｓｈ（ｘ，ｙ）は、図６に示すように水平方向（ｘ軸方向）に累積されて、下記式（１２）に示すヒストグラムＨｒｄｓｈ（ｙ）が生成される。
【００８３】
【数１２】

【００８４】
ここで、Ｖ（ｎ）及びＡ（ｘ，ｙ）は、いずれも肌色領域抽出部１２から送られたデータであって、夫々肌色領域ｎに対応する長方形領域の頂点座標、及び領域マップを示す。
【００８５】
次に、ヒストグラムＨｒｄｓｈ（ｙ）は、ノイズ等を除去するため、必要に応じて１次元ローパスフィルタによって平滑化された後、ヒストグラムＨｒｄｓｈ（ｙ）の最大値における垂直座標ｙが口の高さＨＯＭとして検出される。検出された口の高さＭＯＵは、眼検出部１５、顎検出部１６、中心線検出部１７、及び判定部１９に送られる。
【００８６】
なお、更に正確に口の高さを検出するために、赤みの強さｒｄｓｈを算出する際に、注目画素の位置及び注目画素の明るさ等に応じて重み付けすることもできる。例えば、頂点リストＶにおける頂点座標Ｖ（ｎ）で表される長方形領域内において、水平方向の中央位置に口が出現する可能性が高いことが予め期待できる場合には、注目画素のｘ座標が（Ｖ（ｎ）．ｓｔｘ＋Ｖ（ｎ）．ｅｄｘ）／２に近いほど大きな係数を上記式（１１）で算出される値ｒｄｓｈに積算し、その結果を上記式（１２）によって累積することもできる。また、口の領域の明るさとして適当な範囲を限定できる場合には、注目画素の明るさがその範囲に近いほど大きな係数を上記式（１１）で算出される値ｒｄｓｈに積算し、その結果を上記式（１２）によって累積すること等もできる。
【００８７】
（２−３）人物の眼検出
次に、眼検出部１５は、肌色領域抽出部１２で抽出された各肌色領域に対して眼の位置（高さ）を検出する。先ず、頭頂部検出部１３によって検出された頭頂部の高さＴＯＨと口検出部１４によって検出された口の高さＨＯＭとから、垂直方向（ｙ軸方向）の眼の探索範囲を例えば下記式（１３）により算出する。
【００８８】
【数１３】

【００８９】
ここで、ｅ１及びｅ２は予め設定された係数である。ｅｔｏｐ及びｅｂｔｍは、夫々検索範囲の垂直座標における下限値及び上限値である。そして、これら垂直座標における下限値及び上限値に挟まれ、且つ注目する肌色領域に対応する長方形領域内に存在する画素に対して水平方向のエッジ（以下、水平エッジという。）の強度ｅｄｇｅ（ｘ，ｙ）を検出する。水平エッジの強度を検出する方法としては、既存のどのようなものを使用してもよいが、例えば図７に示すようなフィルタ係数を有する微分フィルタを、Ｒ、Ｇ、Ｂの各画像に施し、各画像に対するフィルタ出力の絶対値和を水平エッジの強度として使用することができる。
【００９０】
入力カラー画像の各座標において算出された水平エッジの強度ｅｄｇｅ（ｘ，ｙ）は、水平方向（ｘ軸方向）に累積されて、長方形領域内における垂直方向の水平エッジを示すヒストグラムＨｅｄｇｅ（ｙ）が下記式（１４）により算出される。
【００９１】
【数１４】

【００９２】
ここで、Ｖ（ｎ）は肌色領域抽出部１２で得られた肌色領域ｎに対応する長方形領域の頂点座標である。図８は、生成されたヒストグラムＨｅｄｇｅ（ｙ）と肌色領域７１に対応する長方形領域７２との関係を示す模式図である。
【００９３】
ヒストグラムＨｅｄｇｅ（ｙ）は、ノイズ等を除去するため、必要に応じて１次元ローパスフィルタによって平滑化された後、その最大値に対応する垂直座標ｙが眼の高さＨＯＥとして検出される。
【００９４】
なお、眼の高さの探索範囲としては、上記式（１３）に示したものの他に、対応する長方形領域の頂点座標を使用して、下記式（１５）により設定することもできる。
【００９５】
【数１５】

【００９６】
ここで、ｅ３、ｅ４は予め設定された係数である。また、上記式（１３）によって算出されるｅｂｔｍが、肌色領域を囲む長方形領域の頂点座標のＶ（ｎ）．ｓｔｙより小さい場合、頭頂部の高さＴＯＨ又は口の高さＨＯＭの検出が適切に行なわれていない可能性が高い。そこで、このような場合には、対応する長方形領域の頂点座標Ｖ（ｎ）に位置座標としては無効な値である例えば−１を格納して頂点リストＶを修正することができる。
【００９７】
検出された眼の高さＨＯＥは、顎検出部１６及び判定部１９に送られる。また、修正された頂点リストＶは顎検出部１６、中心線検出部１７、及び領域修正部１８に送られる。
【００９８】
（２−４）人物の顎検出
顎検出部１６では、眼検出部１５において修正された頂点リストＶに無効ではない頂点座標を有する各肌色領域に対して、顎の位置（高さ）を検出する。顎の高さの検出は、例えば図９に示すように、人物の顔８０においては顎と口との間の距離８１と、眼と口との間の距離８２との比がほぼ一定であると仮定して、下記式（１５）により推定することができる。
【００９９】
【数１６】

【０１００】
ここで、ｃは、予め設定された係数であり、ＨＯＣは顎の高さを示す。算出された顎の高さＨＯＣは領域修正部１８に送られる。
【０１０１】
（２−５）人物の顔の中心線検出
次に、顔の中心線検出部１７は、眼検出部１５において修正された頂点リストＶに無効ではない頂点座標を有する各肌色領域に対して、顔を左右に分割する中心線の位置を検出する。
【０１０２】
ここでは、はじめに口検出部１４で検出された口の高さＨＯＭを中心として垂直方向の座標における口探索範囲を設定する。この探索範囲は、例えば対応する長方形領域の垂直方向における幅から下記式（１７）により算出することができる。
【０１０３】
【数１７】

【０１０４】
ここで、ｍは予め設定された係数であり、Ｖ（ｎ）は肌色領域ｎに対応する長方形領域の頂点座標である。上記式（１７）により算出された夫々ｍｔｏｐ及びｍｂｔｍを、探索範囲のｙ座標の夫々下限値及び上限値とする。また、水平方向の探索範囲は、長方形領域の水平方向の幅とすることができる。即ち、ｘ座標の上限及び下限は、長方形領域の夫々左端Ｖ（ｎ）．ｓｔｘ及び右端Ｖ（ｎ）．ｅｄｘとすることができる。図１０は、肌色領域９１に対応する長方形領域９２における口の高さＨＯＭ及び検索範囲ｍｔｏｐ、ｍｂｔｍを示す模式図である。
【０１０５】
次に、設定された探索範囲に存在し、かつ肌色領域に含まれない画素に対して上記式（１１）により赤みの強さを算出し、図１０に示すように、赤みの強さの値が閾値よりも大きくなる画素の水平座標の平均値を中心線の水平座標位置ＣＯＨとして検出する。赤みの強さを算出する際に、肌色領域に属する画素を除くことにより、肌色領域に属する画素の影響を排除することができ、極めて高精度に顔の中心線を検出することができる。こうして、検出された顔中心線の位置ＣＯＨは領域修正部１８及び判定部１９に送られる。
【０１０６】
なお、赤みの強さに対する閾値は予め設定された固定値を使用することもできるが、次に述べるように適応的に変化させることもできる。即ち、はじめに探索範囲内における全ての画素の平均値、又は最大値を算出し、これらに予め設定された係数を積算して得られる値を閾値として使用ようにすることもできる。
【０１０７】
また、水平座標の平均値を算出する際、ノイズ等の影響を軽減するために，ロバスト推定と呼ばれる方法を使用することもできる。即ち、例えば赤みの強さが閾値よりも大きい画素のｘ座標の算術平均値を初期値ｃｅｎｔｅｒ（０）として、下記式（１８）に示す演算を繰り返し行なう。
【０１０８】
【数１８】

【０１０９】
ここで、ｃｅｎｔｅｒ（ｎ）は第ｎ回目の繰り返しにおいて算出される中心位置であり、ｒｄｓｈ（ｘ，ｙ）は座標（ｘ，ｙ）おける赤みの強さを示し、Ｔｒｄｓｈはその赤みの強さに対する閾値を表している。ｈ（ｄ）は、各ｘ座標に対する重みを示し、既に算出されている中心位置ｃｅｎｔｅｒ（ｎ）とｘ座標との差ｄの関数として与えられる。関数ｈ（ｄ）としては、差ｄが大きいｘ座標ほど小さな重みが与えられるように、例えば下記式（１９）に示すものを使用することができる。
【０１１０】
【数１９】

【０１１１】
ｇは、関数ｈ（ｄ）の広がりを示すパラメータであり、各繰り返し毎に予め定められた方法により減少させていく。繰り返し処理は、ｇの値が予め設定された閾値よりも小さくなるか、あるいは算出されたｃｅｎｔｅｒ（ｎ＋１）と前の繰り返しにおいて算出されたｃｅｎｔｅｒ（ｎ）との差が、予め設定された別の閾値よりも小さくなるまで行なわれる。
【０１１２】
なお、収束するまでの繰り返し回数は増大するが、より確実に正しい値に収束させるために、同じｇを使用して、上記式（１８）の演算を繰り返すこともできる。即ち、同じｇを使用して、ｃｅｎｔｅｒ（ｎ＋１）とｃｅｎｔｅｒ（ｎ）との差が閾値よりも小さくなるまで繰り返し、ｃｅｎｔｅｒ（ｎ）が収束した後、ｇの値を所定の方法により減少させる。この場合、ｇの値が閾値よりも小さくなった時点で繰り返し処理を終了する。
【０１１３】
（２−６）長方形領域の修正
領域修正部１８は、眼検出部１５において修正された頂点リストＶに無効ではない頂点座標を有する各肌色領域に対して、長方形領域を改めて算出し、頂点リストＶの修正を行なう。例えば、頭頂部検出部１３で得られた頭頂部の高さＴＯＨ、顎検出部１６で得られた顎の高さＨＯＣ、及び中心線検出で得られた中心線の位置ＣＯＨを使用して、図１１に示すように、長方形領域１００を設定することができる。即ち、修正後の長方形領域１００を示す２つの頂点座標｛（ｓｔｘ、ｓｔｙ），（ｅｄｘ、ｅｄｙ）｝は下記式（２０）により算出することができる。
【０１１４】
【数２０】

【０１１５】
ここで、ａｓｐは人物の顔の幅に対する高さの比を示す係数であり、適当な値が予め設定されているものとする。
【０１１６】
肌色領域ｎに対して新たに算出された頂点座標は、頂点リストＶに上書きされ判定部１９に送られる。
【０１１７】
（３）顔判定
判定部１９は、領域修正部１８において修正された頂点リストＶに無効ではない頂点座標を有する各肌色領域に対して、その肌色領域が顔領域であるか否かの判定を行なう。顔領域の判定は、例えば人物の顔領域では眼の部分及び口の部分に水平エッジが多く分布すること、また唇の色が他の部分に比べて赤みが強いことを利用し、これらの条件が口検出部１３で検出された口の高さＨＯＭ、及び眼検出部１４で検出された眼の高さＨＯＥにおいて成立しているか否かを検証することにより行なうことができる。判定結果は、顔領域であるか否かを表す２値のフラグｆａｃｅｆｌａｇとして出力される。
【０１１８】
以下、判定部１９について更に詳細に説明する。図１２は判定部１９を示すブロック図である。図１２に示すように、判定部１９は、領域修正部１８において修正された頂点リストＶと、口の高さＨＯＭ、眼の高さＨＯＥ及び顔の中心線の位置ＣＯＨとが入力され、左右の眼の領域及び口領域を設定する領域設定部２１と、修正された頂点リストＶとカラー画像とが入力されて、長方形領域内の肌色画素における水平方向のエッジ（水平エッジ）を算出する水平エッジ検出部２２と、左右の眼の領域データ及び長方形領域の水平方向のエッジデータが入力され、眼であるか否かを判定する眼パターン判定部２３と、口領域のデータ及び長方形領域の水平方向のエッジデータが入力され、口であるか否かを判定する口パターン判定部２４と、眼パターン判定部２３及び口パターン判定部２４の判定結果から顔であるか否かを判定する総合判定部２５とから構成される。
【０１１９】
（３−１）各特徴点の領域設定
領域設定部２１は、頂点座標Ｖ（ｎ）、口の高さＨＯＭ、及び眼の高さＨＯＥから推定される右眼、左眼及び口の領域を図１３に示すように、口の高さＨＯＭ、及び眼の高さＨＯＥ近傍の長方形領域として設定する。
【０１２０】
即ち、右眼領域は、例えば下記式（２１）によって算出される頂点座標｛（ｅｙｅＲｓｔｘ、ｅｙｅＲｓｔｙ），（ｅｙｅＲｅｄｘ、ｅｙｅＲｅｄｙ）｝を有する長方形領域とすることができる。
【０１２１】
【数２１】

【０１２２】
ここで、ｅｙｅｓｉｚｅｘは片眼の領域の水平方向の幅を示し、頂点座標Ｖ（ｎ）で表される長方形領域の幅に係数ｓ１を積算することで算出する。また、ｂｅｙｅｓは両眼間の距離であり、同様に、頂点座標Ｖ（ｎ）で表される長方形領域の幅に係数ｓ２を積算することで算出することができる。また、ｅｙｅｓｉｚｅｙは眼の領域の高さ（垂直方向の幅）を示し、上記眼の幅に予め設定された割合ｅｙｅａｓｐを積算することで算出する。
【０１２３】
同様に、左眼の領域は、下記式（２２）によって算出される頂点座標｛（ｅｙｅＲｓｔｘ、ｅｙｅＬｓｔｙ），（ｅｙｅＬｅｄｘ、ｅｙｅＬｅｄｙ）｝を有する長方形領域とすることができる。
【０１２４】
【数２２】

【０１２５】
ここで、ｅｙｅｓｉｚｅｘ、ｅｙｅｓｉｚｅｙ、ｂｅｙｅｓは上記式（２１）と同一の値を有する。
【０１２６】
一方、口領域は下記式（２３）によって算出される頂点座標｛（ｍｏｕｓｔｘ、ｍｏｕｓｔｙ），（ｍｏｕｅｄｘ、ｍｏｕｅｄｙ）｝を有する長方形領域とすることができる。
【０１２７】
【数２３】

【０１２８】
ここで、ｍｏｕｓｉｚｅｘは口領域の水平方向の幅を示し、頂点座標Ｖ（ｎ）で表される長方形領域の水平方向の幅に係数ｓ３を積算することで算出する。また、ｍｏｕｓｉｚｅｙは口領域の高さ（垂直方向の幅）を示し、口の幅に予め設定された割合ｍｏｕａｓｐを積算することで算出することができる。
【０１２９】
設定された右眼領域、左眼領域、及び口領域の頂点座標は、夫々頂点リストＶｅｙｅＲ、ＶｅｙｅＬ、Ｖｍｏｕに上記式（６）と同様に格納される。ＶｅｙｅＲ、ＶｅｙｅＬは眼パターン判定部２３に、Ｖｍｏｕは口パターン判定部２４にそれぞれ送られる。
【０１３０】
（３−２）各領域の水平エッジの検出
水平エッジ検出部２２は、各肌色領域に対して、頂点座標Ｖ（ｎ）で表される長方形領域内に存在し、且つ領域マップＡ（ｘ，ｙ）の値がｎである肌色画素において水平方向のエッジの検出を行なう。水平エッジを検出する方法は既存のどのようなものを用いてもよいが、例えば対象となる画素毎において、図７に示すフィルタ係数を有する微分フィルタで水平方向のエッジ強度を算出し、図１４に示す微分係数を有する微分フィルタを使用して垂直方向のエッジ強度を算出し、下記式（２０）に示す条件を満たす画素を水平方向のエッジを構成する画素（水平方向のエッジが所定の閾値以上である画素）として検出する等の方法がある。
【０１３１】
【数２４】

【０１３２】
ここで、ｅｄｇｅＨ、ｅｄｇｅＶ及びＴｅは、夫々水平方向のエッジ強度、垂直方向のエッジ強度及び閾値を表している。
【０１３３】
水平方向のエッジを構成する画素の検出結果は、２次元配列であるエッジマップＭｅｄｇｅに２値画像として格納される。例えば、座標（ｘ，ｙ）の画素が上記式（２４）の条件を満足する画素、即ち、水平エッジを構成する画素として検出された場合には１がＭｅｄｇｅ（ｘ，ｙ）に格納され、座標（ｘ，ｙ）の画素が水平エッジを構成しない場合には０がＭｅｄｇｅ（ｘ，ｙ）に格納される。生成されたエッジマップＭｅｄｇｅは眼パターン判定部２３、及び口パターン判定部２４に送られる。
【０１３４】
なお、水平エッジ検出のための閾値Ｔｅは、予め設定された固定値を使用することもできるが、次のように適応的に変化させることも可能である。即ち、例えば予め対象となる画素、即ち、肌色領域抽出部１２において抽出された肌色領域に含まれる画素全てに対して水平エッジ強度を算出し、その平均値に予め設定された係数を積算した値を閾値として使用することもできる。
【０１３５】
また、ここでは肌色領域抽出部１２において抽出された肌色領域に含まれる画素のみを対象に水平エッジの検出を行なう例を示したが、通常、眼及び口の部分の色は肌色とは異なるため、これらの部分自体は肌色領域としては抽出されない。従って、眼及び口のエッジはちょうど肌色領域と非肌色領域との境界部に相当するため、肌色領域上のみでエッジの検出を行なうとエッジを構成する十分な数の画素が検出されない可能性がある。従って、より安定にエッジを検出するためには、例えば、領域修正部１８により修正された頂点座標により示される長方形領域内の肌色ではない画素であって、その隣接画素が肌色領域ｎに含まれるものに対しても水平エッジを検出するようにしてもよい。
【０１３６】
（３−３）眼領域における眼パターンらしさの判定
眼パターン判定部２３は、右眼頂点リストＶｅｙｅＲ及び左眼頂点リストＶｅｙｅＬ、及びエッジマップＭｅｄｇｅを使用して、眼検出部１５において検出された眼の高さＨＯＥ近傍の領域が眼の特徴を備えているか否かを判定する。
【０１３７】
ここでは、先ず、右眼頂点リストＶｅｙｅＲ及び左眼頂点リストＶｅｙｅＬで示される夫々右眼領域及び左眼領域内に存在する水平方向のエッジを構成する夫々画素数ｅｃｏｕｎｔＲ及びｅｃｏｕｎｔＬを下記式（２５）としてカウントする。
【０１３８】
【数２５】

【０１３９】
ここで、ｅｙｅＲ及びｅｙｅＬは、夫々右眼領域及び左眼領域の内部に存在する画素の集合である。
【０１４０】
また、垂直方向の座標が右眼領域又は左眼領域の下限と上限との間であり、且ついずれの眼領域にも含まれていない水平エッジの画素数ｅｃｏｕｎｔＯを下記式（２６）としてカウントする。
【０１４１】
【数２６】

【０１４２】
一般に、人物の顔領域の眼の位置においては、実際に眼が存在する部分（領域）に多くの水平方向のエッジを構成する画素が多く存在し、それ以外の部分では水平方向のエッジを構成する画素が少ない。これに従い、下記式（２７）に示す眼のパターンらしさの条件が満たされた場合に、眼検出部１５において検出された眼の高さＨＯＥ近傍の領域が眼の特徴を備えているものと判定する。
【０１４３】
【数２７】

【０１４４】
ここで、上記式（２７）の第１の条件における左辺の値は、右眼と左眼の領域に含まれる水平方向のエッジを構成する画素が多く、またそれ以外の領域に水平方向のエッジを構成する画素数が少ない場合に大きくなる。また、上記式（２７）の第２の条件におけるａｒｅａＲ及びａｒｅａＬは、夫々右眼領域及び左眼領域の内部に存在する画素の総数であり、左辺全体は右眼領域及び左眼領域における水平方向のエッジを構成する画素の密度を示す。即ち、いずれの条件も上述した眼の特徴を反映していることになる。この各条件における閾値であるＴｅｄｓｔ及びＴｅｄｎｓは、予め適切な値が設定される。
【０１４５】
上記式（２７）による判定結果は２値のフラグによって表される。例えば、上記式（２７）の条件が全て満たされる場合には、フラグに１を設定し、判定結果が眼であることを示す。一方、上記式（２７）の条件がいずれか一方でも満足されない場合は、フラグに０を設定し判定結果が眼ではないことを示す。判定結果を表すフラグｅｙｅｆｌａｇは総合判定器２５に送られる。
【０１４６】
なお、ここでは判定条件として上記式（２７）式を使用したが、上述した眼の特徴を反映したものであれば他の条件を使用することも可能である。
【０１４７】
（３−４）口領域における口パターンらしさの判定
口パターン判定部２４では、口領域Ｖｍｏｕ及びエッジマップＭｅｄｇｅを使用して、口検出部１４において検出された口の高さＨＯＭ近傍の領域が口の特徴を備えているか否かを判定する。
【０１４８】
ここでは、先ず、口領域Ｖｍｏｕの内部に存在する水平方向のエッジを構成する画素数ｍｃｏｕｎｔＭ及び口領域Ｖｍｏｕ内部に存在する全画素の赤み強さの平均値ａｖｅｒｄｓｈＭを下記式（２８）により算出する。
【０１４９】
【数２８】

【０１５０】
ここで、ｍｏｕは口領域内部に存在する画素の集合であり、ＮＭは口領域内部に存在する画素の総数である。また、ｒｄｓｈ（ｘ，ｙ）は画素（ｘ，ｙ）における赤み強さで、例えば上記式（１１）によって算出される値を使用することができる。
【０１５１】
次に、垂直方向の座標が口領域の下限と上限との間であり、且つ口領域に含まれていない画素のうち、水平エッジを構成する画素の数ｍｃｏｕｎｔＯ、及び垂直方向の座標が口領域の下限と上限との間であり、且つ口領域に含まれていない全画素の赤み強さの平均値ａｖｅｒｄｓｈＯを下記式（２９）により算出する。
【０１５２】
【数２９】

【０１５３】
ここで、ＮＯは、対象としている領域、即ち、垂直方向の座標が口領域の下限と上限との間であり、且つ口領域に含まれていない画素の総数を示す。
【０１５４】
一般に、人物の顔領域の口の高さ位置では、実際に口が存在する部分（領域）に水平方向のエッジを構成する画素が多く存在し、それ以外の部分では水平方向のエッジを構成する画素が少ない。また、口の部分はその他の部分と比べて赤みが強い。これに従い、下記式（３０）に示す口のパターンらしさの条件が満たされた場合に、口検出部１４において検出された口の高さＨＯＭ近傍の領域（口領域）が口の特徴を備えているものと判定する。
【０１５５】
【数３０】

【０１５６】
ここで、上記式（３０）の第１の条件における左辺の値は、口領域に含まれる水平方向のエッジを構成する画素が多く、口領域以外の領域に水平方向のエッジを構成する画素が少ない場合に大きくなる。また、上記式（３０）の第２の条件の左辺におけるａｒｅａＭは、口眼領域内に存在する画素の総数であり、左辺全体は口領域における水平方向のエッジを構成する画素の密度を示す。更に、上記式（３０）の第３の条件における左辺の値は、口領域における赤みが強く、口領域以外の領域における赤みが弱い場合に大きくなる。更にまた、上記式（３０）の第４の条件は、口領域内部の赤みの強さ自体が所定の閾値Ｔｒｄｓｈｍより大きいことを要求するものである。即ち、いずれの条件も上述した口の特徴を反映していることになる。上記式（３０）の第１乃至第４の条件における夫々閾値Ｔｅｄｓｔｍ、Ｔｅｄｎｓｍ、Ｔｃｄｓｔｍ及びＴｒｄｓｈｍは、予め適切な値が設定される。
【０１５７】
上記式（３０）による判定結果は、２値のフラグによって表される。例えば、上記式（３０）の条件が全て満たされる場合には、フラグに１に設定し、口領域を口のパターンと判定したことを示す。一方、上記式（３０）のうち、いずれか１つでも条件が満たされていない場合には、フラグに０を設定し、口領域を口ではないと判定したことを示す。この判定結果を表すフラグｍｏｕｆｌａｇは総合判定器２５に送られる。
【０１５８】
なお、ここでは判定条件として上記式（３０）を使用したが、上述した口の特徴を反映したものであれば他の条件を使用することも可能である。
【０１５９】
（３−５）抽出された領域が顔であるか否かの判定
総合判定部２５は、眼判定部２３及び口領域２４から送られてくる夫々フラグｅｙｅｆｌａｇ及びｍｏｕｆｌａｇがいずれも１である場合、即ち、領域設定部２１で設定した眼の領域及び口の領域が、夫々眼の特徴及び口の特徴を具えていると判定された場合に限り、現在注目している肌色領域を顔領域と判定し、その結果を２値のフラグｆａｃｅｆｌａｇとして出力する。
【０１６０】
なお、本実施の形態では、頭頂部の高さＴＯＨは眼の探索範囲を制限するために使用されているが、眼の高さＨＯＥが検出された後、この眼の高さＨＯＥと口の高さＨＯＭとを使用し、頭頂部の高さＴＯＨの検証を行なうようにしてもよい。即ち、眼と口との距離から頭頂部が存在すべき上限及び下限を例えば夫々下記式（３１）により算出する。
【０１６１】
【数３１】

【０１６２】
頭頂部の高さＴＯＨがこれら２つの値の間にない場合には、頭頂部検出部１３において正しい高さが検出されなかったものとして、予め定められた方法により頭頂部の高さの修正を行なう。例えば、眼と口との距離に対する頭頂部と口との距離の比ｈ３を予め設定しておき、下記式（３２）により、頭頂部の高さＴＯＨを設定し直す。
【０１６３】
【数３２】

【０１６４】
また、ここでは眼の領域、両眼間距離、及び口の領域の大きさを、上記式（２１）、（２２）、及び（２３）に示す肌色領域を囲む長方形領域の大きさに対して相対的に算出する方法について述べたが、肌色領域抽出において、影によって顔の一部が欠けて抽出されたり、又は、顔から肩のあたりまでが１つの肌色領域として抽出された場合、眼領域や口領域が適切に設定されない可能性がある。この問題を回避するために、例えば上記式（２１）乃至（２３）において、領域の大きさを決める係数Ｓ１、Ｓ２、Ｓ３を複数用意しておき、それぞれの値に対して判定を行なうようにすることができる。そして、何れかの係数に対して顔領域であるという判定が得られた場合、現在注目している肌色領域を顔領域と判定する。
【０１６５】
本実施の形態においては、抽出された肌色領域に基づき、頭頂部及び口の位置を検出し、これらの位置から眼の検索範囲を設定して眼の位置を検出するため、極めて高精度に眼の位置を検出することができる。また、顎の位置は、眼と口の位置から算出することにより、顔と首との輝度及び色の差が小さく、高精度に検出することが難しい場合にも顎の位置の検出を正確に行うことができる。更に、顔の中心線は、口の赤みの強さに基づき検出されるため、極めて高精度に顔中心線を検出することができる。更にまた、顔判定部において、眼のパターンらしさ及び口のパターンらしさを判定し、この判定結果に基づき顔であるか否かの総合判定をするため、複数の顔が含まれている場合であっても、顔であるか否かの判定結果の信頼性が高い。
【０１６６】
また、判定部１９により顔と判定される肌色領域が複数存在する場合に、複数の顔領域から、例えばその顔領域の位置に基づき１つの顔領域を選択する選択部（図示せず）を設けることもできる。これにより、例えば、複数の顔領域が存在する画像から１つの顔領域を抽出してトリミング処理を施すことができる。なお、判定部１９に、顔領域を選択する機能をもたせるようにしてもよい。
【０１６７】
第２の実施の形態
次に、本発明の第２の実施の形態について説明する。図１５は、本実施の形態の肖像画像撮影装置を示すブロック図である。本実施の形態においては、第１の実施の形態における画像処理装置を肖像画像（写真）撮影装置に応用したものである。
【０１６８】
図１５に示すように、肖像画像撮像装置３０は、電子スチルカメラ及びビデオカメラ等により、人物が撮影されたカラー画像が入力され、デジタルデータとして出力する撮像器３１と、画像データが入力されて顔領域を検出する顔検出器３２と、検出された顔領域が複数ある場合に、トリミングの対象となる顔領域を選択する領域選択器３３と、選択された顔領域のトリミングを行うトリミング器３４とから構成される。
【０１６９】
顔領域抽出器３２は、第１の実施の形態における画像処理装置の処理方法に従って、人物の顔領域を抽出する。但し、第１の実施の形態における判定部１９では、各肌色領域に対して顔であるか否かを表すフラグｆａｃｅｆｌａｇを出力するものとしたが、本実施の形態では、顔領域ではないと判定された肌色領域に対応する頂点座標Ｖ（ｎ）に、位置座標として無効な値の例えば−１を格納した後、これを領域選択器３３に送る。
【０１７０】
領域選択器３３では、顔領域判定器３２によって複数の領域が顔と判定された場合、これらの複数の領域の中からトリミングの対象となる顔領域を１つ選択する。例えば、撮像された画像には１人の人物しか存在していないことが明らかであれば、真の顔領域は１つであり、その他の領域は顔領域判定器３２における誤検出と考えられる。このような場合、真の顔領域は他の肌色領域よりも上部に位置するものと推定される場合は、頂点リストＶに含まれる有効な長方形のうち、その中心が最も上部に位置するものを選択するようにすることができる。
【０１７１】
また、撮像された画像に複数の人物が存在している可能性がある場合でも、撮像時の指示等により、入力画像においてトリミングの対象となる人物の顔が、例えば最も画像中央に近い位置に存在することを期待できる場合には、頂点リストＶに含まれる有効な長方形のうち、その中心が画像中央に最も近いものを選択するようにすることができる。
【０１７２】
更に、複数の顔の領域が抽出される主な理由が、例えば子供の顔を撮像するために、親がその子供を抱いて撮像することである場合、トリミングの対象となる子供の顔領域は、抽出された顔の領域の中で最も下部に位置すると推定できるため、頂点リストＶに含まれる有効な長方形のうち、その中心が最も下部に位置するものを選択するようにすることができる。
【０１７３】
これら対象領域選択の方法は、利用形態を考慮して予め設定しておく他に、インターフェース（図示せず）を設けることにより、撮影者及び／又は被撮影者等が手動で設定できるようにしておくことも可能である。
【０１７４】
選択された顔の領域以外の肌色領域に対応する頂点リストＶには位置座標として無効な値、例えば−１等が格納される。修正された頂点リストＶは、トリミング器３４に送られる。
【０１７５】
トリミング器３４は、送られてきた頂点リストＶに含まれる有効な長方形領域Ｖ（ｎ）に対応して、切り出す部分画像の位置を決定する。図１６は、入力カラー画像から切り出す部分画像の切り出し位置を示す模式図である。図１６に示すように、カラー画像１１０において、頂点リストＶの頂点座標Ｖ（ｎ）によって示される長方形領域１１１の中心位置（ｘ０、ｙ０）がトリミング後の画像１１２の所定の位置（ｔｘ０，ｔｙ０）と一致するように、下記式（３３）に示す２つの頂点座標で表される長方形領域を部分画像１１２として切り出す。
【０１７６】
【数３３】

【０１７７】
ここで、ｔｗｄｔ及びｔｈｇｔは、予め設定されたトリミング後の画像の夫々幅（水平方向の長さ）及び高さ（垂直方向の長さ）を示す。
【０１７８】
トリミングされた画像は、プリンタ及びモニタ等、図示しない画像表示装置や、画像記憶装置等に出力される。
【０１７９】
なお、トリミング器３４におけるトリミング処理は、ここで述べたものに限定されるものではなく、例えば次に述べるような方法を用いることも可能である。
【０１８０】
即ち、顔領域抽出器３２は、頂点リストＶのほかに、各顔領域に対して算出された眼の高さＨＯＥ、及び中心線位置ＣＯＨをトリミング器３４に送る。トリミング器３４では、眼の高さと中心線がトリミングされた画像上において所定の位置となるように部分画像の切り出しを行なう。眼の高さＨＯＥの代わりに頭頂部の高さＴＯＨ又は口の高さＨＯＭを使用してもよい。
【０１８１】
特に、トリミング後の画像において頭部がはみ出すことが許容されない場合には、頭頂部の高さＴＯＨを用いることが有効である。但し、図１７に示すように、同一の入力カラー画像１２０において、複数の人物１２１，１２２が存在し、この人物１２１，１２２が水平方向に重なっている場合には、夫々頭頂部探索１２３，１２４範囲も重なってしまうため、人物１２１，１２２のうち、真の頭頂部がより下方（垂直座標が大きい）に位置する顔領域、即ち、肌色領域に対応する長方形領域の水平方向の辺のうち垂直座標が小さい方の辺における垂直座標が、より大きい顔領域を有する人物１２２に対しては頭頂部の高さＴＯＨが正しく検出できないことになる。従って、頭頂部の高さを使用したトリミングでは、トリミングすべき顔領域が一番上に位置していない限り、良好な結果が得られない可能性がある。このような問題を回避するために、頭頂部の高さのチェックを行なうと共に、その値が不適切である場合には修正を行なうようにしてもよい。
【０１８２】
例えば、ＴＯＨのチェックとしては、既に検出されている眼の高さＨＯＥ及び口の高さを使用し、下記式（３４）に示す条件が満たされているか否かを判定する。
【０１８３】
【数３４】

【０１８４】
ここで、ｍｅｄｓｔは眼の高さと口の高さとの差ＨＯＥ−ＨＯＭであり、ｈ１，ｈ２は予め設定された係数である。上記式（３４）に示す条件が満たされない場合には、下記式（３５）によってＴＯＨの値を修正する。
【０１８５】
【数３５】

【０１８６】
また、顔領域判定器３２において複数の領域が顔と判定された場合、複数の領域全てをトリミングの対象として選択するようにすることもできる。この場合、トリミング器３４では、顔と判定された複数の領域を全て含むような部分画像を切り出し、必要に応じて縮小処理を施して所定の大きさの画像を得る。
【０１８７】
本実施の形態においては、第１の実施の形態における画像処理装置を使用しているため、顔領域検出器により顔として検出された領域の信頼性が高く、更に顔領域検出器により、顔領域だけでなく、特徴点情報、即ち、人物の頭頂部、顔の口及び眼、顔中心線の位置を検出し、これらのデータを使用してトリミングを行うため、誤検出がなく、高信頼度及び高精度で所望の顔領域のトリミングが自動的に行える肖像画像撮影装置を得ることができる。
【０１８８】
また、画像に複数の人物が含まれている場合においても、アプリケーションに応じて適切なトリミングを行うことができる。例えば、病院等におけるフォトブースで親が子供を抱いて撮影した場合、抽出される顔領域の位置又は大きさ等の情報に基づき、領域選択器３３により、複数の顔領域から特定の顔領域を選択することができる。即ち、例えば、顔領域が２つ抽出された場合に、領域選択器３３により、その位置が画面下方又はその大きさが小さいものを選択するように設定しておけば、親が子供を抱いて撮影した場合に、子供の顔を抽出してトリミングすることができる。また、何れの顔領域についてもトリミングを行うように設定しておけば、１回の画像で複数のトリミング画像を得ることができる。
【０１８９】
第３の実施の形態
次に、本発明の第３の実施の形態について説明する。本実施の形態は、第２の実施の形態と同様に画像処理装置を肖像画像撮影装置に適用したものであって、図１５に示す第２の実施の形態と同様の構成を有する。即ち、撮像器３１、顔領域抽出器３２、領域選択器３３及びトリミング器３４から構成されるものであるが、トリミング器３４における処理が異なる。第２の実施の形態では、入力画像から所定の大きさの部分画像を切り出すことでトリミングを行なっているが、本実施の形態においては、切り出す顔領域の大きさの調整も合わせて行なうものとする。
【０１９０】
例えば、上述した第１の実施の形態と同様の方法で顔領域を抽出する図１５に示す顔領域抽出器３２により、頂点リストＶのほかに、人物の眼の高さＨＯＥ、口の高さＨＯＭ、及び中心線ＣＯＨがトリミング器３４に送られる。
【０１９１】
トリミング器３４は、先ず、トリミング画像上で要求される眼と口との間隔ｍｅｄｓｔと、入力カラー画像から得られたＨＯＥとＨＯＭとの差の比ｐとを下記式（３５）により算出し、縦及び横の長さがそれぞれｐ倍となるように入力画像を拡大又は縮小する。
【０１９２】
【数３６】

【０１９３】
画像の拡大又は縮小に応じて、眼の高さＨＯＥ、口の高さＨＯＭ、及び中心線位置ＣＯＨの値もそれぞれ修正を行なう。画像の拡大処理及び縮小処理については既存のいかなる方法を使用してもよい。
【０１９４】
拡大又は縮小の施された画像に対して、眼の高さと中心線とがトリミングされた画像上において所定の位置となるように部分画像の切り出しを行なう。
【０１９５】
なお、ここでは大きさの調整を行なうために眼と口との間の間隔を使用したが、顔領域検出器３２で算出されている他の情報を使用できることもできることはいうまでもない。例えば、頭頂部と顎との間の間隔に基づいて上記式（３５）の比の値を算出してもよい。
【０１９６】
また、トリミングのための位置決めは、大きさの調整に使用した特徴点、即ち、眼及び口とは異なる特徴点に基づいて行なうことも可能である。即ち、眼と口との間の間隔で大きさの調整を行うとともに、頭頂部の高さ又は顎の高さと中心線とによって位置決めを行なうこともできる。
【０１９７】
更に、頭頂部の高さを使用する場合には、第２の実施の形態と同様、眼及び口の高さを使用してその値ＴＯＨを修正するようにしてもよい。
【０１９８】
本実施の形態においては、第２の実施の形態と同様に、顔領域抽出器から極めて高精度に検出された顔領域が入力されると共に、人物（顔）の特徴点を示す各データが入力されるため、トリミングの位置決め精度が高く、トリミングの際に拡大・縮小処理を行うことができる。
【０１９９】
第４の実施の形態
次に、本発明の第４の実施の形態について説明する。図１８は、本実施の形態における画像処理装置を示すブロック図である。本実施の形態は、上述の第１の実施の形態の画像処理装置と同様に、入力されたカラー画像から人物の顔領域を抽出する画像処理装置に適用したものであり、以下に示す第４の実施の形態において、図１乃至図１４に示す第１の実施の形態とは異なる部分についてのみ詳細な説明を行う。
【０２００】
本実施の形態における画像処理装置２１０は、カラー画像が入力され、このカラー画像をデジタルデータとして出力する画像入力部２１１と、このカラー画像から肌色領域を抽出する肌色領域抽出部２１２と、人物の頭頂部を検出する頭頂部検出部２１３と、人物の顔の中心線を検出する中心線検出部２１７と、人物の口を検出する口検出部２１４と、必要に応じて中心線を再検出し、中心線の修正を行う中心線修正部２２０と、人物の眼を検出する眼検出部２１５と、人物の顎位置を算出する顎検出部２１６と、顔領域を修正する領域修正部２１８と、抽出された肌色領域が人物の顔であるか否かを判定する判定部２１９とから構成されている。
【０２０１】
画像入力部２１１は、第１の実施の形態と同じものであるが、その出力であるカラー画像は、肌色領域抽出部２１２、頭頂部検出部２１３、口検出部２１４、眼検出部２１５、判定部２１９、及び中心線修正部２２０に送られる。
【０２０２】
また、肌色領域抽出部２１２も第１の実施の形態と同じものであるが、算出された領域マップＡは口検出部２１４、中心線検出部２１７、判定部２１９、及び中心線修正部２２０へ、頂点リストＶ（ｎ）は頭頂部検出部２１３へ送られる。
【０２０３】
頭頂部検出部２１３は、第１の実施の形態と同様に、頭頂部の高さＴＯＨを検出するものであるが、その検出に先立ち、頭頂部がカラー画像内に含まれているか否かの判定を行う機能を有している。頭頂部がカラー画像内に存在するか否かの判定には、例えば上記式（１０）と同様な方法によって、画像最上部からｍラインにおけるＲ（ｘ，ｙ），Ｇ（ｘ，ｙ），Ｂ（ｘ，ｙ）の値の分散を算出し、それらの和が予め設定された閾値以下であるか否かで判定することができる。即ち、上記分散値の和が予め設定された閾値より大きい場合には、頭頂部がカラー画像内に存在しないと判定する。カラー画像内に頭頂部が存在しないと判定された肌色領域に対応する頂点リストＶ（ｎ）には、位置座標としては無効な値、例えばー１が格納される。ここで、頭頂部がカラー画像内に存在しないと判定された場合、カラー画像の最上端を頭頂部の位置としてもよい。又は、頭頂部が検出されなかった旨を例えば音声又は音等により通知し、使用者に再度撮影するよう催促するようにしてもよい。
【０２０４】
一方、頭頂部が存在すると判定された場合は、頭頂部の高さＴＯＨの検出を行い、検出された頭頂部の高さＴＯＨは、眼検出部２１５、及び領域修正部２１８に送られ、修正された頂点リストＶ（ｎ）は、口検出部２１４、眼検出部２１５、顎検出部２１６、中心線検出部２１７、領域修正部２１８、及び中心線修正部２２０に送られる。
【０２０５】
中心線検出部２１７は、頭頂部検出部２１３で修正された頂点リストＶ（ｎ）に無効ではない頂点座標を持つ各肌色領域に対して、肌色領域を左右に２分割する中心線の位置を検出する。但しここでは、第１の実施の形態と異なり、頂点リストＶ（ｎ）によって表される長方形内部において、肌色領域として抽出された画素（ｘ，ｙ）の水平方向の座標ｘの平均値を中心線の位置ＣＯＨとして検出する。検出された中心線位置ＣＯＨは口検出部２１４、及び中心線修正部２２０に送られる。
【０２０６】
口検出部２１４は、頭頂部検出部２１３で修正された頂点リストＶ（ｎ）に無効ではない頂点座標を持つ各肌色領域に対して、口の高さを検出する。第１の実施の形態と同様に、本実施の形態においても、赤みの強さを表す上記式（１１）の値ｒｄｓｈ（ｘ，ｙ）を水平方向に累積して得られるヒストグラムＨｒｄｓｈ（ｙ）を用いる。このとき、その累積範囲は、図１９の斜線で示すように、中心線検出部２１７によって検出された中心線位置ＣＯＨの周辺部の領域２４１に制限する。すなわち本実施の形態におけるヒストグラムＨｒｄｓｈ（ｙ）は下記式（３７）によって生成される。
【０２０７】
【数３７】

【０２０８】
ここで、ｌｗは、あらかじめ設定された０より大きく１より小さい係数である。ヒストグラムＨｒｄｓｈ（ｙ）は、必要に応じて１次元ローパスフィルタによって平滑化された後、その最大値に対応する垂直座標ｙが口の高さＨＯＭとして検出される。
【０２０９】
更に、本実施の形態においては、赤みの強い画素の空間的な分布に応じて検出された口の高さＨＯＭの信頼度ｌｉｐｃｏｎｆを算出する。口は、水平方向には顔の中心にあり、口を構成する画素は中心線位置ＣＯＨを中心として比較的狭い範囲に存在していると推測できる。そこで、赤みの強い画素が空間的に大きく広がっている場合、即ち、赤みが所定の閾値以上である画素の分布が大きい場合には、検出された口の高さの信頼度が低いと評価する。
【０２１０】
信頼度ｌｉｐｃｏｎｆは例えば、次のように算出することができる。即ち、先ず、検出された中心線位置ＣＯＨと口の高さＨＯＭとから決まる座標（ＣＯＨ，ＨＭＯ）を中心とする図２０に斜線で示すような長方形領域Ｒｍを設定して、この長方形領域Ｒｍの内部に存在する各画素に対して上記式（１１）によって定義される赤みの強さを算出すると共に、算出された赤みの強さｒｄｓｈ（ｘ，ｙ）が予め設定された閾値Ｔｒｄｓｈよりも大きい画素を選択し、それらの水平座標ｘの標準偏差をｘｓｄｖ算出する。即ち、ｘｓｄｖの算出は、下記式（３８）に示すように行われる。
【０２１１】
【数３８】

【０２１２】
信頼度ｌｉｐｃｏｎｆを算出するための長方形領域Ｒｍは、中心線位置ＣＯＨと現在注目している肌色領域に対応する頂点リストＶ（ｎ）から、例えば下記式（３９）のように設定することができる。
【０２１３】
【数３９】

【０２１４】
ここで、ｍｔｏｐ，ｍｂｔｍは上記式（１７）に示したものと同じである。
【０２１５】
次に、下記式（４０）に示すように、値域が０以上１以下である適当な単調増加関数Ｆを用い、算出された標準偏差ｘｓｄｖを信頼度ｌｉｐｃｏｎｆに変換する。
【０２１６】
【数４０】

【０２１７】
ここで、Ｖ（ｎ）．ｅｄｘ−Ｖ（ｎ）．ｓｔｘ＋１は、現在注目している肌色領域を囲む長方形領域の水平方向の幅を表している。
【０２１８】
検出された口の高さＨＯＭは眼検出部２１５、顎検出部２１６、中心線修正部２２０及び判定部２１９に送られる。また、信頼度ｌｉｐｃｏｎｆは中心線修正部２２０、判定部２１９に送られる。
【０２１９】
なお、更に正確に口の高さを検出するために、第１の実施の形態と同様に、注目画素の位置やその注目画素の明るさに応じて赤みの強さｒｄｓｈに重み付けを行なうようにすることもできることはいうまでもない。また、口の高さの信頼度ｌｉｐｃｏｎｆを算出する際には、各画素の水平座標に対して同様の重み付けを行うこともできる。更に、ここでは信頼度の計算に寄与する画素を予め設定された閾値Ｔｒｄｓｈに基づいて選択しているが、この閾値を適応的に設定することも可能である。例えば、図１９に斜線で示した領域２４４の内部において、各画素の赤みの強さの平均値を算出し、その値に適当な係数を積算して得られる値を信頼度算出のための閾値Ｔｒｄｓｈとして用いることもできる。
【０２２０】
中心線修正部２２０は、頭頂部検出部２１３で修正された頂点リストＶ（ｎ）に無効ではない頂点座標を持つ各肌色領域に対して、口検出部２１４で算出された口の高さの信頼度ｌｉｐｃｏｎｆの値が予め設定された閾値よりも大きい場合にのみ、第１の実施の形態における中心線検出部１７と同様の方法、即ち例えば赤みの強さの値が閾値よりも大きくなる画素の水平座標の平均値をロバスト推定により求め、これを中心線の水平座標位置として検出する再検出を行い、その結果を出力する。信頼度ｌｉｐｃｏｎｆが閾値以下であるときは、第１の実施の形態のように、赤みの強さに基づいて中心線を検出すると誤検出が生じるおそれがあるため、既に中心線検出部２１７で検出された中心線位置ＣＯＨをそのまま出力する。出力された中心線位置ＣＯＨは領域修正部２１８、判定部２１９に送られる。
【０２２１】
眼検出部２１５は、頭頂部検出部２１３で修正された頂点リストＶ（ｎ）に無効ではない頂点座標を持つ各肌色領域に対して、眼の高さを検出する。但し、第１の実施の形態では、上記式（１４）に示したエッジ強度のヒストグラムＨｅｄｇｅ（ｙ）の最大値に対応する唯一の垂直座標のみを眼の高さＨＯＥとして検出したが、本実施の形態においては、このヒストグラムの極大点に対応する複数の垂直座標を眼の高さ候補として検出するものとする。ヒストグラムＨｅｄｇｅ（ｙ）の極大点座標は、例えば下記式（４１）を満たすような座標ｙとして検出することができる。
【０２２２】
【数４１】

【０２２３】
肌色領域ｎに対して検出された眼の高さの候補の座標は、予め定められた優先順位ｉに従って、リストｅｙｅｌｉｓｔ（ｎ，ｉ）に格納される。優先順位の決定には、例えば、ヒストグラムＨｅｄｇｅ（ｙ）の値を用いることができる。この場合、リストｅｙｅｌｉｓｔ（ｎ，ｉ）には、肌色領域ｎに対して得られたヒストグラムＨｅｄｇｅ（ｙ）の値がｉ番目に大きな極大点の垂直座標が格納されることになる。眼の高さの候補の座標が格納されたリストｅｙｅｌｉｓｔ（ｎ，ｉ）は、顎検出部２１６及び判定部２１９に送られる。
【０２２４】
なお、各肌色領域に対する眼の高さ候補の座標リストｅｙｅｌｉｓｔ（ｎ，ｉ）には、検出された全ての極大点の垂直座標を格納する必要はなく、例えばノイズなどの影響を避けるため、ヒストグラムの値が大きいものから順に、あらかじめ設定された個数の極大点のみを格納するようにしてもよい。
【０２２５】
顎検出部２１６では、頭頂部検出部２１３で修正された頂点リストＶ（ｎ）に無効ではない頂点座標を持つ各肌色領域に対して、第１の実施の形態１と同様の方法によって顎の高さを検出する。但し、本実施の形態においては、第１の実施の形態にて説明した１つの眼の高さＨＯＥではなく、複数の眼の高さ候補があるため、各眼の高さ候補に対して顎の高さの候補を算出し、顎の高さリストｃｈｉｎｌｉｓｔ（ｎ，ｉ）に格納する。ここで、ｎは現在処理している肌色領域の番号であり、ｉは対応する眼の高さ候補の優先順位を表している。算出されたリストｃｈｉｎｌｉｓｔ（ｎ，ｉ）は、領域修正部２１８に送られる。
【０２２６】
領域修正部２１８は、頭頂部検出部２１３で修正された頂点リストＶ（ｎ）に無効ではない頂点座標を持つ各肌色領域に対して、第１の実施の形態における上記式（２０）と同様の方法によって肌色領域ｎを囲む長方形領域を算出する。但し、本実施の形態においては、目の高さ候補に対応して複数の顎の高さ候補ｃｈｉｎｌｉｓｔ（ｉ）が存在するため、各顎の高さ候補に対して長方形領域を算出するとともに、その頂点座標を上記式（６）で示したのと同様の方法によって頂点リストＶ’（ｎ，ｉ）に格納する。ここで、ｉは顎の高さ候補に対応する眼の高さ候補の優先順位を表している。算出された新たな頂点リストＶ’（ｎ，ｉ）は、判定部２１９に送られる。なお、肌色領域ｎに対応する頂点リストＶ（ｎ）に格納された頂点座標が無効である場合には、新たな頂点リストＶ’（ｎ，ｉ）の全ての優先順位ｉに対しても無効な座標値を格納することとする。
【０２２７】
判定部２１９は、領域修正部２１８において算出された頂点リストＶ’（ｎ，ｉ）に無効ではない頂点座標を持つ各肌色領域に対して、その肌色領域が顔領域であるか否かの判定を行ない、その判定結果を表す２値のフラグｆａｃｅｆｌａｇを出力するが、その判定の方法が第１の実施の形態とは異なる。第１の実施の形態においては、眼の高さＨＯＥと口の高さＨＯＭとがそれぞれ１つずつ検出されるため、その組み合わせは１通りであり、第１の実施の形態における判定部１９ではこの１つの組み合わせについて顔領域であるか否かの判定を行う。これに対して、本実施の形態において、眼の高さに関して複数の眼の高さ候補が検出されており、眼の高さ候補の数をＮ個とした場合、口の高さＨＯＭとの組み合わせはＮ個存在することになる。判定部２１９は、これら複数個の組み合わせに対して顔領域であるか否かの判定を行い、いずれかの組み合わせにおいて顔領域の条件を満足するものが存在した場合には、対応する肌色領域を顔領域と判定すると共に、その組み合わせにおける眼の高さ候補を真の眼の高さＨＯＥとする。顔領域の条件を満たす複数の組み合わせが存在する場合には、最も優先順位ｉが高い眼の高さ候補をＨＯＥとする。なお、いずれの組み合わせに対しても顔領域の条件が満足されない場合、対応する肌色領域が顔領域ではないと判定する。
【０２２８】
図２１は、本実施の形態の判定部２１９を示すブロック図である。図２１に示すように、判定部２１９は、左右の眼の領域及び口領域を設定する領域設定部２３１と、長方形領域内の肌色画素における水平方向のエッジ（水平エッジ）を算出する水平エッジ検出部２３２と、眼であるか否かを判定する眼パターン判定部２３３と、口であるか否かを判定する口パターン判定部２３４と、口の高さに基づき眼の高さが正確か否かを判定する目口間距離判定部２３６と、眼パターン判定部２３３、口パターン判定部２３４及び目口間距離判定部２３６の判定結果から顔であるか否かを判定する総合判定部２３５とから構成される。
【０２２９】
領域設定部２３１では、肌色領域ｎに対して、口の高さＨＯＭ及び頂点リストＶ’（ｎ，ｉ）から、第１の実施の形態と同様な方法によって口領域が設定され、その結果がＶｍｏｕに格納される。一方、リストｅｙｅｌｉｓｔ（ｎ，ｉ）から眼の高さ候補が１つ選択され、これと頂点リストＶ’（ｎ，ｉ）から、第１の実施の形態と同様な方法によって右眼領域、及び左眼領域が設定され、夫々ＶｅｙｅＲ及びＶｅｙｅＬに格納される。このとき、選択される眼の高さ候補は、この時点までにいまだ顔領域の判定に用いていないものの中で、もっとも優先順位ｉが高いものとする。設定された右眼領域ＶｅｙｅＲ、左眼領域ＶｅｙｅＬ及び口領域Ｖｍｏｕは、眼口間距離判定部６６に送られると共に、ＶｅｙｅＲ、ＶｅｙｅＬは眼パターン判定部２３３へ、Ｖｍｏｕは口判定部２３４へも送られる。
【０２３０】
なお、この領域設定部２３１において設定された右眼領域、左眼領域及び口領域に基づいて顔領域の判定が行われるが、その結果である２値のフラグｆａｃｅｆｌａｇは後述する総合判定部２３５から領域設定部２３１に戻される。この判定結果が顔領域でないことを示す場合には、リストｅｙｅｌｉｓｔ（ｎ，ｉ）に格納された眼の高さ候補のうち、次に優先順位ｉの高い眼の高さ候補が選択されて、右眼領域ＶｅｙｅＲ。左眼領域ＶｅｙｅＬが再設定されると共に。これらを用いた顔領域の判定が繰り返される。
【０２３１】
また、ここで用いた眼の高さ候補がリストｅｙｅｌｉｓｔ（ｎ，ｉ）における最低順位のものである場合には、２値のフラグｌａｓｔｏｆｌｉｓｔに、例えば１の値を設定する。これに対し、判定に用いるべき眼の高さ候補がまだ存在する場合には、フラグｌａｓｔｏｆｌｉｓｔに、例えば０の値を設定する。値の設定されたフラグｌａｓｔｏｆｌｉｓｔは後述の総合判定部２３５に送られる。
【０２３２】
眼口間距離判定部２３６では，図２２に示すように、口の高さＨＯＭに対して領域設定部２３１で選択された眼の高さ候補ｅｙｅｌｉｓｔ（ｎ，ｉ）が適切であるか否かの判定を行う。この判定は、例えば、領域設定部２３１で設定された口領域の幅ｍｏｕｓｉｚｅｘを用いて、下記式（４２）式が満たされるか否かを調べることによって行うことができる。
【０２３３】
【数４２】

【０２３４】
ここで、ｔ１，ｔ２は眼と口との間の垂直方向の距離Ｌｍｅに対する閾値を決めるための係数であり、あらかじめ適切な値が設定されている。
【０２３５】
上記式（４２）が満たされない場合には、口の幅ｍｏｕｓｉｚｅｘに対して、眼と口との間の距離Ｌｍｅが大きすぎるか、又は小さすぎると判断し、眼の高さ候補ｅｙｅｌｉｓｔ（ｎ，ｉ）が不適切であると判定する。これに対し、上記式（４２）が満足される場合には、眼の高さ候補ｅｙｅｌｉｓｔ（ｎ，ｉ）が適切であると判定する。この判定結果は２値のフラグｅｙｅｌｉｐｄｓｔで表され、例えば眼の高さ候補が適切であると判定された場合には１の値が、また、不適切であると判定された場合には０の値が設定される。ここで設定されたフラグｅｙｅｌｉｐｄｓｔは後述の総合判定部２３５に送られる。
【０２３６】
水平エッジ検出部２１２は、第１の実施の形態のものと同一のものとすることができ、ここで生成されたエッジマップＭｅｄｇｅは眼パターン判定部２１３，及び口パターン判定部２１４に送られる。
【０２３７】
眼パターン判定部２３３は、図１２に示した第１の実施の形態における眼パターン判定部２３とほぼ同様の処理を行うが、本実施の形態では、更に眼鏡の有無を判定し、その判定結果に応じて眼のパターン判定を行う点が異なる。
【０２３８】
被写体が眼鏡を装着している場合、図２３に示すように眼鏡フレーム２４２の両眼２つのレンズをつなぐ部分２４２ａが両眼の間に存在するため、眼鏡のデザインによっては、水平エッジ検出部２３２において両眼間の領域に多くのエッジが検出される。ところが、上記式（２７）に示した第１の判定条件は、両眼間に存在する水平エッジが少ないことを仮定しているため、被写体が眼鏡を装着している場合には、その眼のパターンが正しく判定されない可能性が高くなる。
【０２３９】
そこで本実施例では、領域設定部２３１で設定した右眼領域及び左眼領域の周辺部分において、眼鏡の特徴を示すパターンが存在するか否かを調べ、眼鏡の特徴が検出された場合には、上記式（２７）の閾値Ｔｅｄｓｔとして、より小さい値を用いることで、眼鏡を装着した被写体の眼のパターンが正しく判定されるようにする。
【０２４０】
眼鏡の特徴としては、例えば、両眼間に存在する眼鏡のフレームが水平方向に細長く伸びていることを利用し、下記式（４３）に示すような条件によって、その特徴の存在を判定することができる。
【０２４１】
【数４３】

【０２４２】
上記式（４３）の第１の不等式において、ａｒｅａＭは，右眼領域２４３Ｒと左眼領域２４３Ｌとに挟まれた、図２４に斜線で示すような両眼間の領域２４４に存在する画素の総数であり、ｅｃｏｕｎｔＭはこの両眼間領２４４において下記式（４４）のようにカウントされる水平エッジの画素数である。
【０２４３】
【数４４】

【０２４４】
ここでｅｙｅＭは両眼間領域２４４に存在する画素の集合を現す。また、（４３）式の第２の不等式におけるｅｓｄｖｘ，及びｅｓｄｖｙは、両眼間領域２４４における夫々水平エッジ画素の水平座標ｘの標準偏差及び垂直座標ｙの標準偏差を表している。Ｔｇｌｓｄｎｓｔｙ，Ｔｇｌｓａｓｐｃｔは、予め設定された閾値である。
【０２４５】
以上のことより、上記式（４３）式の条件は、両眼間領域２４４における水平エッジの密度が高く、またそれらの水平エッジが水平方向に広がっている場合に、これを眼鏡のフレームによるパターンであるとみなして被写体が眼鏡を装着しているものと判定することを表している。
【０２４６】
口パターン判定部２３４は、図１２に示した第１の実施の形態における口パターン判定部２４とほぼ同様の処理を行うが、本実施の形態では、更に口検出部２３４で算出された口の高さの信頼性ｌｉｐｃｏｎｆに応じて口のパターン判定を行う点が異なる。
【０２４７】
第１の実施の形態の口パターン判定部２４では、上記式（３０）によって口のパターン判定を行うが、その中の第３の判定条件は、口の高さにおいて、口領域Ｖｍｏｕ内部における赤みの強さがその周辺よりも強いという仮定に基づいている。しかしながら、図２５に示すように、赤い服の着用等により赤みの強い領域が口と同じ高さに存在すると、上述の仮定が必ずしも成立するとは限らず、正しい顔領域の判定が行えない場合がある。
【０２４８】
そこで、本実施の形態においては、口の高さを検出する際に算出した口の高さの信頼性ｌｉｐｃｏｎｆが低い場合には、上記式（３０）における第３の判定条件を緩和する。図２５の例のように、赤みの強い服２４５を着用する等、被写体の着用した服によって顔の両側に赤みの強い領域が存在する場合、赤みの強い画素は水平方向に広く分布することになるが、これにより、上記式（３８）によって算出される赤みの強い画素の水平座標ｘの標準偏差は大きな値となるため、口の高さの信頼性ｌｉｐｃｏｎｆが低下する。このとき、上記式（３０）の第３の判定条件における閾値Ｔｃｄｓｔｍを、例えば下記式（４５）に示すようにｌｉｐｃｏｎｆに応じて小さくすることにより、口パターンを正しく判定することができるようになる。
【０２４９】
【数４５】

【０２５０】
総合判定部２３５では、眼口間距離判定部２３６、眼パターン判定部２３３、並びに口パターン判定部２３４から送られてくる２値のフラグｅｙｅｌｉｐｄｓｔ、ｅｙｅｆｌａｇ、及びｍｏｕｆｌａｇがいずれも１である場合、即ち、領域設定部２３１で設定した眼の領域及び口の領域がそれぞれ眼の特徴、口の特徴を備えていると判定された場合、判定結果を表す２値のフラグｆａｃｅｆｌａｇに、例えば１の値を設定して出力するとともに、現在の肌色領域ｎに対する顔領域の判定処理を修了し、次の肌色領域ｎ＋１に対する顔領域の判定を開始する。
【０２５１】
一方，ｅｙｅｌｉｐｄｓｔ、ｅｙｅｆｌａｇ及びｍｏｕｆｌａｇのいずれかのフラグが１でない場合には、フラグｆａｃｅｆｌａｇに例えば０の値を設定する。このとき、領域設定部２３１から送られてくるフラグｌａｓｔｏｆｌｉｓｔを参照し、その値が１である場合、即ち用いた眼の高さ候補がリストｅｙｅｌｉｓｔ（ｎ，ｉ）の最低順位のものである場合には、現在の肌色領域ｎが顔領域ではないと判定し、例えば０に設定されたｆａｃｅｆｌａｇを出力すると共に、現在の肌色領域ｎに対する顔領域の判定処理を修了し、次の肌色領域ｎ＋１に対する顔領域の判定を開始する。
【０２５２】
これに対して、フラグ（制御信号）ｌａｓｔｏｆｌｉｓｔが０である場合には、判定結果であるｆａｃｅｆｌａｇを領域判定部２３１に戻し、次に優先順位ｉが大きい眼の高さ候補を用いて同じ肌色領域ｎに対する顔領域の判定処理を継続する。
【０２５３】
なお、本実施の形態においても、第１の実施の形態と同様、顔や肌色領域の連結の問題を回避するために、上記式（２１）、（２３）において領域の大きさを決める係数Ｓ１，Ｓ２，Ｓ３を複数用意しておき、それぞれの値に対して判定を行なうようにすることが可能なことは言うまでもない。
【０２５４】
本実施の形態においては、カラー画像内に頭頂部が含まれていない場合、適切に人物の顔領域が撮影されていないとして、撮影のしなおしを促すことが可能となる。
【０２５５】
また、顔を左右に分割する中心線を、肌色領域を構成する画素の空間的な分布から算出すると共に、口の高さを中心線周辺における赤みの強さから検出するため、赤みの強い服を着用している場合にも、正確に口の高さ及び中心線を検出することができ、検出された口の高さの信頼度を算出し、この信頼度が高い場合にはロバスト推定によって中心線位置の修正を行うため、肌色領域が影などの影響で欠落している場合にも、正確に中心線を検出することが可能となる。
【０２５６】
更に、口の高さの信頼度を考慮することにより、より正確に口パターンらしさを評価することができ、また、複数の眼の高さの候補を算出し、各眼の高さ候補に対して眼のパターンらしさの判定を行うと共に、眼鏡装着の有無の判定を行うことにより、眼鏡装着の場合にもより正確に眼のパターンらしさを判定することができ、眼の高さの検出における誤検出を軽減することが可能となる。
【０２５７】
更にまた、口の幅に対する眼と口との間の距離を評価し、口のパターンらしさ及び眼のパターンらしさに反映させるため、より正確に顔領域の判定を行うことが可能となる。
【０２５８】
なお、本実施の形態における画像処理装置を上述した第２及び第３の実施の形態における肖像画像撮影装置に適用できることは言うまでもない。
【０２５９】
【発明の効果】
以上詳細に説明したように本発明に係る画像処理方法によれば、入力されたカラー画像から肌色の領域を抽出する肌色領域抽出工程と、上記肌色領域に基づき上記人物の頭頂部、上記人物の口、上記人物の眼、及び上記人物の顎からなる群から選択される１以上の特徴点を検出する特徴点検出工程と、上記特徴点に基づき、上記肌色領域が顔領域であるか否かを判定する判定工程とを有して顔領域を抽出するので、抽出した肌色領域に基づき顔の特徴点を検出し、この特徴点に基づき肌色領域が顔であるか否かを判定することができ、誤検出を低減し、入力カラー画像から高速且つ正確に顔領域を抽出することが可能となると共に、特徴点に基づき顔か否かを判定するため、正確な判定をすることができる。
【０２６０】
また、本発明に係る肖像写真撮影装置によれば、人物を撮影して肖像写真の作成を行なう肖像写真撮影装置において、入力カラー画像から肌色の領域を抽出する肌色領域抽出手段と、上記肌色領域に基づき上記人物の頭頂部、上記人物の口、上記人物の眼、及び上記人物の顎からなる群から選択される１以上の特徴点を検出する特徴点検出手段と、上記特徴点から、上記肌色領域が顔領域であるか否かを判定する判定手段とを具備し、顔領域を抽出する顔領域抽出手段と、上記顔領域抽出手段によって抽出された顔領域から、トリミングの対象となる顔領域を選択する選択手段と、上記選択された顔の領域からトリミングの位置決めを行なう位置決め手段とを有し、上記カラー画像から上記人物の顔領域を抽出し該顔領域が所定の位置となるように自動的にトリミングを行なう自動トリミング手段を有するので、顔領域抽出手段により、入力カラー画像から高速且つ正確に顔領域を抽出し、正確に顔領域を顔であるか否かを判定することができ、顔領域抽出手段より検出された肌色領域及び各特徴点の情報を使用してトリミング処理をするため、正確に自動トリミング処理を行なうことが可能であり、これにより、良好な肖像写真を自動で撮影することが可能となる。
【図面の簡単な説明】
【図１】本発明の第１の実施の形態における画像処理装置１０を示すブロック図である。
【図２】抽出された肌色画素を示す模式図である。
【図３】肌色領域を囲む閉領域を示す模式図である。
【図４】長方形領域の頂点座標を示す模式図である。
【図５】カラー画像における人物の頭頂部を検索する際の検索範囲を示す模式図である。
【図６】長方形領域の水平方向の赤み強度が累積されて生成されたヒストグラムＨｒｄｓｈと長方形領域との関係を示す模式図である。
【図７】水平エッジの強度を算出する際に使用される微分フィルタの微分係数を示す図である。
【図８】人物の眼、口及び顎の位置の関係を示す模式図である。
【図９】エッジを構成する画素が水平方向に累積されて生成されたヒストグラムＨｅｄｇｅ（ｙ）と肌色領域７１に対応する長方形領域７２との関係を示す模式図である。
【図１０】肌色領域に対応する長方形領域における口の高さＨＯＭ及び検索範囲ｍｔｏｐ、ｍｂｔｍを示す模式図である。
【図１１】修正後の長方形領域の頂点座標｛（ｓｔｘ、ｓｔｙ），（ｅｄｘ、ｅｄｙ）｝を示す模式図である。
【図１２】本発明の第１の実施の形態における画像処理装置の判定部を示すブロック図である。
【図１３】口の高さＨＯＭ、及び眼の高さＨＯＥから推定され、頂点座標Ｖ（ｎ）が示す長方形領域内に設定された右眼、左眼及び口の領域を示す模式図である。
【図１４】垂直エッジ強度を算出する際に使用する微分フィルタの微分係数を示す図である
【図１５】本発明の第２の実施の形態の肖像画像撮影装置を示すブロック図である。
【図１６】入力カラー画像から切り出す部分画像の切り出し位置を示す模式図である
【図１７】同一の入力カラー画像１２０において、複数の人物１２１，１２２が存在し、この人物１２１，１２２が水平方向に重なっている場合の頭頂部検索範囲を示す模式図である。
【図１８】本発明の第４の実施の形態における画像処理装置を示すブロック図である。
【図１９】長方形領域における赤み強さの累積範囲を示す模式図である。
【図２０】口の高さの信頼性を算出するために設定される長方形領域Ｒｍを示す模式図である。
【図２１】本発明の第４の実施の形態における画像処理装置の判定部を詳細に示すブロック図である。
【図２２】口の高さＨＯＭに対して眼の高さ候補ｅｙｅｌｉｓｔ（ｎ，ｉ）が適切であるか否かの判定を行う眼口間距離判定部における判定方法を説明するための模式図である。
【図２３】眼鏡を装着している被写体を示す模式図である。
【図２４】右眼領域と左眼領域とに挟まれた両眼間の領域を示す模式図である。
【図２５】赤い服の着用等により赤みの強い領域が口と同じ高さに存在する場合を示す模式図である。
【符号の説明】
１０，２１０　画像処理装置、１１，２１１　画像入力部、１２，２１２　肌色領域抽出部、１３，２１３　頭頂部検出部、１４，２１４　口検出部、１５，２１５　眼検出部、１６，２１６　顎検出部、１７，２１７　中心線検出部、１８，２１８　領域修正部、１９，２１９　判定部、２１，２３１　領域設定部、２２，２３２　水平エッジ検出部、２３，２３３　眼パターン判定部、２４，２３４　口パターン判定部、２５，２３５　総合判定部、３０　肖像画像撮像装置、３１　撮像器、３２　顔検出器、３３　領域選択器、３４　トリミング器、２２０　中心線修正部、　２３６　目口間距離判定部[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention extracts, for example, a face area of a person from an image input from a video camera, a still camera, a surveillance camera, a TV phone, an image database, a printer, and the like, and performs tracking of the person, parameter control, image quality correction, or search. The present invention relates to an image processing method, an image processing apparatus, a program for executing image processing, a recording medium on which the program is recorded, an automatic trimming apparatus and a portrait photographing apparatus using the image processing.
[0002]
[Prior art]
Methods for extracting a person's face region from an image can be roughly classified into two methods. The first method uses information on a color, which is one of the salient features of the face. A range corresponding to a skin color is set in advance in some color space, and is included in the range. The face area is extracted as a set of pixels having colors. This method is widely used because its processing is simple and the color itself does not change much even if the direction or size of the face changes.However, there is a possibility that objects other than the face have the same color. It cannot be excluded and it is essentially impossible to avoid overdetection. Therefore, many methods have been proposed in which the extracted region is regarded as a candidate for the region to be finally extracted, and in the subsequent stage, the determination is made in more detail using the region shape, the light and shade pattern in the region, and the like. In this case, even if an area that does not correspond to a face is extracted by color information, it can be removed by other information, and more accurate extraction processing can be performed (conventional example 1).
[0003]
On the other hand, the second method is to prepare a template representing the features of the face in advance and perform pattern matching at all positions on the input image. Is determined. This method has the advantage of being less affected by changes in face color due to differences in lighting conditions, race, etc., by using the features of the facial shading pattern. If a good template can be generated by appropriate learning using a large amount of data, highly accurate extraction can be expected (Conventional Example 2).
[0004]
By the way, in various types of illumination photographs, there are cases where there is a restriction on the size and position of the face region. After photographing in a size larger than originally required, trimming is performed so that the face region becomes a predetermined size. ing.
[0005]
For example, Patent Literature 1 below discloses an image processing apparatus that processes a portrait image in which an upper body is photographed on a background of a uniform color so that a person's head is located at a predetermined position in a frame of a predetermined size. (Hereinafter referred to as Conventional Example 3).
[0006]
In the image processing apparatus according to the third conventional example, first, a frame of the size of a face photograph printed on a card with an ID photo is attached so that the head of a person in the portrait image is accommodated in the read portrait image. Is set by the operator, then the left end, right end, and top of the head of the person in the frame are detected, the position of the frame is corrected based on the detection result, and finally the background outside the frame is deleted. The trimming is performed.
[0007]
Further, Patent Document 2 below discloses a foreground image extraction method for extracting a foreground image by deleting a background image from one image (hereinafter, referred to as Conventional Example 4).
[0008]
In the foreground image extraction method described in Conventional Example 4, first, a background image in which no object is present is photographed, and a target image in which the object is positioned in the foreground in the same frame as the background image is photographed. And a brightness difference in a predetermined common area between the image and the target image. Then, a difference image is obtained from the background image and the target image, and the difference image is corrected based on the difference between the common areas to obtain a first mask image. Next, the first mask image is subjected to brightness expansion / reduction processing and color expansion / reduction processing, and further, a boundary between the object and the background is determined, and the area is divided at this boundary. The generated second mask image is generated. Finally, the target image and the second mask image are superimposed to generate an image in which the background of the target image is converted to a single color.
[Patent Document 1]
JP-A-2002-42116
[Patent Document 2]
JP 2000-36032 A
[0009]
[Problems to be solved by the invention]
By the way, in the region extraction based on the color, if the range of the color to be extracted is set to be wide to some extent in order to cope with the individual difference and the change of the illumination condition, the overdetection of extracting the unnecessary region increases. On the other hand, when the range of colors is reduced to suppress overdetection, there is a problem in that detection omissions in which a desired region is not extracted increase due to the effects of individual differences and changes in lighting conditions and the like.
[0010]
However, assuming that some discrimination processing is provided in the subsequent stage as in Conventional Example 1, detection omission in the preprocessing becomes a serious problem, and it is considered that overdetection can be tolerated. There is a possibility that the determined area may significantly degrade the performance of the determination processing. For example, if adjacent different objects have similar colors, regions corresponding to each object may be connected on the image and extracted as one region, but in this case, the region shape and the inside of the region are no longer used. There is a problem that it is difficult to make a correct discrimination because the shading pattern of does not reflect the properties of the corresponding object.
[0011]
On the other hand, as in Conventional Example 2, there is a method of scanning the extracted area and searching for a specific shape and a light and shade pattern. However, size normalization processing and the like are complicated, and discrimination processing is not performed. There is a problem that the amount of calculation required becomes large.
[0012]
Further, in the technique described in the conventional example 3, after a large number of images are captured, trimming is performed collectively, but at least the first image requires the intervention of an operator to determine a frame position. I need. As described above, the shooting of various types of illumination photographs satisfies the requirements for the position and size of the face region, and thus requires manual intervention in the shooting and post-processing, which is complicated. .
[0013]
Furthermore, in the technique described in the third conventional example, the automatic processing is performed for the second and subsequent images without setting the frame again by using the setting conditions of the first frame. However, the head in the portrait image to be processed is not always present inside the frame position set in the first image, and the position of the head is restricted at the time of shooting, or the frame is set at the time of trimming processing. Need to be repeated. Furthermore, the left end of the head, the right end of the head, and the top of the head are detected as positions where the luminance information has significantly changed. However, such a detection method is easily affected by the background luminance. Even if it is assumed to be uniform, there is a problem that shadows and unevenness in luminance of the background actually occur due to the proof conditions and the like, which may cause an erroneous position to be detected.
[0014]
Further, in the technique described in the conventional example 4, since the subject is extracted based on the difference between the background image and the target image, it is necessary to capture an image of only the background in advance. Further, since the difference from the background image is obtained, the difference may be affected by a change in the background color of the target image. For this reason, in the technique described in Conventional Example 4, although the background color of the background image is corrected, it is difficult to set a threshold value for correcting the background image, and the correction cannot always be performed accurately. Further, according to the general method described in Conventional Example 4, it is not possible to correct a local change such as a shadow of a subject itself occurring in a background portion depending on an illumination condition.
[0015]
Furthermore, in the technique described in Conventional Example 4, the center line of the face image is obtained by the horizontal barycentric arrangement of the skin color image. However, when the face has a horizontal shadow, the dark portion is extracted as a skin color pixel. Instead, the center line may shift significantly. In addition, since the lowermost part of the skin color closed area is defined as the position of the chin, when the face area is a closed area connected to the neck area, there is a problem that the position of the chin is greatly shifted. In order to avoid this problem, the range of the skin color to be extracted is changed depending on the brightness, but this is also a threshold problem, and there is a problem that the face and neck are not always separated.
[0016]
Further, in the above-described conventional examples 1 to 4, there is a problem that appropriate trimming cannot be performed on an image in which a plurality of persons exist, that is, an image in which a plurality of face regions (skin color regions) exist.
[0017]
The present invention has been proposed in view of such circumstances, and a combination of a face extraction process based on skin color and a simple determination process enables a fast and accurate face region extraction even when a plurality of face regions exist. And apparatus for extracting images, a recording medium on which a program for executing them by a computer is recorded, an automatic trimming device for automatically trimming a face area extracted by these processes, and a portrait image having the function of this automatic trimming It is an object to provide an imaging device.
[0018]
[Means for Solving the Problems]
In order to achieve the above-described object, an image processing method according to the present invention includes, in an image processing method for extracting a face region of a person from an input color image, a skin color region extraction step of extracting a skin color region; A feature point detecting step of detecting a position of one or more feature points selected from the group consisting of the top of the person, the mouth of the person, the eyes of the person, and the chin of the person based on the skin color region; A determining step of determining whether or not the skin color area is a face area based on the positions of the skin color area and the feature points.
[0019]
In the present invention, a skin color region is extracted from an input image, a position of a feature point of the face is detected based on the skin color region, and it is determined whether the skin color region is a face based on the position of the feature point. Even when a plurality of skin color regions are extracted, it is possible to determine whether or not the skin color regions are faces, thereby reducing erroneous detection and extracting a face region with high accuracy.
[0020]
Further, the feature point detecting step includes a step of detecting vertical coordinates as a position of the mouth based on a redness intensity distribution indicating a horizontal redness intensity in the closed region including the skin color region, for example, The position of the mouth in the vertical direction where the average value of the redness intensity is the largest can be determined as the position of the mouth, and the position of the mouth is detected based on the distribution of the redness intensity indicating the intensity of redness. It can be detected well.
[0021]
Further, the redness intensity is a value obtained by integrating the ratio of R to G and the ratio of B to G when G is larger than B, where R, G, and B are the values of the three primary colors in each pixel, When G is smaller than B, a value obtained by integrating the ratio of R to B and the ratio of G to B can be used.
[0022]
Furthermore, in the mouth detection step, the position of the mouth can be detected using only the redness intensity of a pixel that is present in the closed region and does not belong to the skin color region, and the region serving as the mouth is Since the position of the mouth is detected by utilizing the fact that the mouth does not belong to the skin color region, the accuracy of the position detection of the mouth can be further improved.
[0023]
In the mouth detection step, the redness intensity is weighted differently depending on the position and / or brightness of a pixel in the closed area, and the vertical coordinate position of the position having the largest horizontal weighted average value is determined. This can be detected as the position of the mouth, whereby the detection accuracy can be improved when the position and luminance information of the mouth area can be grasped in advance.
[0024]
Further, the feature point detecting step includes a vertex detecting step of detecting a vertical coordinate position at the vertex of the person, and a vertical coordinate as a mouth position based on a horizontal redness intensity distribution in the closed region. The eye detection step sets an eye search range based on the position of the mouth and the position of the crown, and detects the position of the eye from within the search range Since the search position is set and the position of the eye is detected, the detection accuracy is extremely high.
[0025]
Furthermore, in the above-mentioned crown detection step, when the background of the person is a single color, the area above the flesh color area is scanned from the top of the input color image to the bottom, and pixels having a color different from the background color are scanned. The appearance position can be detected as the position of the top of the head, and further, in the scanning from the top to the bottom of the input color image, the background color updated as needed and the color of each pixel can be compared, As a result, the top of the head can be accurately detected.
[0026]
Further, in the top detection step, an image of only the background that does not include the person, and a difference for each pixel between the image that includes the person is calculated, and among the pixels whose difference is equal to or greater than a preset threshold, The coordinates of the uppermost pixel may be detected as the position of the top of the head, and the top of the head can be detected with higher accuracy by using the difference image.
[0027]
Further, the feature point detecting step includes a head detecting step of detecting a vertical coordinate position at the head of the person, wherein the head detecting step is such that the head is present in the input color image. A top part determination step of determining whether the skin color area is a face area based on the top part determination result.In the top part determination step, The variance of pixel values in a predetermined number of scan lines in the upper region from the top of the input color image to the bottom is calculated, and if this variance is greater than a predetermined threshold, the top of the input It can be determined that the image does not exist in the color image, and if the image does not include the top of the head, it is possible to determine that the face area of the person has not been properly imaged and to prompt the user to perform image shooting again.
[0028]
Still further, the feature point detecting step may include a center line detecting step of detecting a position of a center line dividing the face of the person into right and left based on the position of the mouth. In the coordinates near the position of the mouth, the center of gravity of the pixel with the large redness intensity can be calculated and the position of the center of gravity in the horizontal direction can be detected, or the center of gravity can be calculated using robust estimation. The face center line can be detected with extremely high accuracy.
[0029]
In the center line detecting step, a barycentric position of a pixel which does not belong to the flesh color region in the coordinates near the position of the mouth and has a large redness intensity is calculated, and a horizontal position coordinate of the barycentric position is detected. It is preferable that, if the face center line is determined by calculating the center of gravity of a large pixel that does not belong to the skin color area and the redness intensity is equal to or more than a predetermined threshold, the pixel belonging to the skin color area may be affected. In addition, the face center line can be detected with higher accuracy.
[0030]
Further, the feature point detecting step includes a center line detecting step of detecting a position of a center line that divides a face into right and left from a spatial distribution of pixels forming the skin color region. The position of the mouth can be detected by weighting the redness intensity in accordance with the distance from the center line, and the position and center line of the mouth can be accurately determined even when wearing reddish clothing. Can be detected.
[0031]
Furthermore, in the mouth detecting step, the reliability of the position of the mouth is calculated from the spatial distribution of the redness intensity of the pixel in the vicinity of the detected position of the mouth, and the reliability is equal to or more than a predetermined threshold. In such a case, the position of the center line can be corrected by, for example, robust estimation, whereby the center line can be accurately detected even when the skin color region is missing due to the influence of a shadow or the like.
[0032]
Further, in the determination step, the pattern likeness of the eye is calculated from the distribution state of the horizontal edge near the position of the eye, and the distribution of the edge in the horizontal direction and the distribution state of the color near the position of the mouth are calculated. The pattern-likeness is calculated, and when the pattern-likeness of the eye and the pattern-likeness of the mouth are equal to or greater than a predetermined threshold, the skin color region can be determined to be a face region.
[0033]
Here, the pattern likeness of the eye sets a right eye region and a left eye region in the vicinity of the position of the eye, and a pixel constituting a horizontal edge in the right eye region and the left eye region is the position of the eye. In the case where the number of pixels constituting the horizontal edges in the area other than the right eye area and the left eye area in the vicinity is greater than the density of the pixels constituting the horizontal edges in the right eye area and the left eye area, Large, or large when the density of pixels forming horizontal edges in the interocular region between the right eye region and the left eye region, and between the right eye region and the left eye region is large, Can be determined.
[0034]
Further, the determining step includes a spectacles detecting step of detecting the presence or absence of spectacles, it is possible to evaluate the likeness of the eye pattern according to the spectacles detection result, by determining the presence or absence of spectacles, Even in the case of wearing, it is possible to more accurately determine the likeness of the eye pattern.
[0035]
Further, in the eye detection step, one or more eye position candidates ranked as the eye positions are detected, and in the determination step, the one or more eye position candidates are ranked in descending order of eye position. Pattern likeness can be calculated, multiple eye position candidates are calculated, and the eye pattern likeness is evaluated for each candidate, thus reducing false detection in eye position detection. It becomes.
[0036]
Further, the likeness of the pattern of the mouth is such that a mouth region is set in the vicinity of the position of the mouth, and a pixel constituting a horizontal edge in the mouth region is a horizontal pixel in an area other than the mouth region in the vicinity of the position of the mouth. Direction, the number of pixels constituting the horizontal edge in the mouth region is large, the redness intensity in the mouth region is large, or the mouth region Is larger than the redness intensity in an area other than the mouth area.
[0037]
Further, in the mouth detecting step, the reliability of the position of the mouth is calculated from the spatial distribution of the redness intensity of the pixel near the detected position of the mouth, and in the determining step, the reliability of the position of the mouth is calculated. Since the likelihood of the mouth pattern is evaluated according to the degree, the likelihood of the mouth pattern can be more accurately evaluated by considering the reliability of the position of the mouth.
[0038]
Furthermore, in the determination step, when the distance between the position of the eye and the position of the mouth is larger than an upper limit calculated from the width of the mouth region, or calculated from the width of the mouth region. If smaller than the lower limit, it can be evaluated that the pattern likeness of the eyes and the pattern likeness of the mouth are small, the distance between the eyes and the mouth with respect to the width of the mouth is evaluated, and the pattern likeness of the mouth and the pattern of the eyes In order to reflect on the likelihood, it is possible to more accurately determine the face area.
[0039]
Further, a plurality of right eye regions, left eye regions, and mouth regions having different sizes are set near the eye position and near the mouth position, respectively, and the eye pattern and the mouth pattern are calculated. By using a plurality of regions corresponding to each feature point, a highly reliable determination result can be obtained.
[0040]
Further, in the determination step, when it is determined that the plurality of skin color areas are face areas, the image processing apparatus may include a selection step of selecting one or more face areas based on, for example, positions of the plurality of face areas. For example, one face area can be selected and extracted from a plurality of face areas according to the purpose, and a specific face can be extracted from an image having a plurality of face areas, that is, an image in which a plurality of persons exist. Extraction and trimming can be performed.
[0041]
An image processing apparatus according to the present invention is an image processing apparatus for extracting a face area of a person from an input color image, wherein a skin color area extracting unit for extracting a skin color area, Feature point detecting means for detecting the position of one or more feature points selected from the group consisting of a crown, the mouth of the person, the eyes of the person, and the chin of the person; and the positions of the skin color region and the feature points Determining means for determining whether or not the skin color area is a face area based on the following.
[0042]
A program according to the present invention is a program for executing an operation of extracting a face region of a person from an input color image, wherein a skin color region extracting step of extracting a skin color region, and a step of extracting the person based on the skin color region. A feature point detecting step of detecting a position of one or more feature points selected from the group consisting of a crown, the mouth of the person, the eye of the person, and the chin of the person; and a position of the skin color region and the feature point. A determination step of determining whether or not the skin color area is a face area based on
[0043]
A recording medium according to the present invention is a computer-readable recording medium that records a program for causing a computer to execute an operation of extracting a face area of a person from an input color image. An extraction step, and feature point detection for detecting a position of one or more feature points selected from the group consisting of the top of the person, the mouth of the person, the eyes of the person, and the chin of the person based on the skin color region And a determining step of determining whether or not the skin color area is a face area based on the positions of the skin color area and the feature points.
[0044]
An automatic trimming apparatus according to the present invention is an automatic trimming apparatus for extracting a face area of a person from an input color image and trimming the face area so that the face area is located at a predetermined position. Extracting means for detecting a position of one or more characteristic points selected from the group consisting of a top of the person, a mouth of the person, an eye of the person, and a chin of the person based on the skin color region; Means, determining means for determining whether the skin color area is a face area based on the positions of the skin color area and the feature points, and a closed area including the skin color area determined to be a face area by the determining means A face area extracting means comprising: extracting means for extracting a face area as a face area; selecting means for selecting a face area to be trimmed from the face areas extracted by the face area extracting means; Characterized in that from the region of-option face and a positioning means for positioning of the trimming.
[0045]
In the present invention, since the face area is detected by the face area extracting means and the position of the feature point in the person's face is detected, trimming is performed using these data, so that automatic processing is possible. In addition, the trimming process can be performed very accurately.
[0046]
Further, the face area extraction means detects a position of a face center line dividing the face of the person into right and left based on the positions of the feature points, and the positioning means detects a top of the person, a mouth of the person, Positioning of trimming can be performed based on the position of one or more feature points selected from the group consisting of the eye of the person and the chin of the person and the center line of the face. Can be.
[0047]
A portrait photographing apparatus according to the present invention is a portrait photographing apparatus for photographing a person to create a portrait photograph, the skin color region extracting means for extracting a skin color region from an input color image, and the person based on the skin color region. Feature point detection means for detecting the position of one or more feature points selected from the group consisting of the top of the head, the mouth of the person, the eyes of the person, and the chin of the person; A determination unit configured to determine whether the skin color region is a face region based on a position; and an extraction unit configured to extract a closed region including the skin color region determined to be a face region by the determination unit as a face region. A face area extracting unit, a selecting unit for selecting a face area to be trimmed from the face area extracted by the face area extracting unit, and a trimming positioning from the selected face area. And a positioning means, said pigment region extracting a face region of the person from the color image is characterized by having an automatic trimming means for automatically trimmed to a predetermined position.
[0048]
According to the present invention, the face area extracting unit can quickly and accurately extract the face area from the input color image, and can accurately determine whether the face area is a face. Since the trimming process is performed using the obtained skin color region and the position information of each feature point, it is possible to accurately perform the automatic trimming process, thereby enabling a good portrait photograph to be automatically taken. Become.
[0049]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.
[0050]
First embodiment
This embodiment is applied to an image processing apparatus for extracting a face area of a person from an input color image. FIG. 1 is a block diagram showing an image processing apparatus 10 according to the first embodiment of the present invention.
[0051]
As shown in FIG. 1, the image processing apparatus 10 receives a color image from an electronic still camera, a video camera, a scanner, or the like, and outputs the color image as digital data (hereinafter, simply referred to as a color image). An image input unit 11, a skin color region extraction unit 12 for extracting a skin color region from the color image, a top detection unit 13 that receives a color image and a skin color region and detects a top of a person, a skin color region and a color image Is input, a mouth detection unit 14 that detects the mouth of a person, a color image, a skin color region, and data of a crown and a mouth are input, and an eye detection unit 15 that detects the eyes of a person, and A jaw detecting unit 16 for calculating a jaw position of a person, a center line detecting unit 17 for receiving a color image and mouth and eye data to detect a center line of a person's face, The center line data is input and the area correction unit 18 that corrects the face area, the color image, the skin color area, the eyes, the mouth, and the correction data from the area correction unit 18 are input, and the extracted skin color area is And a determination unit 19 for determining whether or not the face is a person's face.
[0052]
The color image output from the image input unit 11 is sent to the skin color region extraction unit 12, the crown detection unit 13, the mouth detection unit 14, the eye detection unit 15, the center line detection unit 16, and the determination unit 19. Although there are various methods of expressing colors in a color image output from the image input unit 11, in the present embodiment, each pixel is represented by three primary colors of R, G, and B. It is assumed that When the value at the position (x, y) on the image is represented, it is represented as R (x, y), G (x, y), B (x, y). Further, in the position (coordinates) (x, y), x indicates the horizontal coordinates of the input color image, and y indicates the vertical coordinates of the input color image.
[0053]
Hereinafter, each component of the image processing apparatus according to the present embodiment will be described in detail.
[0054]
(1) Skin color extraction
The skin color region extraction unit 12 extracts a skin color region as a set of pixels having a skin color. Any existing method may be used as a method for extracting the skin color region. For example, on a two-dimensional plane having r and g as two coordinate axes as shown in the following equation (1), a skin color range to be extracted is set in advance, and a pixel having a color included in this skin color range is set as a skin color pixel. Can be extracted.
[0055]
(Equation 1)

[0056]
Here, R, G, and B indicate the values of the three primary colors of each pixel. The range of the skin color to be extracted can be set, for example, by the following equation (2) that defines the lower limit values rmin and gmin and the upper limit values rmax and gmax for the respective values of r and g.
[0057]
(Equation 2)

[0058]
As a result, a skin color region is extracted as a set of pixels satisfying the above expression (2). However, as shown in FIG. 2, when the extracted flesh-colored pixels 41 form a plurality of lumps such as the lumps 42 and 43 on the image 40, for each of the pixel lumps 42 and 43, The area 40 is divided and each chunk is distinguished as a different area. For the division of the image 40, for example, a set of connected pixels is formed into one lump, and a labeling process or the like for extracting a region corresponding to the lump can be used.
[0059]
Each extracted skin color region is assigned a unique number, and is represented as a multi-valued image on a region map A (x, y) which is a two-dimensional array. That is, when the color of the coordinates (x, y) is included in the flesh-colored area n, the area map A is represented by the following equation (3).
[0060]
(Equation 3)

[0061]
Further, a closed region surrounding each skin color region is calculated. As the closed region, for example, as shown in FIG. 3, rectangular regions 53 and 54 surrounding the skin color regions 51 and 52, respectively, can be used. As shown in FIG. 4, the rectangular area is indicated by two vertex coordinates facing each other on a diagonal line. The vertex coordinates {(stx, sty), (edx, edy)} corresponding to one skin color area are, for example, , Can be calculated as follows. That is, first, the average value xave of the x coordinate, the average value yave of the y coordinate, the standard deviation xsdv of the x coordinate, and the standard deviation ysdv of the y coordinate of the pixels included in the skin color region are calculated by the following equation (4).
[0062]
(Equation 4)

[0063]
Here, n is a number shown in the above equation (3) for identifying the skin color area of interest at present. Using these values, the vertex coordinates of the rectangular area can be calculated by the following equation (5).
[0064]
(Equation 5)

[0065]
Here, a is a preset coefficient. The vertex coordinates of the rectangular area calculated for the skin color area n are stored in the vertex list V, which is a one-dimensional array, as the following equation (6).
[0066]
(Equation 6)

[0067]
The area map A representing the skin color area is sent to the mouth detection unit 14 and the determination unit 19, and the vertex list V is sent to the top detection unit 13, the mouth detection unit 14, and the eye detection unit 15.
[0068]
(2) Feature point detection
The feature point detection unit assumes that each skin color region extracted by the skin color region extraction unit 12 is a face region, and detects each feature point based on the rectangular region indicated by the vertex coordinates V (n) corresponding to this skin color region. You. The feature point detection unit includes a top detection unit 13 that detects the position of the top of the person, a mouth detection unit 14 that detects the position of the mouth of the person based on the intensity of redness in the skin color region, An eye detection unit 15 that sets a search range based on the position of the mouth to detect the eyes, a jaw detection unit 16 that calculates the position of the jaw from the positions of the eyes and the mouth, and sets a mouth area from the position of the mouth. A center line detector 17 for detecting the center line of the face based on the intensity of redness in the mouth region; and vertex coordinates V (n) calculated by the skin color extractor 12 from the positions of the top, chin and face center line. And an area correction unit 18 for correcting Hereinafter, each detection unit will be described in more detail.
[0069]
(2-1) Detection of the top of the person
The crown detector 13 detects the crown of a person having a skin color area as a face. The top of the head is detected, for example, assuming that the background region other than the person is a single color and that only the background region can exist above the person, that is, on the side where the vertical coordinate is small, and a color different from the background color Is detected at the position where the vertical coordinate is the smallest among the pixels having. Hereinafter, the vertical coordinate at the position of the crown is referred to as the height of the crown.
[0070]
Specifically, as shown in FIG. 5, in the input color image 60 sent from the image input unit 11, a region above the rectangular region 62 corresponding to the skin color region 61 of interest, that is, the rectangular region 62 An area where the vertical coordinate is small, and V (n). stx ≦ horizontal coordinate (x coordinate) ≦ V (n). The top search range 63 set in the range of edx is scanned from above in the figure, and the difference d between the value of each pixel and the background color of the background area 64 is calculated by the following equation (7).
[0071]
(Equation 7)

[0072]
Here, R (x, y), G (x, y), and B (x, y) are the values of R, G, and B of the pixel at the coordinates (x, y) on the color image, and Rbg, Gbg , Bbg are the values of R, G, B of the background color. As the background color, for example, as shown in the following equation (8), an average value of pixels above the current pixel of interest, that is, in a region where the vertical coordinate (y coordinate) is small, can be used.
[0073]
(Equation 8)

[0074]
Here, V (n) is the vertex coordinates of the rectangular area corresponding to the skin color area n obtained by the skin color area extraction unit 12, and therefore, the above equation (8) expresses the current attention within the search range 63. It indicates that the average value of m lines above the pixel coordinate position (x, y) is used as the background color. In this case, it is impossible to calculate the background color by the above equation (8) for m lines from the top of the image 60. That is, when the y coordinate of the uppermost part of the image 60 is y = y0, the background used to calculate the background is from the uppermost part (y = y0) to y = y1 = y0 + m. The y coordinate of the minimum pixel of interest using the background of y0 ≦ y ≦ y1 = y2 = y1 + 1 = y0 + (m + 1), and the (m + 1) th line from the top (y = y0). In this case, a method of calculating the background color using only available lines in the same manner as in the above equation (8) or starting the search range 63 from the (m + 1) th line from the top of the image is used. Can be.
[0075]
With respect to the background color sequentially updated by the above equation (8), the color difference d of the above equation (7) is calculated, and when a pixel whose value is larger than a predetermined threshold T appears, its vertical coordinate y Is the height TOH of the crown. The detected head height TOH is sent to the eye detection unit 15 and the area correction unit 18.
[0076]
The threshold value for the difference d between the value of each pixel in the search range and the background color may be a fixed value set in advance, but may be adaptively changed as described below. That is, for example, according to the update of the background color, the threshold value T can be calculated from the variance of the pixel values in the m lines above the target pixel (x, y) by the following equation (9). As described above, by changing the threshold value for the difference d according to the update of the background color, it is possible to detect the top of the head more accurately.
[0077]
(Equation 9)

[0078]
Here, w is a preset coefficient. Also, var (R (x, y)), var (G (x, y)) and var (B (x, y)) are the variances of the values of R, G, and B, respectively, and are calculated by the following equation (10). Is done.
[0079]
(Equation 10)

[0080]
(2-2) Mouth detection of person
Next, the mouth detection unit 14 detects the position (height) of the mouth for each skin color area extracted by the skin color area extraction unit 12. First, in the rectangular area represented by the vertex list V (n), for each pixel (x, y) not extracted as a skin color area, the value rdsh of the following equation (11) indicating the intensity of reddishness (X, y) is calculated.
[0081]
[Equation 11]

[0082]
Here, R (x, y), G (x, y), and B (x, y) indicate respective values at the coordinates (x, y) of the color image. The calculated values rdsh (x, y) are accumulated in the horizontal direction (x-axis direction) as shown in FIG. 6 to generate a histogram Hrdsh (y) represented by the following equation (12).
[0083]
(Equation 12)

[0084]
Here, V (n) and A (x, y) are data sent from the skin color area extraction unit 12, and indicate the vertex coordinates of the rectangular area corresponding to the skin color area n and the area map, respectively. .
[0085]
Next, the histogram Hrdsh (y) is smoothed by a one-dimensional low-pass filter as necessary in order to remove noise and the like, and then the vertical coordinate y at the maximum value of the histogram Hrdsh (y) is set to the mouth height HOM. Is detected as The detected mouth height MOU is sent to the eye detection unit 15, the jaw detection unit 16, the center line detection unit 17, and the determination unit 19.
[0086]
In order to more accurately detect the height of the mouth, when calculating the redness intensity rdsh, weighting may be performed according to the position of the target pixel, the brightness of the target pixel, and the like. For example, in a rectangular area represented by vertex coordinates V (n) in the vertex list V, if it is expected in advance that there is a high possibility that a mouth appears at the center position in the horizontal direction, the x coordinate of the pixel of interest is A coefficient closer to (V (n) .stx + V (n) .edx) / 2 is integrated with the value rdsh calculated by the above equation (11), and the result can be accumulated by the above equation (12). . If an appropriate range can be limited as the brightness of the mouth area, a larger coefficient is added to the value rdsh calculated by the above equation (11) as the brightness of the target pixel is closer to the range, and as a result, Can be accumulated by the above equation (12).
[0087]
(2-3) Human Eye Detection
Next, the eye detection unit 15 detects the position (height) of the eye for each skin color region extracted by the skin color region extraction unit 12. First, the search range of the eyes in the vertical direction (y-axis direction) is calculated based on the height TOH of the crown detected by the crown detector 13 and the height HOM of the mouth detected by the mouth detector 14, for example, by the following formula. It is calculated by (13).
[0088]
(Equation 13)

[0089]
Here, e1 and e2 are preset coefficients. “etop” and “ebtm” are a lower limit value and an upper limit value in the vertical coordinate of the search range, respectively. The intensity edge (x) of a horizontal edge (hereinafter, referred to as a horizontal edge) of a pixel sandwiched between the lower limit value and the upper limit value in the vertical coordinates and present in a rectangular region corresponding to a skin color region of interest is present. , Y). As a method for detecting the intensity of the horizontal edge, any existing method may be used. For example, a differential filter having a filter coefficient as shown in FIG. 7 is applied to each of the R, G, and B images. , The sum of the absolute values of the filter outputs for each image can be used as the strength of the horizontal edge.
[0090]
The intensity edge (x, y) of the horizontal edge calculated at each coordinate of the input color image is accumulated in the horizontal direction (x-axis direction), and the histogram Hedge (y) indicating the vertical horizontal edge in the rectangular area Is calculated by the following equation (14).
[0091]
[Equation 14]

[0092]
Here, V (n) is the vertex coordinates of the rectangular area corresponding to the skin color area n obtained by the skin color area extraction unit 12. FIG. 8 is a schematic diagram showing the relationship between the generated histogram Hedge (y) and the rectangular area 72 corresponding to the flesh color area 71.
[0093]
The histogram Hedge (y) is smoothed by a one-dimensional low-pass filter as necessary to remove noise and the like, and then the vertical coordinate y corresponding to the maximum value is detected as the eye height HOE.
[0094]
The search range of the eye height can be set by the following equation (15) using the vertex coordinates of the corresponding rectangular area in addition to the one shown in the above equation (13).
[0095]
(Equation 15)

[0096]
Here, e3 and e4 are preset coefficients. Further, ebtm calculated by the above equation (13) is V (n) .V (n) of the vertex coordinates of the rectangular area surrounding the skin color area. If it is smaller than sty, it is highly likely that the detection of the top height TOH or the mouth height HOM is not properly performed. Therefore, in such a case, the vertex list V can be modified by storing, for example, -1 which is an invalid value as the position coordinates in the vertex coordinates V (n) of the corresponding rectangular area.
[0097]
The detected eye height HOE is sent to the jaw detection unit 16 and the determination unit 19. The corrected vertex list V is sent to the jaw detecting unit 16, the center line detecting unit 17, and the area correcting unit 18.
[0098]
(2-4) Jaw detection of a person
The chin detecting section 16 detects the position (height) of the chin for each skin color area having vertex coordinates that are not invalid in the vertex list V corrected by the eye detecting section 15. For example, as shown in FIG. 9, the ratio of the distance 81 between the chin and the mouth and the distance 82 between the eyes and the mouth of the person's face 80 are substantially constant, as shown in FIG. And can be estimated by the following equation (15).
[0099]
(Equation 16)

[0100]
Here, c is a preset coefficient, and HOC indicates the height of the chin. The calculated jaw height HOC is sent to the area correction unit 18.
[0101]
(2-5) Detection of center line of human face
Next, the face center line detection unit 17 detects the position of the center line that divides the face into right and left for each skin color region having vertex coordinates that are not invalid in the vertex list V corrected by the eye detection unit 15. I do.
[0102]
Here, first, a mouth search range in vertical coordinates around the mouth height HOM detected by the mouth detection unit 14 is set. This search range can be calculated, for example, from the width in the vertical direction of the corresponding rectangular area by the following equation (17).
[0103]
[Equation 17]

[0104]
Here, m is a preset coefficient, and V (n) is the vertex coordinates of the rectangular area corresponding to the skin color area n. Let mtop and mbtm, respectively, calculated by equation (17) be the lower and upper limits of the y-coordinate of the search range, respectively. The horizontal search range may be the horizontal width of the rectangular area. That is, the upper and lower limits of the x coordinate are respectively set to the left end V (n). stx and right end V (n). edx. FIG. 10 is a schematic diagram showing the mouth height HOM and the search ranges mtop and mbtm in the rectangular area 92 corresponding to the skin color area 91.
[0105]
Next, the intensity of redness is calculated for the pixels that are present in the set search range and are not included in the flesh color region by using the above equation (11), and as shown in FIG. Is detected as the horizontal coordinate position COH of the center line, where the average value of the horizontal coordinates of the pixels where is larger than the threshold value is detected. When calculating the intensity of redness, by removing the pixels belonging to the skin color region, the influence of the pixels belonging to the skin color region can be excluded, and the center line of the face can be detected with extremely high accuracy. Thus, the detected position COH of the face center line is sent to the area correction unit 18 and the determination unit 19.
[0106]
The threshold value for the intensity of redness may be a fixed value set in advance, but may be adaptively changed as described below. That is, first, an average value or a maximum value of all the pixels in the search range is calculated, and a value obtained by multiplying these by a preset coefficient may be used as the threshold value.
[0107]
When calculating the average value of the horizontal coordinates, a method called robust estimation can be used to reduce the influence of noise and the like. That is, for example, the arithmetic operation represented by the following expression (18) is repeatedly performed with the arithmetic mean value of the x-coordinate of the pixel having the redness greater than the threshold value as the initial value center (0).
[0108]
(Equation 18)

[0109]
Here, center (n) is the center position calculated in the n-th iteration, rdsh (x, y) indicates the intensity of redness at coordinates (x, y), and Trdsh is the intensity of redness. Represents a threshold value for. h (d) indicates the weight for each x coordinate, and is given as a function of the difference d between the already calculated center position center (n) and the x coordinate. As the function h (d), for example, a function expressed by the following equation (19) can be used so that a smaller weight is given to an x coordinate having a larger difference d.
[0110]
[Equation 19]

[0111]
g is a parameter indicating the spread of the function h (d), and is decreased by a predetermined method at each iteration. In the iterative process, the value of g becomes smaller than a predetermined threshold value, or the difference between the calculated center (n + 1) and the center (n) calculated in the previous iteration is determined by another predetermined value. The process is performed until the value becomes smaller than the threshold value.
[0112]
Although the number of repetitions until convergence increases, in order to more reliably converge to a correct value, the calculation of the above equation (18) can be repeated using the same g. That is, using the same g, the process is repeated until the difference between center (n + 1) and center (n) becomes smaller than the threshold value, and after center (n) converges, the value of g is reduced by a predetermined method. In this case, the repetition processing ends when the value of g becomes smaller than the threshold value.
[0113]
(2-6) Correction of rectangular area
The region correction unit 18 calculates a rectangular region again for each skin color region having vertex coordinates that are not invalid in the vertex list V corrected by the eye detection unit 15, and corrects the vertex list V. For example, using the height TOH of the crown obtained by the crown detector 13, the height HOC of the jaw obtained by the jaw detector 16, and the position COH of the center line obtained by the center line detection, As shown in FIG. 11, a rectangular area 100 can be set. That is, two vertex coordinates {(stx, sty), (edx, edy)} indicating the corrected rectangular area 100 can be calculated by the following equation (20).
[0114]
(Equation 20)

[0115]
Here, asp is a coefficient indicating the ratio of the height of the person's face to the width, and it is assumed that an appropriate value is set in advance.
[0116]
The newly calculated vertex coordinates for the skin color area n are overwritten on the vertex list V and sent to the determination unit 19.
[0117]
(3) Face judgment
The determination unit 19 determines, for each skin color region having vertex coordinates that are not invalid in the vertex list V corrected by the region correction unit 18, whether the skin color region is a face region. The determination of the face area is based on the fact that, for example, in the face area of a person, many horizontal edges are distributed in the eyes and the mouth, and the lip color is more reddish than the other parts. Is verified at the mouth height HOM detected by the mouth detection unit 13 and the eye height HOE detected by the eye detection unit 14. The determination result is output as a binary flag faceflag indicating whether or not the area is a face area.
[0118]
Hereinafter, the determination unit 19 will be described in more detail. FIG. 12 is a block diagram illustrating the determination unit 19. As illustrated in FIG. 12, the determination unit 19 receives the vertex list V corrected by the region correction unit 18, the mouth height HOM, the eye height HOE, and the center line position COH of the face. An area setting unit 21 for setting an eye area and a mouth area of the image, a corrected vertex list V and a color image are input, and a horizontal edge (horizontal edge) of a skin color pixel in a rectangular area is calculated. An edge detection unit 22, left and right eye area data and horizontal edge data of a rectangular area are input, and an eye pattern determination unit 23 that determines whether or not the eye is an eye. Direction edge data is input, and a mouth pattern determination unit 24 that determines whether the face is a mouth, and a comprehensive determination that determines whether the face is a face based on the determination results of the eye pattern determination unit 23 and the mouth pattern determination unit 24. Consisting of part 25.
[0119]
(3-1) Area setting for each feature point
As shown in FIG. 13, the area setting unit 21 calculates the area of the right eye, the left eye, and the mouth estimated from the vertex coordinates V (n), the mouth height HOM, and the eye height HOE as shown in FIG. It is set as a rectangular area near the HOM and the eye height HOE.
[0120]
That is, the right eye area can be a rectangular area having vertex coordinates {(eyeRstx, eyeRsty), (eyeRedx, eyeRedy)} calculated by the following equation (21), for example.
[0121]
(Equation 21)

[0122]
Here, eyesizex indicates the horizontal width of the region of one eye, and is calculated by multiplying the coefficient s1 by the width of the rectangular region represented by the vertex coordinates V (n). Also, "beys" is the distance between the eyes, and similarly, can be calculated by multiplying the width of the rectangular area represented by the vertex coordinates V (n) by the coefficient s2. Further, eyesize indicates the height (width in the vertical direction) of the eye area, and is calculated by integrating a predetermined ratio eyeasp with the eye width.
[0123]
Similarly, the left eye area can be a rectangular area having vertex coordinates {(eyeRstx, eyeLsty), (eyeLedx, eyeLedy)} calculated by the following equation (22).
[0124]
(Equation 22)

[0125]
Here, eyesizexex, eyesizey, and Bayes have the same value as the above equation (21).
[0126]
On the other hand, the mouth area can be a rectangular area having the vertex coordinates {(moustx, mousety), (moedx, mousey)} calculated by the following equation (23).
[0127]
[Equation 23]

[0128]
Here, mouseizex indicates the horizontal width of the mouth area, and is calculated by multiplying the horizontal width of the rectangular area represented by the vertex coordinates V (n) by the coefficient s3. Further, mousesize indicates the height (width in the vertical direction) of the mouth area, and can be calculated by multiplying a predetermined ratio mouasp by the width of the mouth.
[0129]
The set vertex coordinates of the right eye region, the left eye region, and the mouth region are stored in the vertex lists VeyeR, VeyeL, and Vmou, respectively, in the same manner as in the above equation (6). VeyR and VeyeL are sent to the eye pattern determination unit 23, and Vmou is sent to the mouth pattern determination unit 24.
[0130]
(3-2) Detection of horizontal edge of each area
The horizontal edge detection unit 22 outputs a horizontal edge to a skin color pixel which is present in a rectangular region represented by vertex coordinates V (n) and whose value of the region map A (x, y) is n for each skin color region. The edge in the direction is detected. As a method for detecting the horizontal edge, any existing method may be used. For example, for each target pixel, the edge intensity in the horizontal direction is calculated by a differential filter having a filter coefficient shown in FIG. The edge strength in the vertical direction is calculated by using a differential filter having a differential coefficient shown in the following expression, and the pixels satisfying the condition shown in the following expression (20) are classified into the pixels constituting the horizontal edge (the horizontal edge is a predetermined threshold value). There is a method of detecting as the above (pixel).
[0131]
(Equation 24)

[0132]
Here, edgeH, edgeV, and Te represent a horizontal edge strength, a vertical edge strength, and a threshold, respectively.
[0133]
The detection result of the pixels constituting the horizontal edge is stored as a binary image in an edge map Medge which is a two-dimensional array. For example, when the pixel at the coordinates (x, y) is detected as a pixel that satisfies the condition of the above equation (24), that is, a pixel forming a horizontal edge, 1 is stored in Medge (x, y), If the pixel at the coordinates (x, y) does not form a horizontal edge, 0 is stored in the Mediage (x, y). The generated edge map Medge is sent to the eye pattern determination unit 23 and the mouth pattern determination unit 24.
[0134]
The threshold Te for detecting a horizontal edge can use a preset fixed value, but can be adaptively changed as follows. That is, for example, a value obtained by calculating the horizontal edge strength for all the pixels included in the skin color region extracted by the skin color region extraction unit 12 in advance, that is, by integrating a preset coefficient with the average value thereof Can also be used as the threshold.
[0135]
In addition, here, an example in which the horizontal edge is detected only for the pixels included in the skin color region extracted by the skin color region extraction unit 12 has been described. However, the colors of the eyes and the mouth are usually different from the skin color. However, these parts themselves are not extracted as the skin color area. Therefore, since the edges of the eyes and the mouth exactly correspond to the boundary between the skin color region and the non-skin color region, if an edge is detected only on the skin color region, a sufficient number of pixels constituting the edge may not be detected. is there. Therefore, in order to more stably detect an edge, for example, a pixel that is not a flesh color in a rectangular region indicated by the vertex coordinates corrected by the region correcting unit 18 and an adjacent pixel is included in the flesh color region n The horizontal edge may be detected for the object.
[0136]
(3-3) Judgment of the likeness of the eye pattern in the eye region
The eye pattern determination unit 23 uses the right eye vertex list VeyeR and the left eye vertex list VeyeL, and the edge map Medge, and an area near the eye height HOE detected by the eye detection unit 15 has an eye feature. Is determined.
[0137]
Here, first, the number of pixels ecountR and ecountL constituting horizontal edges existing in the right-eye area and the left-eye area, respectively, indicated by the right-eye vertex list VeyeR and the left-eye vertex list VeyeL are calculated by the following equation (25). Count as
[0138]
(Equation 25)

[0139]
Here, eyeR and eyeL are sets of pixels existing inside the right eye region and the left eye region, respectively.
[0140]
In addition, the number of pixels ecountO of the horizontal edge that has a vertical coordinate between the lower limit and the upper limit of the right eye region or the left eye region and is not included in any of the eye regions is counted as the following equation (26). .
[0141]
(Equation 26)

[0142]
In general, at the position of the eyes in a person's face area, there are many pixels forming many horizontal edges in a part (area) where the eyes actually exist, and the horizontal edges are formed in other parts. There are few pixels. Accordingly, when the condition of the eye pattern likelihood represented by the following equation (27) is satisfied, it is determined that the area near the eye height HOE detected by the eye detection unit 15 has the features of the eye. I do.
[0143]
[Equation 27]

[0144]
Here, the value of the left side in the first condition of the above equation (27) is that many pixels constituting the horizontal edge included in the right eye and left eye regions, and the horizontal edge is included in the other regions. Becomes large when the number of pixels constituting the pixel is small. AreaR and areaL in the second condition of the above equation (27) are the total number of pixels existing inside the right eye area and the left eye area, respectively, and the entire left side is the horizontal direction in the right eye area and the left eye area. Shows the density of the pixels constituting the edge of. That is, both conditions reflect the above-described features of the eye. As the threshold values Tedst and Tedns under these conditions, appropriate values are set in advance.
[0145]
The determination result by the above equation (27) is represented by a binary flag. For example, if all of the conditions of the above equation (27) are satisfied, the flag is set to 1 to indicate that the determination result is eye. On the other hand, if any one of the conditions of the above equation (27) is not satisfied, the flag is set to 0 to indicate that the determination result is not the eye. The flag eyeflag representing the determination result is sent to the overall determiner 25.
[0146]
Here, the above equation (27) is used as the determination condition, but other conditions may be used as long as the above-mentioned eye characteristics are reflected.
[0147]
(3-4) Judgment of Likeness of Mouth Pattern in Mouth Area
The mouth pattern determination unit 24 determines whether or not a region near the mouth height HOM detected by the mouth detection unit 14 has the features of the mouth, using the mouth region Vmou and the edge map Medge.
[0148]
Here, first, the number mcountM of pixels constituting the horizontal edge existing inside the mouth area Vmou and the average value avershM of the redness intensity of all the pixels existing inside the mouth area Vmou are calculated by the following equation (28). .
[0149]
[Equation 28]

[0150]
Here, mou is a set of pixels existing inside the mouth area, and NM is the total number of pixels existing inside the mouth area. Further, rdsh (x, y) is a redness intensity at the pixel (x, y), and for example, a value calculated by the above equation (11) can be used.
[0151]
Next, among the pixels whose vertical coordinates are between the lower limit and the upper limit of the mouth area and which are not included in the mouth area, the number mcountO of the pixels constituting the horizontal edge and the vertical coordinates are the mouth area. The average value redshO of the redness intensity of all the pixels which are between the lower limit and the upper limit of and is not included in the mouth region is calculated by the following equation (29).
[0152]
(Equation 29)

[0153]
Here, NO indicates the target area, that is, the total number of pixels whose vertical coordinates are between the lower limit and the upper limit of the mouth area and are not included in the mouth area.
[0154]
In general, at the mouth height position of a person's face region, there are many pixels forming horizontal edges in a portion (region) where the mouth actually exists, and in other portions, horizontal edges are formed. There are few pixels. In addition, the mouth part is more reddish than other parts. Accordingly, when the condition of the mouth pattern likelihood represented by the following equation (30) is satisfied, the area (mouth area) near the mouth height HOM detected by the mouth detection unit 14 has the mouth feature. It is determined that there is.
[0155]
[Equation 30]

[0156]
Here, the value of the left side in the first condition of the above equation (30) is that many pixels constituting the horizontal edge included in the mouth region, and pixels constituting the horizontal edge in the region other than the mouth region are It becomes big when there is little. Also, areaM on the left side of the second condition in the above equation (30) is the total number of pixels existing in the mouth area, and the entire left side indicates the density of pixels constituting the horizontal edge in the mouth area. Further, the value on the left side in the third condition of the above equation (30) increases when the redness in the mouth region is strong and the redness in the region other than the mouth region is weak. Furthermore, the fourth condition of the above equation (30) requires that the intensity of redness inside the mouth area itself is larger than a predetermined threshold value Trdshm. In other words, both conditions reflect the above-mentioned mouth characteristics. As the threshold values Tedstm, Tednsm, Tcdstm, and Trdshm under the first to fourth conditions of the above equation (30), appropriate values are set in advance.
[0157]
The determination result by the above equation (30) is represented by a binary flag. For example, when all the conditions of the above equation (30) are satisfied, the flag is set to 1 to indicate that the mouth area is determined to be a mouth pattern. On the other hand, if any one of the conditions in Expression (30) is not satisfied, the flag is set to 0, indicating that the mouth area is determined not to be a mouth. The flag mouflag representing this determination result is sent to the overall determiner 25.
[0158]
Here, the above equation (30) is used as the determination condition, but other conditions may be used as long as the above-mentioned mouth characteristics are reflected.
[0159]
(3-5) Judgment as to whether or not the extracted area is a face
The comprehensive determination unit 25 determines that the flag eyeflag and the mouflag sent from the eye determination unit 23 and the mouth region 24 are both 1, that is, the eye region and the mouth region set by the region setting unit 21 are: Only when it is determined that they have the features of the eyes and the features of the mouth, the skin color area of interest at present is determined as a face area, and the result is output as a binary flag faceflag.
[0160]
In the present embodiment, the height TOH of the crown is used to limit the search range of the eye, but after the height HOE of the eye is detected, the height HOE of the eye and the height HOE of the mouth are determined. The height HOM may be used to verify the top height TOH. That is, the upper limit and the lower limit at which the top of the head should exist from the distance between the eye and the mouth are calculated by, for example, the following equation (31).
[0161]
(Equation 31)

[0162]
If the height of the crown TOH is not between these two values, it is determined that the correct height has not been detected by the crown detector 13 and the height of the crown is corrected by a predetermined method. Do. For example, the ratio h3 of the distance between the crown and the mouth with respect to the distance between the eye and the mouth is set in advance, and the height TOH of the crown is reset by the following equation (32).
[0163]
(Equation 32)

[0164]
Here, the size of the eye area, the distance between the eyes, and the size of the mouth area are determined with respect to the size of the rectangular area surrounding the skin color area shown in the above formulas (21), (22), and (23). Although the method of calculating the relative color has been described, in the skin color region extraction, when a part of the face is extracted due to a shadow, or when the area from the face to the shoulder is extracted as one skin color region, the eye region is extracted. And mouth area may not be set properly. In order to avoid this problem, for example, in Equations (21) to (23), a plurality of coefficients S1, S2, and S3 for determining the size of the area are prepared, and the determination is performed for each value. can do. Then, if it is determined that the skin area is a face area with respect to any of the coefficients, the skin color area of current interest is determined as a face area.
[0165]
In the present embodiment, the positions of the crown and the mouth are detected based on the extracted skin color area, and the eye position is detected by setting an eye search range from these positions. Can be detected. In addition, by calculating the position of the jaw from the positions of the eyes and mouth, the difference in brightness and color between the face and neck is small, and accurate detection of the position of the jaw can be performed accurately even when it is difficult to detect it with high accuracy. It can be carried out. Furthermore, since the center line of the face is detected based on the intensity of redness of the mouth, the center line of the face can be detected with extremely high accuracy. Furthermore, the face determination unit determines the likeness of the eye pattern and the likeness of the mouth pattern, and comprehensively determines whether the face is a face based on the determination result. However, the reliability of the determination result of whether or not the face is high is high.
[0166]
Further, when there are a plurality of skin color regions determined to be faces by the determination unit 19, a selection unit (not shown) for selecting one face region from the plurality of face regions based on, for example, the position of the face region is provided. You can also. Thus, for example, one face region can be extracted from an image having a plurality of face regions and subjected to trimming processing. Note that the determination unit 19 may have a function of selecting a face area.
[0167]
Second embodiment
Next, a second embodiment of the present invention will be described. FIG. 15 is a block diagram showing a portrait image photographing apparatus according to the present embodiment. In the present embodiment, the image processing apparatus according to the first embodiment is applied to a portrait image (photograph) photographing apparatus.
[0168]
As shown in FIG. 15, the portrait image capturing device 30 receives a color image of a person captured by an electronic still camera, a video camera, or the like, and receives an image capturing device 31 that outputs digital data, and image data. A face detector 32 for detecting a face region, an area selector 33 for selecting a face region to be trimmed when there are a plurality of detected face regions, and a trimming device 34 for trimming the selected face region It is composed of
[0169]
The face area extractor 32 extracts a face area of a person according to the processing method of the image processing apparatus according to the first embodiment. However, the determination unit 19 according to the first embodiment outputs a flag faceflag indicating whether or not each of the skin color regions is a face. However, in the present embodiment, it is determined that each of the skin color regions is not a face region. After storing an invalid value, eg, −1, as a position coordinate in the vertex coordinates V (n) corresponding to the extracted flesh color region, the value is sent to the region selector 33.
[0170]
When a plurality of areas are determined to be faces by the face area determiner 32, the area selector 33 selects one face area to be trimmed from the plurality of areas. For example, if it is clear that only one person exists in the captured image, there is only one true face area, and the other areas are considered to be erroneously detected by the face area determiner 32. In such a case, if it is estimated that the true face region is located above the other skin color regions, the valid rectangle included in the vertex list V whose center is located at the top is selected. Can be selected.
[0171]
In addition, even when there is a possibility that a plurality of persons may be present in the captured image, the face of the person to be trimmed in the input image may be located, for example, at a position closest to the center of the image due to an instruction during imaging. If it can be expected to exist, it is possible to select the effective rectangle included in the vertex list V whose center is closest to the center of the image.
[0172]
Furthermore, when the main reason for extracting a plurality of face regions is that the parent holds the child and images the child's face, for example, to image the child's face, the child's face region to be trimmed is , Can be estimated to be located at the bottom of the extracted face region, so that among the valid rectangles included in the vertex list V, the one whose center is located at the bottom can be selected.
[0173]
These target area selection methods are set in advance in consideration of the usage form, and by providing an interface (not shown), the photographer and / or the subject can be set manually. It is also possible to put.
[0174]
An invalid value, for example, -1 or the like is stored as a position coordinate in the vertex list V corresponding to a skin color area other than the selected face area. The corrected vertex list V is sent to the trimming unit 34.
[0175]
The trimming unit 34 determines the position of the partial image to be cut out in accordance with the valid rectangular area V (n) included in the sent vertex list V. FIG. 16 is a schematic diagram showing a cutout position of a partial image cut out from an input color image. As shown in FIG. 16, in the color image 110, the center position (x0, y0) of the rectangular area 111 indicated by the vertex coordinates V (n) of the vertex list V is the predetermined position (tx0, ty0) of the image 112 after trimming. ) Is extracted as a partial image 112 so as to match a rectangular area represented by two vertex coordinates shown in the following equation (33).
[0176]
[Equation 33]

[0177]
Here, twdt and thgt indicate the width (length in the horizontal direction) and the height (length in the vertical direction) of the preset image after trimming, respectively.
[0178]
The trimmed image is output to an image display device (not shown) such as a printer and a monitor, or an image storage device.
[0179]
Note that the trimming process in the trimming unit 34 is not limited to the one described here, and for example, the following method can be used.
[0180]
That is, the face area extractor 32 sends the eye height HOE and the center line position COH calculated for each face area to the trimming unit 34 in addition to the vertex list V. The trimming unit 34 cuts out a partial image so that the eye height and the center line are at predetermined positions on the trimmed image. Instead of the eye height HOE, the crown height TOH or the mouth height HOM may be used.
[0181]
In particular, when it is not permissible for the head to protrude in the image after trimming, it is effective to use the height TOH at the top of the head. However, as shown in FIG. 17, when there are a plurality of persons 121 and 122 in the same input color image 120, and these persons 121 and 122 overlap in the horizontal direction, the top search 123 and 124 respectively. Since the ranges also overlap, of the persons 121 and 122, the face region in which the true vertex is located below (the vertical coordinate is large), that is, the vertical region among the horizontal sides of the rectangular region corresponding to the skin color region For a person 122 having a face area having a larger vertical coordinate on the side having the smaller coordinate, the height TOH of the top of the head cannot be correctly detected. Therefore, in the trimming using the height of the top of the head, good results may not be obtained unless the face area to be trimmed is located at the top. In order to avoid such a problem, the height of the top of the head may be checked, and if the value is inappropriate, a correction may be made.
[0182]
For example, as the TOH check, the eye height HOE and the mouth height that have already been detected are used to determine whether or not the condition represented by the following equation (34) is satisfied.
[0183]
(Equation 34)

[0184]
Here, medst is a difference HOE-HOM between the height of the eyes and the height of the mouth, and h1 and h2 are preset coefficients. If the condition shown in the above equation (34) is not satisfied, the value of TOH is corrected by the following equation (35).
[0185]
(Equation 35)

[0186]
Further, when the face area determination unit 32 determines that a plurality of areas are faces, it is also possible to select all of the plurality of areas as trimming targets. In this case, the trimming unit 34 cuts out a partial image including all of the plurality of areas determined to be a face, and performs a reduction process as necessary to obtain an image of a predetermined size.
[0187]
In the present embodiment, since the image processing apparatus of the first embodiment is used, the reliability of the area detected as a face by the face area detector is high, and the face area is detected by the face area detector. In addition, the feature point information, that is, the top of the person, the mouth and eyes of the face, and the position of the center line of the face are detected, and the trimming is performed using these data. Further, it is possible to obtain a portrait image photographing apparatus capable of automatically trimming a desired face region with high accuracy.
[0188]
Also, even when an image includes a plurality of persons, appropriate trimming can be performed according to the application. For example, when a parent holds a child at a photo booth in a hospital or the like and takes a picture, the area selector 33 selects a specific face area from a plurality of face areas based on information such as the position or size of the extracted face area. You can choose. That is, for example, when two face regions are extracted, if the region selector 33 is set to select a position whose position is below the screen or whose size is small, the parent can hold the child. When photographing, a child's face can be extracted and trimmed. Further, if trimming is performed for any of the face regions, a plurality of trimmed images can be obtained by one image.
[0189]
Third embodiment
Next, a third embodiment of the present invention will be described. In the present embodiment, an image processing apparatus is applied to a portrait image photographing apparatus as in the second embodiment, and has the same configuration as that of the second embodiment shown in FIG. That is, it is configured by the image pickup device 31, the face region extractor 32, the region selector 33, and the trimming device 34, but the processing in the trimming device 34 is different. In the second embodiment, the trimming is performed by cutting out a partial image of a predetermined size from the input image. However, in the present embodiment, the size of the face area to be cut out is also adjusted. I do.
[0190]
For example, the face area extractor 32 shown in FIG. 15 that extracts a face area by the same method as that of the above-described first embodiment uses the face height HOE and the mouth height of the person in addition to the vertex list V. The HOM and the center line COH are sent to the trimming device 34.
[0191]
First, the trimming unit 34 calculates the distance medst between the eye and the mouth required on the trimmed image and the ratio p of the difference between the HOE and the HOM obtained from the input color image by the following equation (35): The input image is enlarged or reduced so that the vertical and horizontal lengths are each p times.
[0192]
[Equation 36]

[0193]
The values of the eye height HOE, the mouth height HOM, and the center line position COH are also corrected according to the enlargement or reduction of the image. Any existing method may be used for image enlargement processing and image reduction processing.
[0194]
With respect to the enlarged or reduced image, a partial image is cut out so that the height of the eye and the center line are at predetermined positions on the trimmed image.
[0195]
Here, the distance between the eye and the mouth is used to adjust the size, but it goes without saying that other information calculated by the face area detector 32 can be used. For example, the value of the ratio of the above equation (35) may be calculated based on the distance between the crown and the chin.
[0196]
Positioning for trimming can also be performed based on a feature point used for size adjustment, that is, a feature point different from eyes and a mouth. That is, the size can be adjusted at the interval between the eyes and the mouth, and the positioning can be performed by the height of the crown or the height of the chin and the center line.
[0197]
Furthermore, when the height of the crown is used, the value TOH may be corrected using the height of the eyes and the mouth, as in the second embodiment.
[0198]
In the present embodiment, similarly to the second embodiment, a face area detected with extremely high accuracy is input from the face area extractor, and data indicating feature points of a person (face) are input. Therefore, the positioning accuracy of trimming is high, and enlargement / reduction processing can be performed during trimming.
[0199]
Fourth embodiment
Next, a fourth embodiment of the present invention will be described. FIG. 18 is a block diagram illustrating the image processing apparatus according to the present embodiment. The present embodiment is applied to an image processing apparatus for extracting a face area of a person from an input color image, similarly to the image processing apparatus of the first embodiment described above. In the second embodiment, only the portions different from the first embodiment shown in FIGS. 1 to 14 will be described in detail.
[0200]
The image processing device 210 according to the present embodiment includes an image input unit 211 that receives a color image and outputs the color image as digital data, a skin color region extraction unit 212 that extracts a skin color region from the color image, A crown detector 213 for detecting the crown, a center line detector 217 for detecting the center line of the face of the person, a mouth detector 214 for detecting the mouth of the person, and re-detecting the center line as necessary. A center line correcting unit 220 that corrects a center line, an eye detecting unit 215 that detects a human eye, a chin detecting unit 216 that calculates a chin position of a person, an area correcting unit 218 that corrects a face region, The determination unit 219 determines whether the extracted skin color area is a human face.
[0201]
The image input unit 211 is the same as that of the first embodiment, but the color image output from the image input unit 211 is a skin color region extraction unit 212, a crown detection unit 213, a mouth detection unit 214, an eye detection unit 215, and a judgment. Unit 219 and the center line correcting unit 220.
[0202]
The skin color region extraction unit 212 is the same as that of the first embodiment, but the calculated region map A is sent to the mouth detection unit 214, the center line detection unit 217, the determination unit 219, and the center line correction unit 220. , The vertex list V (n) is sent to the top detection unit 213.
[0203]
The crown detector 213 detects the height TOH of the crown, as in the first embodiment. Prior to the detection, it is determined whether the crown is included in the color image. It has a function to make a determination. To determine whether or not the top of the head exists in the color image, for example, by the same method as in the above equation (10), R (x, y), G (x, y), The variance of the value of B (x, y) is calculated, and it can be determined whether or not the sum of the variances is equal to or less than a preset threshold. That is, when the sum of the variances is larger than a preset threshold, it is determined that the top of the head does not exist in the color image. In the vertex list V (n) corresponding to the flesh-color area determined to have no vertex in the color image, an invalid value, for example, -1 is stored as the position coordinate. Here, when it is determined that the top of the head is not present in the color image, the top end of the color image may be set as the position of the top of the color image. Alternatively, the fact that the top of the head has not been detected may be notified, for example, by voice or sound, and the user may be prompted to take a picture again.
[0204]
On the other hand, when it is determined that the top of the head exists, the height TOH of the top of the head is detected, and the detected height TOH of the top of the head is sent to the eye detection unit 215 and the area correction unit 218, and the correction is performed. The obtained vertex list V (n) is sent to the mouth detection unit 214, the eye detection unit 215, the chin detection unit 216, the center line detection unit 217, the area correction unit 218, and the center line correction unit 220.
[0205]
The center line detection unit 217 determines the position of the center line that divides the skin color region into two parts for each skin color region having vertex coordinates that are not invalid in the vertex list V (n) corrected by the crown detection unit 213. To detect. However, here, unlike the first embodiment, within the rectangle represented by the vertex list V (n), the average value of the horizontal coordinate x of the pixel (x, y) extracted as the skin color area is set as the center. It is detected as the line position COH. The detected center line position COH is sent to the mouth detection unit 214 and the center line correction unit 220.
[0206]
The mouth detection unit 214 detects the height of the mouth for each skin color region having vertex coordinates that are not invalid in the vertex list V (n) corrected by the crown detection unit 213. As in the first embodiment, also in the present embodiment, a histogram Hrdsh (y) obtained by accumulating the value rdsh (x, y) of the above equation (11) representing the intensity of redness in the horizontal direction. Is used. At this time, the accumulation range is limited to a region 241 around the center line position COH detected by the center line detection unit 217, as indicated by hatching in FIG. That is, the histogram Hrdsh (y) in the present embodiment is generated by the following equation (37).
[0207]
(37)

[0208]
Here, 1w is a coefficient larger than 0 and smaller than 1 which is set in advance. After the histogram Hrdsh (y) is smoothed by a one-dimensional low-pass filter as needed, the vertical coordinate y corresponding to the maximum value is detected as the mouth height HOM.
[0209]
Furthermore, in the present embodiment, the reliability lipconf of the detected mouth height HOM is calculated in accordance with the spatial distribution of strongly reddish pixels. The mouth is at the center of the face in the horizontal direction, and it can be assumed that the pixels constituting the mouth exist in a relatively narrow range around the center line position COH. Therefore, when pixels with strong redness are largely spread spatially, that is, when the distribution of pixels with redness equal to or more than a predetermined threshold is large, it is evaluated that the reliability of the detected mouth height is low. .
[0210]
The reliability lipconf can be calculated, for example, as follows. That is, first, a rectangular area Rm as shown by oblique lines in FIG. 20 centering on coordinates (COH, HMO) determined from the detected center line position COH and the mouth height HOM is set, and this rectangular area Rm is set. Is calculated with respect to each pixel existing inside the image, and the calculated redness intensity rdsh (x, y) is smaller than a preset threshold value Trdsh. Large pixels are selected, and the standard deviation of their horizontal coordinates x is calculated as xsdv. That is, the calculation of xsdv is performed as shown in the following equation (38).
[0211]
[Equation 38]

[0212]
The rectangular area Rm for calculating the reliability lipconf can be set from the center line position COH and the vertex list V (n) corresponding to the currently focused skin color area, for example, as in the following equation (39). .
[0213]
[Equation 39]

[0214]
Here, mtop and mbtm are the same as those shown in the above equation (17).
[0215]
Next, as shown in the following equation (40), the calculated standard deviation xsdv is converted into a reliability lipconf using an appropriate monotone increasing function F whose value range is 0 or more and 1 or less.
[0216]
(Equation 40)

[0219]
Here, V (n). edx-V (n). stx + 1 represents the horizontal width of the rectangular area surrounding the skin color area of interest at present.
[0218]
The detected mouth height HOM is sent to the eye detection unit 215, the jaw detection unit 216, the center line correction unit 220, and the determination unit 219. Further, the reliability lipconf is sent to the center line correction unit 220 and the determination unit 219.
[0219]
In order to more accurately detect the height of the mouth, similar to the first embodiment, the redness intensity rdsh is weighted according to the position of the target pixel and the brightness of the target pixel. It goes without saying that you can do it. When calculating the reliability lipconf of the mouth height, similar weighting can be performed on the horizontal coordinates of each pixel. Furthermore, although the pixels that contribute to the calculation of the reliability are selected here based on a preset threshold value Trdsh, this threshold value can be set adaptively. For example, within the shaded area 244 in FIG. 19, the average value of the redness intensity of each pixel is calculated, and a value obtained by multiplying the average value by an appropriate coefficient is used as a threshold value for calculating reliability. It can also be used as Trdsh.
[0220]
The center line correction unit 220 calculates the height of the mouth calculated by the mouth detection unit 214 for each skin color region having vertex coordinates that are not invalid in the vertex list V (n) corrected by the crown detection unit 213. Only when the value of the reliability lipconf is larger than a preset threshold value, a method similar to that of the center line detecting unit 17 in the first embodiment, that is, for example, a pixel whose redness value is larger than the threshold value The average value of the horizontal coordinates is obtained by robust estimation, the re-detection is performed to detect this as the horizontal coordinate position of the center line, and the result is output. When the reliability lipconf is equal to or less than the threshold, erroneous detection may occur if the center line is detected based on the intensity of redness as in the first embodiment. The output center line position COH is output as it is. The output center line position COH is sent to the area correction unit 218 and the determination unit 219.
[0221]
The eye detection unit 215 detects an eye height for each skin color region having vertex coordinates that are not invalid in the vertex list V (n) corrected by the crown detection unit 213. However, in the first embodiment, only the vertical coordinate corresponding to the maximum value of the histogram Hedge (y) of the edge intensity shown in the above equation (14) is detected as the eye height HOE. In the embodiment, it is assumed that a plurality of vertical coordinates corresponding to the maximum point of the histogram are detected as eye height candidates. The local maximum point coordinates of the histogram Hedge (y) can be detected as coordinates y that satisfy the following equation (41), for example.
[0222]
(Equation 41)

[0223]
The coordinates of the eye height candidate detected for the skin color area n are stored in the list eyelist (n, i) according to a predetermined priority order i. For example, the value of the histogram Hedge (y) can be used to determine the priority. In this case, the list eyelist (n, i) stores the vertical coordinates of the local maximum point at which the value of the histogram Hedge (y) obtained for the skin color area n is the ith largest. The list eyelist (n, i) in which the coordinates of the eye height candidates are stored is sent to the jaw detection unit 216 and the determination unit 219.
[0224]
Note that it is not necessary to store the vertical coordinates of all the detected maximum points in the coordinate list of eye height candidates for each skin color region, ie, the histograms are used to avoid the influence of noise, for example. May be stored in the order from the largest value to the largest number of maximum points set in advance.
[0225]
The chin detecting section 216 applies a method similar to that of the first embodiment to each skin color region having vertex coordinates that are not invalid in the vertex list V (n) corrected by the crown detecting section 213. Detect height. However, in the present embodiment, there is not one eye height HOE described in the first embodiment, but a plurality of eye height candidates. Is calculated and stored in the chin height list chinlist (n, i). Here, n is the number of the skin color area currently being processed, and i represents the priority of the corresponding eye height candidate. The calculated list chinlist (n, i) is sent to the area correction unit 218.
[0226]
The region correcting unit 218 performs the same processing as in the above-described equation (20) in the first embodiment on each skin color region having vertex coordinates that are not invalid in the vertex list V (n) corrected by the vertex detecting unit 213. The rectangular area surrounding the flesh-color area n is calculated by the method described above. However, in the present embodiment, since there are a plurality of chin height candidates chinlist (i) corresponding to the eye height candidates, a rectangular area is calculated for each chin height candidate, The vertex coordinates are stored in the vertex list V ′ (n, i) by the same method as shown in the above equation (6). Here, i represents the priority of the eye height candidate corresponding to the chin height candidate. The calculated new vertex list V ′ (n, i) is sent to the determination unit 219. If the vertex coordinates stored in the vertex list V (n) corresponding to the skin color area n are invalid, the new vertex list V ′ (n, i) is invalid for all the priorities i. It is assumed that various coordinate values are stored.
[0227]
The determination unit 219 determines whether or not each skin color region having a vertex coordinate that is not invalid in the vertex list V ′ (n, i) calculated by the region correction unit 218 is a face region. And outputs a binary flag faceflag indicating the result of the determination, but the determination method is different from that of the first embodiment. In the first embodiment, since one eye height HOE and one mouth height HOM are respectively detected, there is only one combination, and the determination unit 19 in the first embodiment determines It is determined whether or not this one combination is a face area. On the other hand, in the present embodiment, a plurality of eye height candidates are detected with respect to the eye height, and when the number of eye height candidates is set to N, the mouth height HOM There will be N combinations. The determination unit 219 determines whether or not the plurality of combinations are face areas. If any of the combinations satisfies the condition of the face area, the corresponding skin color area is determined. The eye area is determined as the face area, and the eye height candidate in the combination is set as the true eye height HOE. If there are a plurality of combinations that satisfy the condition of the face area, the eye height candidate with the highest priority i is set as the HOE. If the face area condition is not satisfied for any of the combinations, it is determined that the corresponding skin color area is not a face area.
[0228]
FIG. 21 is a block diagram illustrating the determination unit 219 according to the present embodiment. As illustrated in FIG. 21, the determination unit 219 includes an area setting unit 231 that sets left and right eye areas and a mouth area, and horizontal edge detection that calculates horizontal edges (horizontal edges) of flesh-color pixels in a rectangular area. Unit 232, an eye pattern determining unit 233 that determines whether the eye is an eye, a mouth pattern determining unit 234 that determines whether the eye is a mouth, and whether the eye height is accurate based on the height of the mouth An eye-to-mouth distance determination unit 236 that determines whether the face is a face based on the determination results of the eye pattern determination unit 233, the mouth pattern determination unit 234, and the eye-to-mouth distance determination unit 236; Consists of
[0229]
The region setting unit 231 sets the mouth region for the skin color region n from the mouth height HOM and the vertex list V ′ (n, i) in the same manner as in the first embodiment. It is stored in Vmou. On the other hand, one eye height candidate is selected from the list eyelist (n, i), and from the vertex list V ′ (n, i), the right eye area and the right eye area are selected in the same manner as in the first embodiment. A left eye region is set and stored in VeyeR and VeyeL, respectively. At this time, the selected eye height candidate has the highest priority i among those not yet used for the face area determination up to this point. The set right eye region VeyeR, left eye region VeyeL, and mouth region Vmou are sent to the eye-to-mouth distance determination unit 66, and VeyR, VeyeL are also sent to the eye pattern determination unit 233, and Vmou is also sent to the mouth determination unit 234. Can be
[0230]
The face area is determined based on the right eye area, the left eye area, and the mouth area set by the area setting unit 231. The binary flag faceflag, which is the result of the determination, is output from the comprehensive determination unit 235 described later. The process returns to the area setting unit 231. If the determination result indicates that the eye area is not a face area, the eye height candidate with the next highest priority i is selected from the eye height candidates stored in the list eyelist (n, i), and Right eye region VeyeR. The left eye area VeyeL is reset. The determination of the face area using these is repeated.
[0231]
If the eye height candidate used here is the lowest one in the list eyelist (n, i), a value of, for example, 1 is set to the binary flag lastoflist. On the other hand, when there is still an eye height candidate to be used for the determination, a value of, for example, 0 is set in the flag lastoflist. The flag lastoflist for which the value is set is sent to the overall determination unit 235 described later.
[0232]
The eye-to-mouth distance determination unit 236 determines whether or not the eye height candidate eyelist (n, i) selected by the region setting unit 231 is appropriate for the mouth height HOM, as shown in FIG. Is determined. This determination can be made, for example, by checking whether or not the following expression (42) is satisfied, using the width mouseizex of the mouth region set by the region setting unit 231.
[0233]
(Equation 42)

[0234]
Here, t1 and t2 are coefficients for determining a threshold value for the vertical distance Lme between the eye and the mouth, and appropriate values are set in advance.
[0235]
If the above expression (42) is not satisfied, it is determined that the distance Lme between the eye and the mouth is too large or too small for the mouth width mouseizex, and the eye height candidate eyelist (n, It is determined that i) is inappropriate. On the other hand, when the above expression (42) is satisfied, it is determined that the eye height candidate eyelist (n, i) is appropriate. This determination result is represented by a binary flag eyelipdst. For example, a value of 1 is determined when the eye height candidate is determined to be appropriate, and a value of 0 is determined when the eye height candidate is determined to be inappropriate. The value is set. The flag "eyelipdst" set here is sent to the overall determination unit 235 described later.
[0236]
The horizontal edge detection unit 212 can be the same as that of the first embodiment, and the generated edge map Medge is sent to the eye pattern determination unit 213 and the mouth pattern determination unit 214.
[0237]
The eye pattern determination unit 233 performs substantially the same processing as the eye pattern determination unit 23 in the first embodiment illustrated in FIG. 12, but in the present embodiment, further determines the presence or absence of glasses, and determines the determination result. Is different in that eye pattern determination is performed according to.
[0238]
When the subject wears spectacles, as shown in FIG. 23, since a portion 242a connecting the two lenses of the two eyes of the spectacle frame 242 exists between the both eyes, depending on the design of the spectacles, the horizontal edge detector 232 may be used. In, many edges are detected in the region between the eyes. However, since the first determination condition shown in the above equation (27) assumes that there are few horizontal edges existing between the eyes, if the subject is wearing glasses, the The possibility that the pattern is not correctly determined increases.
[0239]
Therefore, in the present embodiment, it is checked whether or not there is a pattern indicating the characteristics of the glasses in the peripheral portion of the right eye region and the left eye region set by the region setting unit 231. By using a smaller value as the threshold value Tedst in the above equation (27), the pattern of the eye of the subject wearing glasses can be correctly determined.
[0240]
As a feature of the eyeglasses, for example, utilizing the fact that the frame of the eyeglasses existing between both eyes is elongated in the horizontal direction, the presence of the feature is determined based on a condition represented by the following equation (43). Can be.
[0241]
[Equation 43]

[0242]
In the first inequality expression of the above equation (43), areaM is the total number of pixels existing in the region 244 between the right and left eyes, which is sandwiched between the right eye region 243R and the left eye region 243L, as shown by hatching in FIG. And ecountM is the number of pixels of the horizontal edge counted in the interocular space 244 as in the following equation (44).
[0243]
[Equation 44]

[0244]
Here, eyeM represents a set of pixels existing in the interocular region 244. In addition, esdvx and esdvy in the second inequality in Expression (43) represent the standard deviation of the horizontal coordinate x and the standard deviation of the vertical coordinate y of the horizontal edge pixel in the interocular region 244, respectively. Tglsdnsty and Tglsaspct are preset threshold values.
[0245]
From the above, the condition of the above formula (43) is that the condition that the density of the horizontal edges in the interocular region 244 is high and the horizontal edges are spread in the horizontal direction is determined by the pattern by the frame of the glasses. It is determined that the subject is wearing glasses.
[0246]
The mouth pattern determination unit 234 performs substantially the same processing as the mouth pattern determination unit 24 according to the first embodiment shown in FIG. 12, but in the present embodiment, the mouth pattern calculated by the mouth detection unit 234 is further determined. The difference is that the mouth pattern is determined according to the height reliability lipconf.
[0247]
In the mouth pattern determination unit 24 of the first embodiment, the mouth pattern is determined by the above equation (30). The third determination condition in the mouth pattern determination unit is that the redness inside the mouth area Vmou at the height of the mouth is determined. Is stronger than its surroundings. However, as shown in FIG. 25, if a reddish region exists at the same height as the mouth due to wearing of red clothes or the like, the above assumption is not always satisfied, and a correct face region may not be determined. is there.
[0248]
Therefore, in the present embodiment, when the reliability of the mouth height, lipconf, calculated when detecting the mouth height is low, the third determination condition in the above equation (30) is relaxed. As in the example of FIG. 25, when there are strong reddish areas on both sides of the face due to the clothes worn by the subject, such as wearing the strong reddish clothes 245, the strong reddish pixels are widely distributed in the horizontal direction. However, since the standard deviation of the horizontal coordinate x of the pixel with strong redness calculated by the above equation (38) becomes large, the reliability lipconf of the mouth height decreases. At this time, the mouth pattern can be correctly determined by reducing the threshold Tcdstm in the third determination condition of the above equation (30) according to, for example, lipconf as shown in the following equation (45). .
[0249]
[Equation 45]

[0250]
In the overall judgment unit 235, the binary flags eyelipdst, eyeflag, and mouflag sent from the eye-to-mouth distance judgment unit 236, the eye pattern judgment unit 233, and the mouth pattern judgment unit 234 are all 1; If it is determined that the eye region and the mouth region set by the region setting unit 231 have eye characteristics and mouth characteristics, respectively, a value of 1, for example, is set to a binary flag faceflag representing the determination result. In addition to setting and outputting, the face area determination processing for the current skin color area n is completed, and the determination of the face area for the next skin color area n + 1 is started.
[0251]
On the other hand, if any of the flags of eyelipdst, eyeflag, and mouflag is not 1, a value of, for example, 0 is set to the flag faceflag. At this time, referring to the flag lastoflist sent from the area setting unit 231, if the value is 1, that is, if the used eye height candidate is the lowest one in the list eyelist (n, i) Determines that the current skin color area n is not a face area, outputs a faceflag set to, for example, 0, completes the face area determination processing for the current skin color area n, and returns to the next skin color area n + 1. The determination of the face area is started.
[0252]
On the other hand, when the flag (control signal) lastoflist is 0, the faceflag as the determination result is returned to the area determination unit 231 and the same skin color area is selected using the eye height candidate having the next highest priority i. The determination process of the face area for n is continued.
[0253]
In this embodiment, as in the first embodiment, the coefficient S1 for determining the size of the area in the above equations (21) and (23) is used in order to avoid the problem of connection between the face and the skin color area. , S2, S3 are prepared in advance, and the determination can be made for each value.
[0254]
In the present embodiment, when the top of the head is not included in the color image, it can be determined that the face area of the person has not been properly photographed, and the photographing can be prompted again.
[0255]
In addition, since the center line for dividing the face into left and right is calculated from the spatial distribution of the pixels constituting the skin color area, and the height of the mouth is detected from the intensity of redness around the center line, the clothes with a strong reddish color Even if you are wearing, you can accurately detect the height and center line of the mouth, calculate the reliability of the detected mouth height, and if this reliability is high, use robust estimation Since the center line position is corrected, the center line can be accurately detected even when the skin color area is missing due to the influence of a shadow or the like.
[0256]
Furthermore, by considering the reliability of the height of the mouth, it is possible to more accurately evaluate the likelihood of the mouth pattern.Also, a plurality of eye height candidates are calculated, and for each eye height candidate, By determining the likeness of the eye pattern and determining the presence or absence of wearing glasses, it is possible to more accurately determine the likeness of the eye pattern even when wearing glasses, and it is possible to make a mistake in the detection of eye height. Detection can be reduced.
[0257]
Furthermore, since the distance between the eye and the mouth with respect to the width of the mouth is evaluated and reflected on the likeness of the mouth pattern and the likeness of the eye pattern, it is possible to more accurately determine the face area.
[0258]
Needless to say, the image processing apparatus according to the present embodiment can be applied to the portrait image photographing apparatuses according to the second and third embodiments described above.
[0259]
【The invention's effect】
As described in detail above, according to the image processing method of the present invention, a skin color region extraction step of extracting a skin color region from an input color image, and a top portion of the person based on the skin color region, A feature point detecting step of detecting one or more feature points selected from the group consisting of a mouth, the eyes of the person, and the chin of the person; and, based on the feature points, whether the skin color area is a face area And determining the facial region based on the extracted flesh color region, and determining whether the flesh color region is a face based on this characteristic point. As a result, erroneous detection can be reduced, a face region can be extracted quickly and accurately from an input color image, and an accurate determination can be made because it is determined whether or not a face is based on a feature point.
[0260]
Further, according to the portrait photographing apparatus according to the present invention, in the portrait photographing apparatus for photographing a person to create a portrait photograph, a flesh color region extracting means for extracting a flesh color region from an input color image; Feature point detecting means for detecting one or more feature points selected from the group consisting of the top of the person, the mouth of the person, the eyes of the person, and the chin of the person based on the feature points, Determining means for determining whether or not the skin color area is a face area; a face area extracting means for extracting the face area; and a face to be trimmed from the face area extracted by the face area extracting means. Selecting means for selecting an area, and positioning means for performing trimming positioning from the selected face area, extracting the person's face area from the color image and setting the face area to a predetermined position As described above, the automatic trimming means for automatically trimming the face area allows the face area extracting means to quickly and accurately extract the face area from the input color image and accurately determine whether the face area is a face. Since the trimming process is performed by using the information on the skin color region and each feature point detected by the face region extracting means, it is possible to accurately perform the automatic trimming process. It is possible to shoot with.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an image processing apparatus 10 according to a first embodiment of the present invention.
FIG. 2 is a schematic diagram showing extracted flesh color pixels.
FIG. 3 is a schematic diagram showing a closed area surrounding a skin color area.
FIG. 4 is a schematic diagram showing vertex coordinates of a rectangular area.
FIG. 5 is a schematic diagram showing a search range when searching for the top of a person in a color image.
FIG. 6 is a schematic diagram showing a relationship between a rectangular area and a histogram Hrdsh generated by accumulating horizontal redness intensity of the rectangular area.
FIG. 7 is a diagram showing a differential coefficient of a differential filter used when calculating the intensity of a horizontal edge.
FIG. 8 is a schematic diagram showing the relationship between the positions of the eyes, mouth, and chin of a person.
9 is a schematic diagram illustrating a relationship between a histogram Hedge (y) generated by accumulating pixels constituting an edge in a horizontal direction and a rectangular area 72 corresponding to a skin color area 71. FIG.
FIG. 10 is a schematic diagram showing a mouth height HOM and search ranges mtop and mbtm in a rectangular area corresponding to a skin color area.
FIG. 11 is a schematic diagram showing vertex coordinates {(stx, sty), (edx, edy)} of a rectangular area after correction.
FIG. 12 is a block diagram illustrating a determination unit of the image processing device according to the first embodiment of the present invention.
FIG. 13 is a schematic diagram showing regions of the right eye, the left eye, and the mouth estimated from the mouth height HOM and the eye height HOE and set in the rectangular region indicated by the vertex coordinates V (n). .
FIG. 14 is a diagram showing a differential coefficient of a differential filter used when calculating a vertical edge intensity.
FIG. 15 is a block diagram showing a portrait image photographing apparatus according to a second embodiment of the present invention.
FIG. 16 is a schematic diagram showing a cutout position of a partial image cut out from an input color image.
FIG. 17 is a schematic diagram showing a top search range when a plurality of persons 121 and 122 exist in the same input color image 120 and the persons 121 and 122 overlap in the horizontal direction.
FIG. 18 is a block diagram illustrating an image processing device according to a fourth embodiment of the present invention.
FIG. 19 is a schematic diagram showing a cumulative range of redness intensity in a rectangular area.
FIG. 20 is a schematic diagram showing a rectangular region Rm set for calculating reliability of a mouth height.
FIG. 21 is a block diagram illustrating in detail a determination unit of an image processing apparatus according to a fourth embodiment of the present invention.
FIG. 22 is a schematic diagram for explaining a determination method in an eye-mouth distance determination unit that determines whether an eye height candidate eyelist (n, i) is appropriate for a mouth height HOM. It is.
FIG. 23 is a schematic diagram showing a subject wearing glasses.
FIG. 24 is a schematic diagram showing a region between both eyes sandwiched between a right eye region and a left eye region.
FIG. 25 is a schematic diagram showing a case where a reddish region is present at the same height as a mouth due to wearing of red clothes or the like.
[Explanation of symbols]
10,210 {image processing device, 11, 211} image input unit, 12,212 {skin color region extraction unit, 13,213} crown detection unit, 14,214} mouth detection unit, 15,215 {eye detection unit, 16,216} jaw detection unit , 17,217} center line detecting unit, 18,218} region correcting unit, 19,219} determining unit, 21,231} region setting unit, 22,232} horizontal edge detecting unit, 23,233 {eye pattern determining unit, 24,234} mouth pattern Judgment unit, 25,235 total judgment unit, 30 portrait image pickup device, 31 image pickup unit, 32 face detection unit, 33 region selection unit, 34 unit trimming unit, 220 center line correction unit, {236} inter-eye distance judgment unit

Claims (60)

In an image processing method for extracting a face area of a person from an input color image,
A skin color region extraction step of extracting a skin color region,
A feature point detecting step of detecting a position of one or more feature points selected from the group consisting of the top of the person, the mouth of the person, the eyes of the person, and the chin of the person based on the skin color region;
A determining step of determining whether or not the skin color area is a face area based on the positions of the skin color area and the feature points. 2. The method according to claim 1, wherein the feature point detecting step includes a mouth detecting step of detecting a vertical position coordinate as a mouth position based on a horizontal redness intensity distribution in the closed area including the skin color area. Image processing method. The feature point detecting step includes a mouth detecting step of detecting, as a mouth position, a vertical position coordinate having a largest cumulative value of a horizontal redness intensity in a closed region including the skin color region. 2. The image processing method according to 2. The redness intensity is a value obtained by integrating the ratio of R to G and the ratio of B to G when G is larger than B, where R, G, and B are the values of the three primary colors in each pixel. 3. The image processing method according to claim 2, wherein when the value is smaller than B, a value obtained by integrating a ratio of R to B and a ratio of G to B is used. The image processing according to claim 2, wherein in the mouth detection step, the position of the mouth is detected using only the redness intensity of a pixel that is present in the closed area and does not belong to the skin color area. Method. 3. The image processing method according to claim 2, wherein in the mouth detection step, the redness intensity distribution is obtained by differently weighting the redness intensity according to the position of the pixel in the closed area. 3. The image processing method according to claim 2, wherein in the mouth detecting step, the redness intensity distribution is obtained by differently weighting the redness intensity according to the brightness of the pixel in the closed area. The feature point detection step includes an eye detection step of detecting a vertical position coordinate as an eye position based on a horizontal edge intensity distribution in the closed region including the skin color region. 2. The image processing method according to 1. The feature point detecting step includes an eye detecting step of detecting, as an eye position, vertical position coordinates in which the horizontal cumulative value of the strength of the horizontal edge in the closed area is the largest. Item 10. The image processing method according to Item 8. The feature point detecting step is a head detecting step of detecting a vertical coordinate position at the top of the person, and a vertical position coordinate as a position of a mouth based on a horizontal redness intensity distribution in the closed area. Having a mouth detection step of detecting,
10. The eye detection step according to claim 9, wherein an eye search range is set based on the position of the mouth and the position of the crown, and the position of the eye is detected from within the search range. Image processing method. In the crown detection step, when the background of the person is a single color, the area above the flesh color area is scanned downward from the top of the input color image, and the position where a pixel of a color different from the background color appears is determined. The image processing method according to claim 10, wherein the position is detected as the position of the crown. 12. The image processing method according to claim 11, wherein in the scanning from the top of the input color image downward, the background color updated as needed and the color of each pixel are compared. In the crown detection step, a pixel-by-pixel difference between the image of only the background that does not include the person and the image that includes the person is calculated. 11. The image processing method according to claim 10, wherein the coordinates of the pixel existing in the image are detected as the position of the top of the head. The feature point detection step has a top detection step of detecting a vertical coordinate position at the top of the person,
The head detection step includes a head determination step of determining whether the head is present in the input color image, and determining whether the skin color area is a face area based on the head determination result. The image processing method according to claim 1, wherein the determination is made. In the top-of-top determination step, the variance of pixel values in a predetermined number of scanning lines in a region above the skin color region is calculated downward from the top of the input color image, and the variance is smaller than a predetermined threshold. 15. The image processing method according to claim 14, wherein when the size is large, it is determined that the crown is not present in the input color image. In the feature point detection step, when it is determined in the top part determination step that the top part does not exist in the input color image, the top of the input color image is set to the position of the top part. The image processing method according to claim 14, wherein: In the determination step, when it is determined in the crown determination step that the crown is not present in the input color image, it is determined that the skin color area is not a face area. 15. The image processing method according to 14. 3. The image processing method according to claim 2, wherein the feature point detecting step includes a center line detecting step of detecting a position of a center line dividing the face of the person into right and left based on the position of the mouth. 19. The center line detecting step according to claim 18, wherein a barycentric position of the pixel having the large reddish intensity at coordinates near the position of the mouth is calculated, and a horizontal position coordinate of the barycentric position is detected. Image processing method. In the center line detecting step, the coordinates near the position of the mouth do not belong to the skin color area, and the redness intensity calculates the barycentric position of a pixel of a predetermined threshold or more, the horizontal position coordinates of the barycentric position 19. The image processing method according to claim 18, wherein the detection is performed. 20. The image processing method according to claim 19, wherein in the center line detecting step, the position of the center of gravity is calculated using robust estimation. The feature point detecting step includes a center line detecting step of detecting a position of a center line that divides a face into right and left from a spatial distribution of pixels forming the skin color region,
3. The image processing method according to claim 2, wherein in the mouth detecting step, the position of the mouth is detected by performing weighting on the redness intensity according to a distance from the center line. In the mouth detection step, from the spatial distribution of redness intensity of the pixel in the vicinity of the detected mouth position, calculate the reliability of the position of the mouth, when the reliability is equal to or more than a predetermined threshold, The image processing method according to claim 22, wherein the position of the center line is corrected. 24. The image processing method according to claim 23, wherein the reliability of the position of the mouth becomes smaller as the spatial distribution of pixels whose redness intensity in the vicinity of the detected position of the mouth is equal to or larger than a predetermined threshold is larger. . The spatial distribution of pixels whose redness in the vicinity of the detected mouth position is equal to or greater than a predetermined threshold is calculated by weighting the position coordinates of the pixel in accordance with the brightness of the pixel. The image processing method according to claim 24, wherein The image processing method according to claim 23, wherein a robust estimation is used for correcting the position of the center line. In the determining step, the likelihood of an eye pattern is evaluated based on the distribution state of pixels in which the horizontal edge in the region near the eye position is equal to or greater than a predetermined threshold, and the horizontal edge in the region near the mouth position is evaluated. Evaluate the likelihood of the mouth pattern based on the distribution state of the pixels and the distribution state of the color that is equal to or more than a predetermined threshold, and when the evaluation of the likeness of the eye pattern and the evaluation of the likeness of the mouth is equal to or more than a predetermined threshold, the skin color 9. The image processing method according to claim 8, wherein the area is determined to be a face area. 28. The image processing method according to claim 27, wherein the determining step includes a spectacle detecting step of detecting the presence or absence of spectacles, and evaluating a pattern likeness of an eye according to the spectacle detection result. A right-eye area and a left-eye area are set near the position of the eye, and a pixel whose horizontal edge in the right-eye area and the left-eye area is equal to or larger than a predetermined threshold is a right-eye area near the position of the eye. 29. The image processing method according to claim 28, wherein the evaluation of the likeness of the eye pattern increases when the number of horizontal edges in an area other than the left eye area is greater than a predetermined threshold value or more. . In the spectacles detecting step, the density of horizontal edges in a region between both eyes sandwiched between the right eye region and the left eye region is equal to or higher than a predetermined threshold, and 30. The image processing method according to claim 29, wherein the presence of glasses is detected when the horizontal distribution of the pixels constituting the horizontal edge equal to or larger than the threshold is equal to or larger than a predetermined threshold. The image processing according to claim 27, wherein the evaluation of the likeness of the eye pattern increases when the pixel density at which the horizontal edges in the right eye region and the left eye region are equal to or larger than a predetermined threshold is high. Method. The right eye region and the left eye region, and when the pixel density at which the horizontal edge in the interocular region between the right eye region and the left eye region is greater than or equal to a predetermined threshold is large, The image processing method according to claim 27, wherein the evaluation of the pattern likeness is increased. In the eye detection step, one or more eye position candidates ranked as the eye positions are detected,
28. The image processing method according to claim 27, wherein, in the determining step, the pattern likeness of an eye is calculated from the one or more eye position candidates having a higher rank. 34. The image processing method according to claim 33, wherein the order of the eye position candidates increases as the vertical coordinate with the greater horizontal cumulative value of the horizontal edge strength increases. A mouth region is set near the position of the mouth, and a pixel whose horizontal edge in the mouth region is equal to or larger than a predetermined threshold is a predetermined pixel in the region other than the mouth region near the position of the mouth. 28. The image processing method according to claim 27, wherein the evaluation of the likelihood of the mouth pattern increases when the number of pixels is larger than the threshold value equal to or larger than the threshold value. A mouth region is set near the position of the mouth, and when the horizontal edge in the mouth region is equal to or more than a predetermined threshold and the pixel density is high, the evaluation of the likeness of the mouth pattern becomes large. An image processing method according to claim 27. 28. The image processing method according to claim 27, wherein a mouth area is set near the position of the mouth, and when the redness intensity in the mouth area is large, the evaluation of the likeness of the mouth pattern is increased. A mouth region is set in the vicinity of the position of the mouth, and the redness intensity in the mouth region is larger than the redness intensity in a region other than the mouth region, whereby the evaluation of the likeness of the mouth pattern is increased. 28. The image processing method according to claim 27. In the mouth detection step, from the spatial distribution of redness intensity of the pixel in the vicinity of the detected position of the mouth, the reliability of the position of the mouth is calculated,
28. The image processing method according to claim 27, wherein in the determining step, the likeness of the mouth pattern is evaluated according to the reliability of the position of the mouth. When the reliability of the position of the mouth is less than a predetermined threshold, a value indicating a relationship between the redness intensity in the mouth region and the redness intensity in other regions near the position of the mouth is a value of the pattern likeness of the mouth. The image processing method according to claim 39, wherein a degree of contribution to the evaluation is reduced. In the determination step, if the distance between the position of the eye and the position of the mouth is greater than the upper limit calculated from the width of the mouth region, or from the lower limit calculated from the width of the mouth region The image processing method according to claim 1, wherein when the value is also small, the likelihood of the eye pattern and the likeness of the mouth pattern are evaluated to be small. In the vicinity of the position of the eye and in the vicinity of the position of the mouth, a plurality of right eye regions and left eye regions and a mouth region each having a different size are set, and the plurality of right eye regions and left eye regions and the mouth region are respectively set. The image processing method according to claim 27, wherein the likeness of the eye pattern and the likeness of the mouth pattern are calculated. 2. The method according to claim 1, further comprising a selecting step of selecting one or more face areas from the plurality of face areas when the plurality of skin color areas are determined to be face areas in the determining step. Image processing method. The image processing method according to claim 43, wherein in the selecting step, one face area is selected based on the positions of the plurality of face areas. In an image processing apparatus for extracting a face area of a person from an input color image,
A skin color region extracting means for extracting a skin color region;
Feature point detecting means for detecting the position of one or more feature points selected from the group consisting of the top of the person, the mouth of the person, the eyes of the person, and the chin of the person based on the skin color region;
Determining means for determining whether or not the skin color area is a face area based on the positions of the skin color area and the feature points. In a program for executing an operation of extracting a person's face region from an input color image,
A skin color region extraction step of extracting a skin color region,
A feature point detecting step of detecting a position of one or more feature points selected from the group consisting of the top of the person, the mouth of the person, the eyes of the person, and the chin of the person based on the skin color region;
A determination step of determining whether the skin color area is a face area based on the positions of the skin color area and the feature points. From the input color image, in a computer-readable recording medium that has recorded a program for causing a computer to perform an operation of extracting a person's face region,
A skin color region extraction step of extracting a skin color region,
A feature point detecting step of detecting a position of one or more feature points selected from the group consisting of the top of the person, the mouth of the person, the eyes of the person, and the chin of the person based on the skin color region;
A determination step of determining whether or not the skin color area is a face area based on the positions of the skin color area and the feature points. An automatic trimming device that extracts a face area of a person from an input color image and performs trimming so that the face area is located at a predetermined position.
Skin color region extracting means for extracting a skin color region; and one or more feature points selected from the group consisting of the top of the person, the mouth of the person, the eye of the person, and the chin of the person based on the skin color region Characteristic point detecting means for detecting the position of the skin color area, a determining means for determining whether or not the skin color area is a face area based on the positions of the characteristic points, and a determining means for determining that the skin color area is a face area. Face area extracting means, comprising: extracting means for extracting a closed area including a flesh-colored area as a face area,
Selecting means for selecting a face area to be trimmed from the face area extracted by the face area extracting means;
An automatic trimming device comprising: positioning means for positioning trimming from the selected face area. The face area extraction unit detects a position of a face center line that divides the face of the person into left and right based on the positions of the feature points,
49. The automatic trimming device according to claim 48, wherein the positioning means performs trimming positioning based on the position of the crown and the center line of the face. The face area extraction unit detects a position of a face center line that divides the face of the person into left and right based on the positions of the feature points,
49. The automatic trimming device according to claim 48, wherein the positioning means performs trimming positioning based on the position of the eye of the person and the center line of the face. The face area extraction unit detects a position of a face center line that divides the face of the person into left and right based on the positions of the feature points,
49. The automatic trimming device according to claim 48, wherein the positioning means performs trimming positioning based on the position of the mouth of the person and the center line of the face. 49. The automatic trimming device according to claim 48, wherein the position of the chin of the person is calculated from the positions of the eyes and mouth of the person. The face area extraction unit detects a position of a face center line that divides the face of the person into left and right based on the positions of the feature points,
The positioning means is configured to perform trimming based on a position of at least one feature point selected from a group consisting of a top portion of the person, a mouth of the person, eyes of the person, and a chin of the person, and the face center line. 49. The automatic trimming device according to claim 48, wherein positioning of the trimming is performed. 49. The automatic trimming device according to claim 48, wherein the selecting unit selects a trimming target based on a position of the face area on the color image. 49. The automatic trimming device according to claim 48, wherein the selecting unit selects the face area closest to the center in the color image as a target to be trimmed. 49. The automatic trimming device according to claim 48, wherein the selecting unit selects the face region located at the uppermost position in the color image as a trimming target. 49. The automatic trimming device according to claim 48, wherein the selecting means selects the lowermost face area in the color image as an object to be trimmed. 49. The automatic trimming device according to claim 48, wherein said selecting means selects all of the extracted face regions as targets for trimming. 49. The automatic trimming device according to claim 48, wherein the positioning unit performs an enlargement or reduction process for setting a face area to a predetermined position and size in the image after trimming. In a portrait photographing apparatus that creates a portrait photograph by photographing a person,
A skin color region extracting means for extracting a skin color region from an input color image; and 1 selected from the group consisting of the top of the person, the mouth of the person, the eye of the person, and the chin of the person based on the skin color region Feature point detecting means for detecting the position of the feature point, determining means for determining whether or not the skin color area is a face area based on the skin color area and the position of the feature point; Face area extracting means for extracting a closed area including a skin color area determined to be a face area as a face area; and a face area to be trimmed from the face area extracted by the face area extracting means. And a positioning means for performing trimming positioning from the selected face area. The face area of the person is extracted from the color image, and the face area becomes a predetermined position. Uni automatically portrait photographing apparatus characterized by having an automatic trimming means for trimming.

Images