JP2004030007A - Makeup simulation apparatus, makeup simulation method, makeup simulation program and recording medium with program recorded thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform more real makeup simulation by confirming a makeup effect upon a change in the expression of the face with a moving image as a target. <P>SOLUTION: The makeup simulation apparatus is provided with: an image acquiring part 100 for acquiring a face image of a tester for each prescribed frame cycle; a position detecting part 610 for detecting positions of eyes in the face image for each frame; a makeup zone setting part 620 for setting a prescribed zone of the face image as a makeup zone with the positions of the eyes detected for each frame as a reference; and a makeup processing part 630 for applying makeup processing to the face image. <P>COPYRIGHT: (C)2004,JPO

Description

【００４９】
ここで、式（１）において、ｙは、顔画像のｙ方向の任意の位置を示す変数であり、Ｔｐは定数であり、その値は一般人の平均的な瞳の半径とほぼ同じ値である。顔画像中の眼、特に瞳の部分は周囲の肌色領域及び白眼領域よりも相対的に暗い。そこで、瞳抽出フィルタｈｐ（ｙ）として、図５に示す横軸の値が０に近づくにつれてフィルタ値が０に近づき、原点から横軸に対して定数Ｔｐ及び−Ｔｐに近づくにつれてフィルタ値が１に近づくような、瞳の抽出に適した１次元の関数が採用されている。
【００５０】
眼輪郭抽出フィルタｈｃ（ｙ）は、式（２）のように表される。
【００５１】
【数２】

【００５２】
式（２）は、式（１）とほぼ同じ式であるが、定数Ｔｃの大きさが式（１）の定数Ｔｐの大きさと異なる。すなわち、眼輪郭抽出フィルタｈｃ（ｙ）は、周囲の肌色の画像に対して相対的に暗い曲線状の眼の輪郭が適切に抽出されるように、すなわち、暗くかつ細いライン状の領域を抽出することができるように定数Ｔｃの値が設定されている。そのため、定数Ｔｃは定数Ｔｐよりも小さい値（Ｔｃ＜Ｔｐ）が設定されている。
【００５３】
フィルタ処理部６１２３は、第１及び第２の検出領域Ｒ１及びＲ２内の画像のｙ方向の１ライン分の画像ｆ（ｙ）に対して瞳抽出フィルタｈｐ（ｙ）を用いて式（３）に示す畳み込み積分を施した後、更にｘ方向に隣接する次のｙ方向の１ライン分の画像ｆ（ｙ）に対して瞳抽出フィルタｈｐ（ｙ）を用いた畳み込み積分を施すというように、第１及び第２の検出領域Ｒ１及びＲ２内のｙ方向の画像ｆ（ｙ）に対して瞳抽出フィルタｈｐ（ｙ）を用いた畳み込み積分を順番に施していき瞳画像を抽出する。
【００５４】
【数３】

【００５５】
さらに、フィルタ処理部６１２３は、式（４）に示すように、ｙ方向の１ライン分の画像ｆ（ｙ）に対して、眼輪郭抽出フィルタｈｃ（ｙ）を用いた畳み込み積分を順番に施していき、眼の輪郭画像を抽出する。そして、抽出した瞳画像と眼の輪郭画像とを合成し、眼の画像を抽出する。
【００５６】
ここで、フィルタ処理部６１２３は、第１及び第２の検出領域Ｒ１及びＲ２の画像に対して、１次元のフィルタ処理を施しているが、画像取得部１００は、眼の中心と口の中心とを結ぶ直線がｙ方向となるように顔画像を取得しているため、眼の輪郭線は多くの部分がｘ方向とほぼ平行に現れることとなり、ｙ方向の１ライン分の画像ｆ（ｙ）に対して眼輪郭抽出フィルタｈｃ（ｙ）を用いた畳み込み積分を施せば、眼の輪郭画像を抽出することができる。その結果、計算量の多い２次元のフィルタ処理を施さなくとも、１次元のフィルタ処理により眼の輪郭画像を抽出することができる。
【００５７】
面積算出部６１２４は、フィルタ処理部６１２３によってフィルタ処理が施された第１及び第２の検出領域Ｒ１及びＲ２の画像に対して２値化処理を施す。上述したようにフィルタ処理部６１２３は、第１及び第２の検出領域Ｒ１及びＲ２に対して、眼輪郭抽出フィルタｈｃ（ｙ）及び瞳抽出フィルタｈｐ（ｙ）を用いて、眼の画像を抽出しているが、これらのフィルタは、肌色に対して相対的に暗い画素を抽出する特性を有しているため、眉、鼻の脇及び眼の下のくまなども検出する場合がある。そのため、フィルタ処理部６１２３は、複数の眼の候補画像を抽出することとなる。そこで、面積算出部６１２４は、画像判定部６１２５が面積を基に、眼の画像を判定することができるように、２値化処理を施した後、各候補画像のそれぞれをラベリングし、各候補画像のドット数をカウントすることにより各候補画像の面積を算出する。
【００５８】
画像判定部６１２５は、面積算出部６１２４が算出した各候補画像のうち、面積の大きな２つ候補画像を眼の画像として判断し、複数の候補画像の中から眼の画像を特定する。ただし、面積の比較だけでは、眉を示す候補画像も大きな面積を有しているため、誤ってこの画像を眼の画像として特定する場合がある。そこで、画像判定部６１２５は、ｙ方向の上下に面積の大きな２つの候補画像を検出した場合、上側の候補画像を眉の画像として排除し、下側の画像を眼の画像として特定する。
【００５９】
位置特定部６１３は、画像判定部６１２５が特定した両眼の画像のそれぞれの重心を算出し、両眼のそれぞれの位置を特定する。位置特定部６１３は、眼の中心を算出するにあたり、例えば眼の画像に外接する四角形を当てはめこの四角形の２本の対角線の交点を眼の重心として算出する。
【００６０】
予測位置算出部６１４は、位置特定部６１３が過去フレームにおいて検出した両眼のそれぞれの位置をもとに眼の移動速度及び移動加速度を算出し、算出した移動速度及び移動加速度からテイラー展開にしたがって現フレームの眼の予測位置を算出する。現フレーム対して１つ前のフレームにおける眼の位置をｘ（ｔ）、ｙ（ｔ）とすると眼の移動速度はｘ（ｔ）及びｙ（ｔ）の１階微分ｘ’（ｔ）及びｙ’（ｔ）で表され、眼の移動加速度はｘ（ｔ）、ｙ（ｔ）の２階微分ｘ’’（ｔ）、ｙ’’（ｔ）で表される（ただし、ｔは時間を表す変数）。したがって、各フレームの時間間隔をΔｔとすると予測位置ｘ（ｔ＋Δｔ）、ｙ（ｔ＋Δｔ）は、式（５）で示すように、２次の項までのテイラー展開を行った式で表すことができる。
【００６１】
【数４】

【００６２】
判断部６１５は、位置特定部６１３が特定した現フレームの眼の位置と、予測位置算出部６１４が算出した現フレームの眼の予測位置との差ｄを求め、この差ｄを基に、予め定められた信頼度関数を用いて信頼度を算出し、位置特定部６１３が特定した眼の位置の正確性を判断する。信頼度関数は、図７に示すような例えばｅｘｐ（−ｄ^２）で表される関数であり、差ｄが０に近づくにつれて信頼度が高くなり、差が大きくなるにつれて信頼度が低くなるような関数である。さらに、判断部６１５は、信頼度が例えば所定の値以上であれば、位置特定部６１３が検出した眼の位置を正しい眼の位置として採用し、信頼度が所定の値以下であれば、位置特定部６１３が眼の位置を誤検出したとして、その位置を眼の位置として採用しない。なお、信頼度関数は、指数関数に限定されず、例えば二次関数や１次関数などでもよい。
【００６３】
濃淡画像変換部６１６は、画像取得部１００が取得した顔画像のＲ（赤）、Ｇ（緑）及びＢ（青）色の成分に対してそれぞれ異なる所定の重みづけを施して合成することにより、その顔画像を輝度成分のみを表す濃淡画像に変換し、画像メモリ２００に記憶させる。上記検出領域設定部６１１〜判断部６１５は、この濃淡画像に対して種々の処理を行う。
【００６４】
次に、位置検出部６１０が行う眼位置検出モードの処理ついて図８に示すフローチャートに従って説明する。
【００６５】
まず、画像取得部１００によって、顔画像の１枚目のフレームが取得される（ステップＳ１０１）。次いで、濃淡画像変換部６１６により、画像取得部１００が取得した顔画像が濃淡画像に変換される（ステップＳ１０２）。次いで、検出領域設定部６１１により、顔画像上に第１の検出領域Ｒ１が設定される（図３参照）（ステップＳ１０３）。次いで、濃度補正部６１２１によって、第１の検出領域Ｒ１の濃度値ヒストグラムが平坦化される（ステップＳ１０４）。
【００６６】
次いで、サブサンプリング部６１２２により、フィルタ処理部６１２３の計算量の低下させるために、第１の検出領域Ｒ１の画像に対して間引き処理が施される（ステップＳ１０５）。
【００６７】
次いで、フィルタ処理部６１２３により、顔画像から第１の検出領域Ｒ１が取り出され、取り出された第１の検出領域Ｒ１に対して、瞳抽出フィルタｈｐ（ｔ）を用いたフィルタ処理が施され（ステップＳ１０６）瞳の画像が抽出される。この場合、図９（ａ）で示すような瞳の画像が抽出される。次いで、フィルタ処理部６１２３により、再度、顔画像から第１の検出領域Ｒ１が取り出され、取り出した第１の検出領域Ｒ１の画像に対して、眼輪郭抽出フィルタｈｃ（ｔ）を用いたフィルタ処理が施され、眼の輪郭の画像が抽出される（ステップＳ１０７）。この場合、図９（ｂ）で示すような眼の輪郭の画像が抽出される。次いで、フィルタ処理部６１２３により、ステップＳ１０６で抽出された瞳の画像とステップＳ１０７で抽出された眼の輪郭との画像が合成され、眼の画像が抽出される（ステップＳ１０８）。この場合、図９（ｃ）で示すような瞳及び眼の輪郭が合成された眼の画像が抽出される。なお、実際には、ステップＳ１０８の段階において、眼以外の複数の眼の候補画像が含まれているが、図９（ａ）〜（ｃ）においては、これら眼以外の眼の候補画像を省略している。
【００６８】
次いで、面積算出部６１２４により、フィルタ処理部６１２３がフィルタ処理した第１の検出領域Ｒ１に対して、２値化処理が施され、複数の眼の候補画像がラベリングされ（ステップＳ１０９）、ドット数に基づいて各候補画像の面積が算出される。次いで、画像判定部６１２５によって、複数の眼の候補画像の中から２つの面積の大きな候補画像が抽出可能な場合（ステップＳ１１０でＹＥＳ）、この候補画像が両眼の画像として判定され、ステップＳ１１１に進む。一方、ステップＳ１１０において、面積の大きな２つの領域を抽出することができない場合（ステップＳ１１０でＮＯ）、再処理を行うためステップＳ１０３に戻る。なお、画像判定部６１２５が、ｙ方向の上下に２つの面積の大きな領域を抽出した場合（すなわち、この場合、眼の候補画像としてふさわしい面積を有する眼の候補画像が少なくとも３つ存在する）、上側の画像は眉の画像として排除され、下側の画像が眼の画像として特定される。
【００６９】
次いで、位置特定部６１３によって、眼の画像の重心が算出され、その中心が眼の位置として検出される（ステップＳ１１１）。次いで、顔画像の次のフレームが取得される（ステップＳ１１２）。次いで、取得されたフレームが所定枚数目未満のフレームである場合（ステップＳ１１３でＮＯ）、ステップＳ１０２に戻る。一方、取得したフレームが所定枚数目のフレームである場合（ステップＳ１１３でＹＥＳ）、図１０に示すステップＳ２０１に進む。ここで、所定枚数目とは、眼追跡モードにおいて、眼の位置の移動速度及び移動加速度を算出することができる十分な枚数が該当する。したがって、眼の位置の移動速度及び加速度を算出するためには、少なくとも３枚のフレームが必要であるため、所定枚数目のフレームは少なくとも４枚目以降のフレームとなる。
【００７０】
次に、眼追跡モードにおける位置検出部６１０が行う処理について図１０に示すフローチャートに従って説明する。眼追跡モードは、眼検出モードのステップＳ１１３の処理に引き続いて行われる。
【００７１】
まず、図８に示すステップＳ１１２で取得されたフレームが濃淡画像に変換される（ステップＳ２０１）。次いで、検出領域設定部６１１により、過去のフレームで検出された眼の位置に基づいて、図３に示すように現フレーム上に第２の検出領域Ｒ２が設定される（ステップＳ２０２）。次いで、ステップＳ２０３〜ステップＳ２０８おいて、図８に示す眼検出モードのステップＳ１０４〜Ｓ１０９で行われた処理と同様の処理が、第２の検出領域Ｒ２に対して行われる。
【００７２】
次いで、予測位置算出部６１４により、現フレームの１つ前のフレームにおける眼の位置、過去のフレームの眼の位置から算出された眼の移動速度及び移動加速度を用いて、テイラー展開によって現フレームの眼の予測位置が算出される（ステップＳ２０９）。次いで、判断部６１５によって、位置特定部６１３が現フレーム上で検出した眼の位置と、ステップＳ２０９で算出された眼の予測位置との差ｄが算出され、算出された差ｄを基に信頼度が算出され、この信頼度が所定の閾値と比較され、位置特定部６１３が検出した眼の位置の正確性が判断される（ステップＳ２１０）。ここで、算出された信頼度が所定の閾値よりも大きい場合は、位置特定部６１３が特定した眼の位置は正しいものであるとして、（ステップＳ２１０でＹＥＳ）その位置が眼の位置として決定される（ステップＳ２１１）。一方、算出された信頼度が所定の閾値よりも小さい場合は、位置特定部６１３は眼の位置を誤って特定したとして、図８で示すステップＳ１０３に戻り、再度眼の検出処理が行われる（ステップＳ２１０でＮＯ）。この場合、表示部３００には、図３で示すように、位置特定部６１３が検出した眼の位置を示すマーカＩＭを被写体の顔の動きに追従するように、瞳の中心の位置に表示させてもよい。
【００７３】
次いで、画像取得部１００により、次のフレームが取得される（ステップＳ２１２）。次いで、最終フレームの処理が終了したか否かが判断され（ステップＳ２１３）、最終フレームの処理が終了済みである場合（ステップＳ２１３でＹＥＳ）、処理を終了する。最終フレームの処理が終了していない場合（ステップＳ２１３でＮＯ）、ステップＳ２０１に戻り、ステップＳ２１２で取得されたフレームについて引き続いて眼追跡モードの処理が施される。
【００７４】
（メークアップ処理）
次に、メークアップ処理について説明する。図１１は、アイメーク領域設定部６２１のブロック構成図を示している。
【００７５】
アイメーク領域設定部６２１は、ノイズ除去部６２１１と、エッジ検出部６２１２と、近似関数算出部６２１３と、制御点作成部６２１４と、境界線算出部６２１５とを備えている。
【００７６】
ノイズ除去部６２１２は、検出領域設定部６１１が設定した第２の検出領域Ｒ２に対して、所定の周波数帯域の画像成分を通過させる空間周波数フィルタを用いて、不要な画像成分を除去する。第２の検出領域Ｒ２内には、眉間やこめかみ付近で周囲の肌色より暗い領域が存在するため、エッジ検出部６２１２は、これらの暗い領域を眼及び眉の輪郭線として誤って検出するおそれがある。そこで、ノイズ除去部６２１２は、空間周波数フィルタを用いて眼及び眉の輪郭を検出するうえで不要となる画像成分を除去している。空間周波数フィルタによるフィルタ処理方法としては、例えば離散フーリエ変換、離散コサイン変換あるいはウェーブレット変換などによるフィルタ処理方法を採用すればよい。
【００７７】
エッジ検出部６２１２は、位置特定部６１３で検出された両眼のそれぞれの位置を基に、ｙ方向上の眉及び眼の間の位置を第１の始点として設定し、第２の検出領域Ｒ２内の画像に対して、上方向（−ｙ方向）に向かって周囲の肌色よりも暗い画素の検出を所定の濃度閾値と比較することにより行い、検出した画素を眉の下側の輪郭線の画素（輪郭画素）とするとともに、下方向（＋ｙ方向）に向かって周囲の肌色よりも暗い画素の検出を、所定の濃度閾値と比較することにより行い、検出した画素を眼の上側の輪郭画素とする。エッジ検出部６２１２は、このような処理を第２の検出領域Ｒ２のｘ軸方向の一辺のほぼ真ん中の位置から左側（＋ｘ側）及び右側（−ｘ側）に向けてｙ方向の１ラインの画像を１つの処理単位として順番に施していく。このように、エッジ検出部６２１２は、眼及び眉の輪郭画素の検出を第２の検出領域のｘ方向の真ん中の位置から左右に分かれて行うため、濃度閾値より暗い画素が検出されなくなった位置が、目頭、目尻、眉頭及び眉尻の位置となる。
【００７８】
さらに、エッジ検出部６２１２は、位置特定部６１３で検出された両眼それぞれの位置を基に、眉の下側の輪郭画素及び眼の上側の輪郭画素を検出する場合と同様にして、眉の上側（ｘ軸側）の所定の位置を第２の始点として、下方向（＋ｙ方向）に向かって周囲の肌色よりも暗い画素の検出を行い、検出した画素を眉の上側の輪郭画素とする。
【００７９】
さらに、エッジ検出部６２１２は、位置特定部６１３で検出された両眼それぞれの位置を基に、眼の下側の所定の位置を第３の始点として、眉の上側の輪郭画素を検出する場合と同様にして、上方向（−ｙ方向）に向かって周囲の肌色よりも暗い画素の検出を行い、検出した画素を眼の下側の輪郭画素とする。
【００８０】
近似関数算出部６２１３は、エッジ検出部６２１２が検出した眉の上端画素の配列を基に、例えば最小二乗法を用いて眉の上側の輪郭を表す近似関数を算出する。さらに近似関数算出部６２１３は、眉の下側の輪郭画素、眼の上側の輪郭画素及び眼の下側の輪郭画素に関しても、画素の配列を基に、最小二乗近似法を用いて眉の下側の輪郭、眼の上側の輪郭及び眼の下側の輪郭を表す近似関数をそれぞれ算出する。ここで、高次の近似関数を用いて輪郭線を表すと、輪郭線の形状が不自然になるため、近似関数としては低次の関数、例えば二次関数を用いることが好ましい。
【００８１】
制御点作成部６２１４は、エッジ検出部６２１２が検出した眉尻、眉頭、目尻及び目頭の画素をｙ方向に平行移動し、これらの画素を線形補間することにより、アイメーク領域を設定するための複数の制御点を作成する。さらに、制御点作成部６２１４は、制御点の作成をするにあたり、化粧パターン記憶部４００に記憶されている複数のアイメークパターンの中からユーザーによって選択されたアイメークパターンにしたがって、制御点を作成する。
【００８２】
境界線算出部６２１５は、制御点作成部６２１４によって作成された複数の制御点のうち、いくつかの制御点を通過するという拘束条件を用いて、アイメーク領域の境界線を表す近似関数を算出する。この近似関数と眼の輪郭線とによって囲まれた領域がアイメーク領域となる。ここで、近似関数は、ｘ座標を変数とする１変数の関数となる。そのため、アイメーク領域内にある画素の判定は、境界線を表す近似関数と眼の上側の輪郭線を表す近似関数とを比較することにより容易に行うことができる。また、アイメーク領域を近似関数により特定することができるため、アイメーク領域境界の画素を容易に特定することができ、境界線付近で、アイメークの色を連続的に変化させることも可能となる。
【００８３】
次に、図１２に示すフローチャートに従って、アイメーク領域設定部６２１が１枚のフレームに対して行う処理について説明する。なお、このフレームは予め、図４に示す濃淡画像変換部６１６で濃淡画像に変換され、検出領域設定部６１１によって第２の検出領域Ｒ２が設定されているものとする。また、第２の検出領域Ｒ２が特定されるまでは、アイメーク領域設定部６２１は、第１の検出領域Ｒ１に対して以下に示す処理を行う。
【００８４】
まず、ノイズ除去部６２１１により、第２の検出領域Ｒ２に対して空間周波数フィルタを用いたフィルタ処理が施される（ステップＳ３０１）。これにより、エッジ検出部６２１２が眼及び眉の輪郭画素を検出するうえで不要となるノイズ成分が除去される。
【００８５】
次いで、エッジ検出部６２１２により、図１３に示すように、第１の始点Ｓ１から上側及び下側のそれぞれの方向に向かって眼の上側の輪郭画素ＥＰ１及び眉の下側の輪郭画素ＭＰ２が検出される（ステップＳ３０２）。次いで、エッジ検出部６２１２により、第２の始点Ｓ２から下方向に眉の上側の輪郭画素ＭＰ１が検出されるとともに、第３の始点Ｓ３から上方向に向かって眼の下側の輪郭画素ＥＰ２が検出される（ステップＳ３０３）。
【００８６】
次いで、近似関数算出部６２１３によって、ステップＳ３０２で検出された眼の上側の輪郭画素ＥＰ１及び眉の下側の輪郭画素ＭＰ２の配列を基に、最小二乗法により眼の上側の輪郭線及び眉の下側の輪郭線を表す近似関数がそれぞれ算出されるとともに、ステップＳ３０３で検出された眉の上側の輪郭画素ＭＰ１及び眼の下側の輪郭画素ＥＰ２の配列を基に、最小二乗法により眉の上側の輪郭線及び眼の下側の輪郭線を表す近似関数がそれぞれ算出される（ステップＳ３０４）。
【００８７】
次いで、エッジ検出部６２１２によって、目頭、目尻、眉頭及び眉尻の画素が検出される（ステップＳ３０５）。次いで、制御点作成部６２１４によって、目頭、目尻、眉頭及び眉尻の画素を基に、制御点が作成される（ステップＳ３０６）。この場合、図１４に示すように、目頭ＥＴが制御点ＣＰとされ、眉尻ＭＥは、例えば＋ｙ方向の位置ＭＥ’まで移動され制御点ＣＰとされる。さらに、瞳の上端の点ＨＴは−ｙ方向の位置ＨＴ’まで移動され、制御点ＣＰとされる。そして、これらの制御点ＣＰのうちいくつかの制御点ＣＰが線形補間されて新たな制御点ＣＰ’が作成される。
【００８８】
次いで、図１２に示すステップＳ３０７において、境界線算出部６２１５によって、制御点ＣＰ及びＣＰ’を基に、アイメーク領域の境界線を表す近似関数ＥＬ１が算出される（ステップＳ３０７）。この場合、図１４に示すように、近似関数ＥＬ１及び眼の輪郭ＥＬ２に囲まれた領域がアイメーク領域ＥＭＲとなる。
【００８９】
さらにこの場合、表示部３００に表示された図１５に示す画面図を用いてオペレータが選択したアイメークパターンの形に、アイメーク領域ＥＭＲの形が一致するように制御点ＣＰ及びＣＰ’を作成してもよい。図１５に示す画面図には、複数のアイメークパターンを一覧表示するアイメークパターン一覧表示部ＰＲと、顔を動画像で表示する顔画像表示部ＦＲと、アイメーク領域ＥＭＲに施す複数のアイカラーパターンを一覧表示するアイカラーパターン表示部ＣＲとが表示されている。この一覧画像は、一覧画像作成部７００が、化粧パターン記憶部４００及びカラーパターン記憶部５００に記憶されているアイメークパターン及びアイカラーパターンを、顔画像と合成することにより作成される。オペレータは、この一覧画像上で目的とするアイメークパターン及びアイカラーパターンにマウスポインタを合わせクリックすると、顔画像表示部ＦＲに表示された顔画像にそのアイメークパターン及びアイカラーパターンが施される。
【００９０】
次いで、メークアップ処理部６３０によって、アイメーク領域内ＥＭＲに、ユーザーによって選択されたアイカラーパターンが着色される（ステップＳ３０８）。以上の処理により、顔画像にアイメーク処理が施される。なお、ステップＳ３０１〜ステップＳ３０８までの処理は、各フレームに対して行われるため、動画像に対するアイメーク処理が行われることとなる。
【００９１】
次に、図１６にしたがってリップメーク領域設定部６２２の説明を行う。リップメーク領域設定部６２２は、上唇の輪郭線の検出と下唇の輪郭線の検出とをそれぞれ個別に行う。
【００９２】
リップメーク領域設定部６２２は、Ｒ−Ｇ画像変換部６２２１と、濃淡画像変換部６２２２と、唇検出領域設定部６２２３と、フィルタ処理部６２２４と、面積算出部６２２５と、下唇画像特定部６２２６と、下唇輪郭検出部６２２７と、上唇画像特定部６２２８と、上唇輪郭検出部６２２９と、近似関数算出部６２３０と、輪郭補正部６２３１とを備える。
【００９３】
６２２１は、下唇の輪郭を検出するために用いられ、顔画像からＲ（赤色）成分とＧ（緑色）成分とを抽出し、Ｒ成分からＧ成分を差し引いた差分（Ｒ−Ｇ）成分を算出し、顔画像を差分（Ｒ−Ｇ）画像に変換する。なお、この場合、Ｒ−Ｇ画像変換部６２２１は、顔画像から抽出したＲ成分とＧ成分との比率（Ｒ／Ｇ）成分を算出し、顔画像を比率（Ｒ／Ｇ）成分に変換してもよい。
【００９４】
濃淡画像変換部６２２２は、上唇の輪郭を検出をするために用いられ、顔画像に含まれるＲ、Ｇ及びＢの成分を合成して、輝度のみを表す濃淡画像に顔画像を変換する。
【００９５】
唇検出領域設定部６２２３は、位置特定部６１３が特定した眼の位置を基準として、唇を含む所定の領域を、上唇及び下唇を検出するための唇検出領域として設定する。図１７に示すように、唇検出領域設定部６２２３は、唇検出領域を設定するにあたり、一般的な人間の眼の位置ＥＰに対する口の重心の平均的な位置ＲＧを重心Ｒ３Ｇとするｘ方向に長尺な長方形状の領域を唇検出領域Ｒ３として設定する。
【００９６】
フィルタ処理部６２２４は、唇検出領域Ｒ３に対して、所定の周波数帯域のみを抽出する空間周波数フィルタを施して、上唇及び下唇を検出するにあたり不要な画像を除去する。
【００９７】
面積算出部６２２５は、唇検出領域Ｒ３を２値化処理して、下唇の複数の候補画像を抽出し、各候補画像のそれぞれをラベリングし、下唇の各候補画像のそれぞれに含まれるドット数をカウントすることにより下唇の候補画像の面積を算出する。
【００９８】
下唇画像特定部６２２６は、複数の下唇の候補画像の中から面積が最大のものを下唇の画像として特定する。
【００９９】
下唇輪郭検出部６２２７は、下唇画像特定部６２２６が判定した下唇の画像の下側の点を第４の始点として、−ｙ方向に赤色の画素の検出を行い、検出した画素を下唇の下側の輪郭画素とする。下唇輪郭検出部６２２７は、このような処理を、唇検出領域Ｒ３において、ｙ方向の１ラインを１つの処理単位として、第４の始点をｘ方向にずらしながら、順次下唇の下側の輪郭画素を検出する。
【０１００】
上唇画像特定部６２２８は、図１７に示すように、下唇画像特定部６２２６が判定した下唇画像の重心ＭＧを重心Ｒ３Ｇとする楕円領域Ｒ４（上唇検出用領域）を、唇検出領域Ｒ３内に設定し、唇検出領域Ｒ３の領域内にあって、楕円領域Ｒ４領域外にある画像を除去し、唇検出領域Ｒ３内に残存する画像を上唇の画像として特定する。
【０１０１】
上唇輪郭検出部６２２９は、上唇画像特定部６２２８が特定した上唇の画像よりも上側の所定の位置を第５の始点として、＋ｙ方向に所定の濃度閾値よりも暗い（輝度の低い）画素の検出を行い、検出した画素を上唇の上側の輪郭画素とする。上唇輪郭検出部６２２９は、このような処理を、唇検出領域Ｒ３において、ｙ方向の１ラインを１つの処理単位として第５の始点をｘ方向にずらしながら、順次上唇の輪郭画素を検出していく。
【０１０２】
近似関数算出部６２３０は、上唇の輪郭画素の配列を基に、最小二乗近似法により、上唇の上側の輪郭線を表す近似関数を算出する。上唇の左右方向の中間の上側の輪郭は、切り込んだ部分があるため、１つの近似関数により上唇の輪郭を表すことは困難である。そのため、近似関数算出部６２３０は、左右の２つの領域に分けて区分的に近似関数を算出している。ここで、自然な輪郭を表現するためには、近似関数としては低次の関数、例えば二次関数を用いることが好ましい。
【０１０３】
輪郭補正部６２３１は、近似関数算出部６２３０が算出した、上唇の上側の輪郭線を表す近似関数及び下唇の下側の輪郭を表す近似関数が、オペレータによって選択されたリップメークパターンの形状となるように、近似関数の補正を行う。ここで、輪郭補正部６２３１は、唇の厚みを制御する場合は、上唇及び下唇の輪郭を表す近似関数を上下動させることにより制御することができる。さらに、唇の輪郭形状を変更する場合は、唇の輪郭を表す近似関数に対して、輪郭形状を変更するための補正成分を合成することにより、輪郭形状を変更することができる。ここで、輪郭形状を修正するための補正成分は、関数で表すことができるため、輪郭形状の変更は容易に行うことができる。
【０１０４】
さらに、輪郭補正部６２３１は、唇の上側の輪郭線と下側の輪郭線で囲まれる領域（唇領域）から、この唇領域に含まれる歯及び口腔の領域を検出して取り除き、リップメーク領域を設定する。この場合、輪郭補正部６２３１は、唇領域内部の白い画素及び黒い画素を検出することにより歯及び口腔の領域を検出する。
【０１０５】
次に図１８に示すフローチャートにしたがって、リップメーク領域設定部６２２が１枚のフレームに対して行う下唇の輪郭を検出する処理について説明する。
【０１０６】
まず、Ｒ−Ｇ画像変換部６２２１によって、顔画像からＲ−Ｇ成分のみが抽出され、顔画像がＲ−Ｇ画像に変換される（ステップＳ４０１）。次いで、唇検出領域設定部６２２３によって、眼の位置を基準として、顔画像に唇検出領域Ｒ３が設定される（ステップＳ４０２）。次いで、フィルタ処理部６２２４により、唇検出領域Ｒ３の画像に空間周波数フィルタによるフィルタ処理が施され（ステップＳ４０３）、ノイズ成分が除去される。
【０１０７】
次いで、面積算出部６２２５により、唇検出領域Ｒ３の画像が２値化される（ステップＳ４０３）。この場合、唇検出領域Ｒ３には、複数の下唇の候補画像が含まれている。次いで、面積算出部６２２５により、各下唇の候補画像のドット数を基に、下唇の候補画像の面積が算出される（ステップＳ４０５）。次いで、下唇画像特定部６２２６によって、下唇の候補画像のうち最も面積が大きな下唇の候補画像が下唇の画像として特定される（ステップＳ４０６）。
【０１０８】
次いで、下唇輪郭検出部６２２７により、図１９に示すように、第４の始点Ｓ４を始点として、−ｙ方向に赤色の画素が検出され、検出された画素が、下唇の下側の輪郭画素ＰＲ２とされる（ステップＳ４０７）。
【０１０９】
次いで、近似関数算出部６２３０により、下唇の下側の輪郭画素の配列を基に、下唇の下側の輪郭を表す近似関数が算出される（ステップＳ４０８）。
【０１１０】
次に、図２０に示すフローチャートにしたがって、リップメーク領域設定部６２２が、１枚のフレームに対して行う上唇の輪郭を検出する処理について説明する。
【０１１１】
まず、濃淡画像変換部６２２２によって、顔画像が濃淡画像に変換される（ステップＳ５０１）。次いで、唇検出領域設定部６２２３によって、眼の位置を基準として、顔画像に唇検出領域Ｒ３が設定される（ステップＳ５０２）。次いで、フィルタ処理部６２２４により、唇検出領域Ｒ３の画像に空間周波数フィルタを用いたフィルタ処理が施される（ステップＳ５０３）。これにより、ノイズ成分が除去される。次いで、上唇画像特定部６２２８によって、下唇画像特定部６２２６が算出した、下唇の重心を重心とする楕円領域Ｒ４が、唇検出領域Ｒ３内に設定され、上唇の画像が特定される（ステップＳ５０４）。
【０１１２】
次いで、上唇輪郭検出部６２２９により、図１９に示すように、上唇の上側の第５の始点Ｓ５を始点として＋ｙ方向に、所定の濃度閾値よりも暗い画素の検出が行われ、検出された暗い画素が、上唇の上側の輪郭画素ＲＰ１とされる（ステップＳ５０５）。次いで、上唇輪郭検出部６２２９により、輪郭画素ＲＰ１の配列を基に、上唇の上側の輪郭線を表す近似関数が最小二乗法によって算出される（ステップＳ５０６）。次いで、輪郭補正部６２３１により、図１８のステップ４０８で算出された下唇の輪郭線とステップＳ５０６で算出された上唇の輪郭線とで囲まれる唇領域内において、歯及び口腔の画像が検出除去される（ステップＳ５０７）。これにより、リップメーク領域が設定される。
【０１１３】
リップメーク領域設定部６２２は、このような処理を各フレームに対して行うため、顔の表情に追従するリップメーク領域の設定が可能となる。
【０１１４】
次に、メークアップ処理部６３０について説明する。図２１は、メークアップ処理部６３０のブロック構成図を示している。メークアップ処理部６３０は、着色部６３１と、ハイライト処理部６３２と、ラメ処理部６３３とを備えている。
【０１１５】
着色部６３１は、リップメーク領域設定部６２２が設定した、リップメーク領域に対して、オペレータにより選択されたリップカラーパターンを着色する。なお、着色部６３１は、オペレータによりリップカラーパターンが選択されていない場合は、予め定められたリップカラーパターンをリップメーク領域に着色する。ここで、着色部６３１がリップメーク領域にリップカラーパターンを着色するにあたり、特願２００１−２３０５２８の技術を用いてカラーパターンを着色する。
【０１１６】
ハイライト処理部６３２は、下唇の光沢を表すためのハイライト処理を行う。この場合、ハイライト処理部６３２は、ハイライト唇領域の下側の輪郭線を−ｙ方向に平行移動することによって、ハイライト領域の中心線を定め、この中心線に近づくにつれて画素の輝度値を上げることによりハイライト処理を行う。これにより、下唇の光沢がよりリアルに表現されることとなる。
【０１１７】
ラメ処理部６３３は、唇にラメ状の光沢粒子が含まれたリップスティックを塗った場合の視覚効果を表現するラメ処理を行う。
【０１１８】
次に、メークアップ処理部６３０が１毎のフレームに対して行うリップメーク処理について、図２２に示すフローチャートに従って説明する。
【０１１９】
まず、輪郭補正部６２３１により、オペレータによって選択されたリップメークパターンに基づいて、唇領域の上側及び下側の輪郭線が補正される（ステップＳ６０１）。これによりリップメーク領域の形状が、リップメークパターンの形状と一致する。この場合、表示部３００には、図２３で示すような、リップメークパターンを選択するための画面が表示され、オペレータは、この画面を操作することにより、化粧パターンを選択することができる。この画面には、リップメークパターンの一覧を示するリップメーク一覧表示エリアＡＡ、顔画像を動画表示する顔画像表示エリアＢＡ及びリップメーク領域に着色する色の一覧を表示するカラーパレット表示エリアＣＡなどから構成されている。リップメーク一覧表示エリアＡＡには、複数（例えば６つ）のリップメークパターンＲＰが表示されている。オペレータは、マウスポインタをリップメークパターンＲＰのいずれかに合わせてクリックすると、そのリップメークパターンが顔画像表示エリアＢＡに表示された顔画像に施される。
【０１２０】
次いで、メークアップ処理部６３０により、オペレータによって選択されたリップカラーが、リップメーク領域に着色される（ステップＳ６０２）。この場合、オペレータは、図２３に示す画面において、カラーパレット表示エリアＣＡに表示されている複数のカラーパレットＩＰのいずれか１つにマウスポインタを合わせてクリックすると、そのカラーパレットＩＰに表示された色が、顔画像のリップメーク領域に着色される。
【０１２１】
次いで、ハイライト処理部６３２により、ハイライト処理が施される（ステップＳ６０３）。この場合、図１９に示すように、下唇領域には、ハイライト領域の中心線ＨＣＬを中心として、下唇の各画素の輝度値が調節され、下唇のハイライトが表現される。
【０１２２】
次いで、ラメ処理部６３３により、リップメーク領域にラメ処理が施される場合（ステップＳ６０４でＹＥＳ）、ステップＳ６０５に進む。一方、ラメ処理部６３３により、リップメーク領域にラメ処理が施されない場合、（ステップＳ６０４でＮＯ）処理を終了する。
【０１２３】
次いで、ラメ処理部６３３により、リップメーク領域内でラメ処理を行う画素がランダムに抽出される（ステップＳ６０５）。次いで、ラメ処理部６３３により、抽出された画素の輝度が高く設定され、ラメによって唇に生じる光沢が表現される（ステップＳ６０６）。
【０１２４】
メークアップ処理部６３０は、このような処理を各フレームに対して行うため、顔画像の動きに追従するリップメーク処理が可能となる。
【０１２５】
なお、本発明は、以下の態様をとることができる。
【０１２６】
（１）上記実施形態では、画像取得部１００は、被写体の顔の頭部が上側となるように顔画像を取得したが、これに限定されず、例えば顔の頭部が下側あるいは斜め上側に傾くように顔画像を取得してもよい。この場合、眼の中心と口の中心とを結ぶ直線と平行な方向であって、頭部側を上側とし、口側を下側とすればよい。さらに、この場合、上唇画像特定部６２２８が唇検出領域Ｒ３内に楕円領域Ｒ４を設定する場合、楕円の短軸が、眼の中心と口の重心ＲＧとを結ぶ直線上に重なるように、楕円領域Ｒ４を設定すればよい。
【０１２７】
（２）上記実施形態では、近似関数算出部６２１３は、最小二乗法により輪郭線を表す近似関数を算出しているが、これに限定されず、スプライン処理などによって、近似関数を算出してもよい。ただし、なめらかな輪郭線を得るためには、最小二乗法で算出された近似関数が好ましい。
【０１２８】
（３）上記実施形態では説明しなかったが、化粧領域設定部は、眼及び口以外の例えば頬などに化粧領域を設定し、メークアップ処理部６３０は、この化粧領域に色を着色することによりメークアップ処理を行ってもよい。ここで、メークアップ処理部６３０が、化粧領域に色を着色するにあたり、特願２００１−２３０５２８の技術を用いて着色する。
【０１２９】
（４）本化粧シミュレーションプログラムを半導体集積回路により構成し、例えばビデオカメラに内蔵してもよい。
【０１３０】
【発明の効果】
以上説明したように、本発明に係る化粧シミュレーション装置によれば、動画像に対して化粧シミュレーションを行うことができる。さらに、その位置を正確に検出することができる眼の位置に基づいて化粧領域を設定しているため、化粧領域の位置がフレーム間でずれることなく、化粧シミュレーションのリアル性を増すことができる。
【図面の簡単な説明】
【図１】本発明に係る化粧シミュレーション装置の概略構成を示した図である。
【図２】本発明に係る化粧シミュレーション装置のブロック構成図である。
【図３】顔画像の一例を示した図である。
【図４】位置検出部のブロック構成図である。
【図５】瞳抽出フィルタを示したグラフである。
【図６】目輪郭抽出フィルタを示したグラフである。
【図７】信頼度関数の一例を示したグラフである。
【図８】位置検出部が行う眼位置検出モードの処理を示すフローチャートである。
【図９】眼の画像が抽出される様子を説明するための図である。
【図１０】眼追跡モードにおける位置検出部が行う処理を示すフローチャートである。
【図１１】アイメーク領域設定部のブロック構成図である。
【図１２】アイメーク領域設定部が１枚のフレームに対して行う処理を示したフローチャートである。
【図１３】眼及び眉の輪郭画素が検出される様子を説明するための図である。
【図１４】アイメーク領域が設定される様子を説明するための図である。
【図１５】アイメークパターン及びアイラインパターンの一覧画像を示した図である。
【図１６】リップメーク設定部のブロック構成図を示している。
【図１７】唇検出領域が設定される様子を説明するための図である。
【図１８】リップメーク領域設定部が１枚のフレームに対して行う下唇の輪郭を検出する処理を示すフローチャートである。
【図１９】下唇の輪郭画素が検出される様子を説明するための図である。
【図２０】リップメーク領域設定部が、１枚のフレームに対して行う上唇の輪郭を検出する処理を示すフローチャートである。
【図２１】メークアップ処理部のブロック構成図である。
【図２２】メークアップ処理部が１毎のフレームに対して行う処理を示すフローチャートである。
【図２３】リップメークパターン及びリップカラーパターンの一覧画像を示した図である。
【符号の説明】
１００　画像取得部
２００　画像メモリ
３００　表示部
４００　化粧パターン記憶部
５００　カラーパターン記憶部
６００　プログラム実行部
６１０　位置検出部
６１１　検出領域設定部
６１２　画像抽出部
６１２１　濃度補正部
６１２２　サブサンプリング部
６１２３　フィルタ処理部
６１２４　面積算出部
６１２５　画像判定部
６１３　位置特定部
６１４　予測位置算出部
６１５　判断部
６１６　濃淡画像変換部
６２０　化粧領域設定部
６２１　アイメーク領域設定部
６２１１　ノイズ除去部
６２１２　エッジ検出部
６２１２　ノイズ除去部
６２１３　近似関数算出部
６２１４　制御点作成部
６２１５　　境界線算出部
６２２　リップメーク領域設定部
６２２１　Ｒ−Ｇ画像変換部
６２２２　濃淡画像変換部
６２２３　唇検出領域設定部
６２２４　フィルタ処理部
６２２５　面積算出部
６２２６　下唇画像特定部
６２２７　下唇輪郭検出部
６２２８　上唇画像特定部
６２２９　上唇輪郭検出部
６２３０　近似関数算出部
６２３１　輪郭補正部
６３０　メークアップ処理部
６３１　着色部
６３２　ハイライト処理部
６３３　ラメ処理部
７００　一覧画像作成部[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a makeup simulation technique for virtually applying makeup to a face image of a subject.
[0002]
[Prior art]
2. Description of the Related Art A makeup simulation apparatus that virtually takes a makeup by capturing a face image of a person, capturing the image in a computer, and performing predetermined image processing has been known. Such a makeup simulation device allows the user to check the effects of makeup on a computer screen without actually applying makeup, so it is widely used for demonstration sales of cosmetics and training for acquiring makeup techniques. Have been.
[0003]
Techniques related to such makeup simulation include a face make-up support device (JP-A-6-319613), a lip color advice system and method (JP-A-8-202754), and a makeup simulation method (JP-A-10-21290). Etc. have been reported.
[0004]
[Problems to be solved by the invention]
However, each of these techniques targets still images obtained from a digital still camera, an image scanner, or the like, and does not target moving images. For this reason, there has been a problem that it is not possible to confirm the makeup effect with respect to the change of the facial expression, and it lacks realism.
[0005]
The present invention has been made to solve the above-described problem, and has as its object to provide a makeup simulation technique capable of performing makeup simulation more realistically.
[0006]
[Means for Solving the Problems]
The makeup simulation apparatus according to the present invention includes: an image acquisition unit that acquires a face image of a subject every predetermined frame cycle; a position detection unit that detects a position of an eye of the face image for each frame; and an eye that is detected for each frame. A makeup area setting means for setting a predetermined area of the face image as a makeup area based on the position of the face image, and makeup processing means for coloring the makeup area with a predetermined color and performing makeup processing on the face image. It is characterized by the following.
[0007]
According to the present makeup simulation device, the position of the eyes of the captured face image is detected for each frame, and based on the detected positions of the eyes, a makeup area is set in the frame, and the makeup area is colored. Since makeup processing is performed on the face image, makeup simulation for moving images can be performed. Furthermore, since the makeup area is set based on the position of the eye that can be easily detected, the accuracy of setting the makeup area increases, and a more realistic makeup simulation can be realized.
[0008]
Further, the makeup area setting means includes a lip detection area setting means for setting an area including the upper lip and the lower lip as a lip detection area based on the position of the eye, and a lower lip based on color information from the lip detection area. And an upper lip contour detecting means for detecting the contour of the upper lip based on the brightness from the lip detecting area. It is preferable to set the area.
[0009]
In this case, the contour of the lower lip is detected by the color information, and the contour of the upper lip, which is difficult to detect by the color information, is detected by the brightness. Can be.
[0010]
The makeup area setting means sets an upper lip detection area around the center of gravity of the lower lip image in the lip detection area, and specifies an upper lip image from the image in the set upper lip detection area. It is preferable to provide a specifying means.
[0011]
In this case, an upper lip detection area having a shape matching the shape of the lip contour is set around the center of gravity of the lower lip calculated from the contour of the lower lip detected by the lower lip contour detection means. An upper lip image is specified based on the image in the detection area. Since many deep wrinkles and the like exist near the upper lip, if the contour of the upper lip is detected based on the brightness, the wrinkles may be erroneously detected as the contour of the upper lip. Therefore, the upper lip detection area, which is the center of gravity of the lower lip, is applied to the lip detection area, the image outside the upper lip detection area is excluded, and the upper lip image is specified more accurately from the remaining images. Can be identified. Here, the upper lip detection area includes an area having a shape such as an ellipse or a polygon.
[0012]
The makeup area setting means calculates an approximation function representing the contour of the upper lip from the array of pixels constituting the contour of the upper lip detected by the upper lip contour detecting means, and calculates the approximate function representing the contour of the upper lip. It is preferable to include an approximation function calculating means for calculating an approximation function representing the contour of the lower lip from the array of pixels constituting the contour of the lip.
[0013]
In this case, since the approximate function representing the outline of the lower lip and the upper lip is calculated based on the detected arrangement of the pixels constituting the outline of the upper lip and the lower lip, a smooth makeup area can be set.
[0014]
Further, it is preferable that the approximate function calculating means divides the outline above the upper lip into two regions at the center in the left-right direction, and calculates an approximate function representing the outline in each region.
[0015]
In this case, the contour line above the upper lip is divided into two regions at the center in the left-right direction, and an approximate function representing the contour line is calculated in each region, and the contour line above the upper lip is divided into two approximate functions. Therefore, the cut portion of the upper lip at the center of the upper lip can be represented.
[0016]
Further, it is preferable that the make-up processing unit performs a highlight process on the lower lip to indicate the gloss of the lower lip, using the approximate function representing the contour of the lip calculated by the approximate function calculating unit.
[0017]
In this case, the highlight (gloss) of the lip usually has a shape along the contour of the lip, and often appears at the center of the lower lip or the upper lip in the vertical direction. In the present invention, since highlight processing is performed using an approximation function that smoothly expresses the outline of the lips, the highlight processing can be performed quickly and the highlights of the lips can be expressed more beautifully. Thus, the realism of the makeup simulation is increased.
[0018]
A storage means for storing a plurality of predetermined makeup patterns; a list image creating means for creating a list image of the plurality of makeup patterns; and an input means for receiving an operator's operation command; The setting means sets the makeup area based on a makeup pattern selected by an operator using the input means from the list image of the plurality of makeup patterns, and the makeup processing means includes: It is preferable that the makeup area is colored based on a makeup pattern selected by the operator using the input means from the list image of the makeup pattern.
[0019]
In this case, the operator can select a makeup pattern and a color pattern from a list image of a plurality of makeup patterns and color patterns stored in advance, so that it is easy to select a makeup pattern and a color pattern suitable for the face image.
[0020]
In the makeup simulation method according to the present invention, the computer acquires a face image of the subject every predetermined frame period, and the computer converts a predetermined area of the face image into a makeup area based on the position of the eye detected for each frame. , And the computer colors the makeup area with a predetermined color and performs a makeup process on the face image.
[0021]
According to the present makeup simulation method, the position of the eyes of the captured face image is detected for each frame, and based on the detected positions of the eyes, a makeup area is set in the frame, and the makeup area is colored. Since makeup processing is performed on the face image, makeup simulation for moving images can be performed. Furthermore, since the makeup area is set based on the position of the eye that can be easily detected, the accuracy of setting the makeup area increases, and a more realistic makeup simulation can be realized.
[0022]
The makeup simulation program according to the present invention includes: an image acquisition unit configured to acquire a face image of a subject every predetermined frame period; a position detection unit configured to detect an eye position of the face image in each frame; and an eye position detected in each frame. A computer that functions as makeup area setting means for setting a predetermined area of the face image as a makeup area based on the makeup area, and for applying makeup to the face image by coloring the makeup area with a predetermined color. It is characterized by the following.
[0023]
According to the makeup simulation program of the present invention, the position of the eyes of the captured face image is detected for each frame, and based on the detected position of the eyes, a makeup area is set in the frame, and the makeup area is set. Is colored and a make-up process is performed on the face image, so that a makeup simulation for a moving image can be performed. Furthermore, since the makeup area is set based on the position of the eye that can be easily detected, the accuracy of setting the makeup area increases, and a more realistic makeup simulation can be realized.
[0024]
A recording medium for recording a makeup simulation program according to the present invention includes: an image acquisition unit that acquires a face image of a subject every predetermined frame period; a position detection unit that detects an eye position of the face image in each frame; A makeup area setting unit that sets a predetermined area of the face image as a makeup area based on the position of the eye that has been set, and a makeup processing unit that colors the makeup area with a predetermined color and performs makeup processing on the face image. A makeup simulation program characterized by causing a computer to function.
[0025]
According to the recording medium on which the present makeup simulation program is recorded, the position of the eyes of the captured face image is detected for each frame, and a makeup area is set in the frame based on the detected position of the eyes. Since the makeup area is colored and the face image is subjected to make-up processing, makeup simulation for a moving image can be performed. Furthermore, since the makeup area is set based on the position of the eye that can be easily detected, the accuracy of setting the makeup area increases, and a more realistic makeup simulation can be realized.
[0026]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, an embodiment of a makeup simulation apparatus according to the present invention will be described. FIG. 1 is a diagram showing a schematic configuration of a makeup simulation apparatus.
[0027]
The makeup simulation apparatus shown in FIG. 1 includes a personal computer 1 that performs various processes, and a video camera 2 that captures a face image of a subject.
[0028]
The personal computer 1 executes various programs and controls a CPU (central processing unit) 11 for controlling the entire makeup simulation apparatus, and a RAM for temporarily storing a face image of the subject captured by the video camera 2 and various data. (Random access memory) 12, a ROM (read only memory) 13 for storing a basic program for operating the personal computer 1, a CRT (cathode ray tube), a liquid crystal panel, a projector, a plasma display, or the like. A display device 14 for displaying a face image and the like, a recording medium driving device 15 for reading data recorded on a recording medium M such as a CD-ROM, a DVD and a flexible disk, and a hard disk and the like. Supplement A storage device 16 includes a mouse and a keyboard, and an input device 17 for receiving an operation instruction of an operator.
[0029]
The video camera 2 is connected to the personal computer 1 by, for example, a video cable, captures a face image of the subject at a predetermined frame rate, and outputs the captured face image to the personal computer 1 via the video cable.
[0030]
In the present embodiment, the personal computer 1 functions as a makeup simulation device by loading a CD-ROM for recording a makeup simulation program into the recording medium drive 15 and installing the CD-ROM in the personal computer 1. The makeup simulation program may be installed in the personal computer 1 by storing the makeup simulation program in a web server provided on the Internet and downloading the makeup simulation program from the web server. Further, the personal computer 1 acquires a face image, transmits the acquired face image to a web server, performs an eye detection process on the web server, and transmits the processing result to the personal computer. The eye detection program may be distributed and executed between the PC and the personal computer 1.
[0031]
FIG. 2 shows a block diagram of the makeup simulation apparatus. The makeup simulation apparatus functionally includes an image acquisition unit 100, an image memory 200, a display unit 300, a makeup pattern storage unit 400, a color pattern storage unit 500, a program execution unit 600, and a list image creation unit. 700.
[0032]
The image acquisition unit 100 acquires a face image of a subject photographed using the video camera 2 and stores the face image in the image memory 200. When acquiring the face image, the image acquisition unit 100 sets the face so that the direction of the straight line connecting the center of the eye and the center of the mouth is perpendicular to the horizontal direction, and the head is on the upper side. Get an image. In the following description, the face image refers to a rectangular image including the face of the subject as shown in FIG. 3, the upper left vertex of the face image is defined as the origin (0, 0), and Set the x coordinate to the left side of the rectangle and the y coordinate.
[0033]
The image memory 200 is configured from the RAM 12 and the like, and temporarily stores and stores the face image acquired by the image acquisition unit 100 and the image obtained by performing various processes on the face image by the program execution unit 600. The displayed face image is output to the display unit 300.
[0034]
The display unit 300 includes the display device 14, a VRAM, and the like, and displays an image output from the image memory 200.
[0035]
The makeup pattern storage unit 400 mainly includes the RAM 12, and stores a plurality of eye makeup patterns, a plurality of lip makeup patterns, and the like. The color pattern storage unit 500 mainly includes the RAM 12, and stores an eye color pattern, a lip color pattern, and the like to be applied to the makeup area.
[0036]
The list image creation unit 700 combines data such as an eye makeup pattern stored in the makeup pattern storage unit 400, data such as an eye color pattern stored in the color pattern storage unit 500, and a face image, A list image as shown in FIG. 15 or FIG. 23 described later is created.
[0037]
The program execution unit 600 is realized by the CPU 11 executing the makeup simulation program, and detects a position of the eye in the face image, and sets a makeup area in the face image based on the detected eye position. A makeup area setting unit 620 for performing the setting and a makeup processing unit 630 for coloring the set makeup area. Further, makeup area setting section 620 includes an eye makeup area setting section 621 that sets an area for applying eye makeup such as an eye shadow or an eye line to the face image, and a lip makeup that sets an area for applying lip makeup to the face image. An area setting unit 622 is provided.
[0038]
(Eye position detection processing)
Next, the eye position detection processing will be described. FIG. 4 is a block diagram of the position detection unit 610.
[0039]
The position detection unit 610 detects, for example, an eye detection mode in which eyes are detected from the first frame to a predetermined number of frames of the face image, and an eye position of a past frame detected in the eye detection mode. Basically, there are two modes: an eye tracking mode for detecting the position of the eye in the current frame.
[0040]
The position detection unit 610 includes a detection area setting unit 611, an image extraction unit 612, a position identification unit 613, a predicted position calculation unit 614, a determination unit 615, and a grayscale image conversion unit 616.
[0041]
In the eye detection mode, the detection area setting unit 611 determines that the image extraction unit 612 converts the face image acquired by the image acquisition unit 100, for example, from the first frame to the frame immediately before the predetermined number of frames. On the other hand, a first detection region R1 including both eyes for performing filter processing is set. As shown in FIG. 3, the first detection region R1 is a long rectangular region parallel to the x direction.
[0042]
In the eye tracking mode, the detection area setting unit 611 applies two areas centered on the respective positions of both eyes detected in the frame immediately before the current frame to the current frame, and Then, a second detection region R2 for the image extraction unit 612 to perform various processes is set. As shown in FIG. 3, one of the two regions included in the second detection region R2 is, for example, a substantially square-shaped region including the left eyebrow and the eye, and the other region is the right eyebrow. For example, a substantially square area including the eyes and eyes. The two regions included in the second detection region R2 may be elliptical regions each centered on the center position of the eye and whose short axis is parallel to the y direction.
[0043]
The image extraction unit 612 includes a density correction unit 6121, a sub-sampling unit 6122, a filter processing unit 6123, an area calculation unit 6124, and an image determination unit 6125.
[0044]
The density correction unit 6121 obtains the number of pixels having the same density value for the first and second detection areas R1 and R2 set by the detection area setting unit 611, creates a density value histogram, and generates the density value histogram. Is performed. As a result, variations in lightness and darkness of each face image due to different lighting conditions and the like when the video camera 2 captures an image of a subject are suppressed.
[0045]
The sub-sampling unit 6122 performs a process of thinning out the number of pixels of the first and second detection regions R1 and R2 as a pre-process for speeding up the filtering process by the filtering unit 6123. In this case, each of the two regions included in the second detection region R2 is subjected to the thinning process so that the number of pixels is, for example, 64 × 64 dots.
[0046]
The filter processing unit 6123 takes out the first detection region R1 in the eye detection mode, takes out the second detection region R2 in the eye tracking mode, and takes out the first detection region R1 and the second detection region taken out. An image of a pupil and an eye contour is extracted using a pupil extraction filter for extracting a pupil for R2 and an eye contour extraction filter for extracting an eye contour. FIG. 5 is a graph showing a pupil extraction filter, and FIG. 6 is a graph showing an eye contour extraction filter. 5 and 6, the vertical axis is the coordinate corresponding to the y coordinate of the face image, and the horizontal axis is the coordinate indicating the filter value.
[0047]
The pupil extraction filter hp (y) is represented by Expression (1).
[0048]
(Equation 1)

[0049]
Here, in Expression (1), y is a variable indicating an arbitrary position in the y direction of the face image, Tp is a constant, and its value is substantially the same as the average pupil radius of ordinary people. . The eyes, especially the pupil, in the face image are relatively darker than the surrounding flesh-colored area and white-eye area. Therefore, as the pupil extraction filter hp (y), the filter value approaches 0 as the value on the horizontal axis shown in FIG. 5 approaches 0, and becomes 1 as the values approach the constants Tp and −Tp with respect to the horizontal axis from the origin. , A one-dimensional function suitable for pupil extraction is adopted.
[0050]
The eye contour extraction filter hc (y) is represented by Expression (2).
[0051]
(Equation 2)

[0052]
Equation (2) is almost the same equation as equation (1), but the magnitude of constant Tc is different from the magnitude of constant Tp in equation (1). In other words, the eye contour extraction filter hc (y) extracts the dark and thin line-shaped region so that the contour of the curved eye that is relatively dark with respect to the surrounding skin color image is appropriately extracted. The value of the constant Tc is set so as to be able to perform the operation. Therefore, the constant Tc is set to a value smaller than the constant Tp (Tc <Tp).
[0053]
The filter processing unit 6123 uses the pupil extraction filter hp (y) for the image f (y) of one line in the y direction of the images in the first and second detection regions R1 and R2, using the pupil extraction filter hp (y). After performing the convolution integral shown in (1), the convolution integral using the pupil extraction filter hp (y) is further performed on the image f (y) for one line in the next y direction adjacent in the x direction. The convolution integral using the pupil extraction filter hp (y) is sequentially performed on the image f (y) in the y direction in the first and second detection regions R1 and R2 to extract a pupil image.
[0054]
[Equation 3]

[0055]
Further, as shown in Expression (4), the filter processing unit 6123 sequentially performs convolution integration using the eye contour extraction filter hc (y) on the image f (y) for one line in the y direction. To extract an eye contour image. Then, the extracted pupil image and the contour image of the eye are combined to extract an eye image.
[0056]
Here, the filter processing unit 6123 performs one-dimensional filter processing on the images of the first and second detection regions R1 and R2, but the image acquisition unit 100 determines the center of the eye and the center of the mouth. Is obtained so that the straight line connecting the image and the line in the y direction, many parts of the outline of the eye appear almost parallel to the x direction, and the image f (y ) Is subjected to convolution integration using the eye contour extraction filter hc (y), whereby an eye contour image can be extracted. As a result, an eye contour image can be extracted by one-dimensional filter processing without performing two-dimensional filter processing requiring a large amount of calculation.
[0057]
The area calculation unit 6124 performs binarization processing on the images of the first and second detection regions R1 and R2 that have been subjected to the filter processing by the filter processing unit 6123. As described above, the filter processing unit 6123 extracts an eye image from the first and second detection regions R1 and R2 using the eye contour extraction filter hc (y) and the pupil extraction filter hp (y). However, since these filters have the property of extracting pixels that are relatively dark with respect to skin color, they may also detect eyebrows, nose flanks, and dark circles under eyes. Therefore, the filter processing unit 6123 extracts a plurality of eye candidate images. Therefore, the area calculation unit 6124 performs a binarization process so that the image determination unit 6125 can determine the image of the eye based on the area, and then labels each of the candidate images. The area of each candidate image is calculated by counting the number of dots of the image.
[0058]
The image determination unit 6125 determines, as the eye images, two candidate images having a large area among the candidate images calculated by the area calculation unit 6124, and specifies the eye image from the plurality of candidate images. However, only by comparing the areas, the candidate image showing the eyebrows also has a large area, so that this image may be erroneously specified as an eye image. Therefore, when detecting two candidate images having large areas in the vertical direction in the y direction, the image determining unit 6125 excludes the upper candidate image as an eyebrow image and specifies the lower image as an eye image.
[0059]
The position identification unit 613 calculates the respective centers of gravity of the images of both eyes identified by the image determination unit 6125, and identifies the respective positions of both eyes. When calculating the center of the eye, the position specifying unit 613 applies, for example, a circumscribed rectangle to the image of the eye and calculates the intersection of two diagonal lines of the rectangle as the center of gravity of the eye.
[0060]
The predicted position calculation unit 614 calculates the moving speed and the moving acceleration of the eyes based on the respective positions of the both eyes detected by the position specifying unit 613 in the past frame, and, based on the calculated moving speed and the moving acceleration, according to Taylor expansion. The predicted position of the eye in the current frame is calculated. Assuming that the position of the eye in the frame immediately before the current frame is x (t) and y (t), the moving speed of the eye is the first order derivative x '(t) and y' of x (t) and y (t). '(T), and the eye's movement acceleration is represented by x (t), the second derivative of y (t) x ″ (t), y ″ (t) (where t is time Variable). Therefore, assuming that the time interval of each frame is Δt, the predicted positions x (t + Δt) and y (t + Δt) can be expressed by Taylor expansion up to the second order term as shown in Expression (5). .
[0061]
(Equation 4)

[0062]
The determination unit 615 obtains a difference d between the eye position of the current frame specified by the position specifying unit 613 and the predicted position of the eye of the current frame calculated by the predicted position calculation unit 614, and based on the difference d, The reliability is calculated using the determined reliability function, and the accuracy of the position of the eye specified by the position specifying unit 613 is determined. The reliability function is, for example, exp (-d ² ), And the reliability increases as the difference d approaches 0, and the reliability decreases as the difference d increases. Further, the determining unit 615 adopts the position of the eye detected by the position specifying unit 613 as a correct eye position if the reliability is equal to or more than a predetermined value, for example. If the specifying unit 613 erroneously detects the position of the eye, the position is not adopted as the position of the eye. Note that the reliability function is not limited to an exponential function, and may be, for example, a quadratic function or a linear function.
[0063]
The grayscale image conversion unit 616 applies different different weights to the R (red), G (green), and B (blue) components of the face image acquired by the image acquisition unit 100 and combines the components. Then, the face image is converted into a grayscale image representing only the luminance component, and stored in the image memory 200. The detection area setting unit 611 to the determination unit 615 perform various processes on the grayscale image.
[0064]
Next, processing in the eye position detection mode performed by the position detection unit 610 will be described with reference to the flowchart shown in FIG.
[0065]
First, the first frame of the face image is obtained by the image obtaining unit 100 (step S101). Next, the face image obtained by the image obtaining unit 100 is converted into a gray image by the gray image conversion unit 616 (step S102). Next, the first detection region R1 is set on the face image by the detection region setting unit 611 (see FIG. 3) (step S103). Next, the density value histogram of the first detection region R1 is flattened by the density correction unit 6121 (step S104).
[0066]
Next, the sub-sampling unit 6122 performs a thinning process on the image of the first detection region R1 in order to reduce the calculation amount of the filter processing unit 6123 (Step S105).
[0067]
Next, the first detection region R1 is extracted from the face image by the filter processing unit 6123, and the extracted first detection region R1 is subjected to filter processing using the pupil extraction filter hp (t) ( Step S106) An image of the pupil is extracted. In this case, an image of the pupil as shown in FIG. 9A is extracted. Next, the first detection region R1 is again extracted from the face image by the filter processing unit 6123, and the extracted image of the first detection region R1 is subjected to filter processing using the eye contour extraction filter hc (t). Is performed, and an image of the outline of the eye is extracted (step S107). In this case, an image of the eye contour as shown in FIG. 9B is extracted. Next, the filter processing unit 6123 combines the image of the pupil extracted in step S106 and the image of the contour of the eye extracted in step S107, and extracts the image of the eye (step S108). In this case, an eye image in which the contours of the pupil and the eye as shown in FIG. 9C are combined is extracted. Actually, at the stage of step S108, a plurality of candidate images of eyes other than eyes are included, but in FIGS. 9A to 9C, candidate images of eyes other than eyes are omitted. are doing.
[0068]
Next, the area calculation unit 6124 subjects the first detection region R1 filtered by the filter processing unit 6123 to binarization processing, labels a plurality of eye candidate images (step S109), and sets the number of dots. The area of each candidate image is calculated based on. Next, when two large candidate images can be extracted from the plurality of eye candidate images by the image determination unit 6125 (YES in step S110), this candidate image is determined as a binocular image, and step S111 is performed. Proceed to. On the other hand, if it is not possible to extract two large areas in step S110 (NO in step S110), the process returns to step S103 to perform reprocessing. Note that when the image determination unit 6125 extracts two large areas in the upper and lower directions in the y direction (that is, in this case, there are at least three eye candidate images having an area suitable for an eye candidate image). The upper image is excluded as an eyebrow image, and the lower image is specified as an eye image.
[0069]
Next, the center of gravity of the image of the eye is calculated by the position specifying unit 613, and the center thereof is detected as the position of the eye (step S111). Next, the next frame of the face image is obtained (step S112). Next, when the obtained frame is less than the predetermined number of frames (NO in step S113), the process returns to step S102. On the other hand, if the acquired frame is the predetermined number of frames (YES in step S113), the process proceeds to step S201 shown in FIG. Here, the predetermined number of sheets corresponds to a number sufficient for calculating the moving speed and the moving acceleration of the position of the eye in the eye tracking mode. Therefore, in order to calculate the moving speed and acceleration of the eye position, at least three frames are required, and the predetermined number of frames is at least the fourth and subsequent frames.
[0070]
Next, processing performed by the position detection unit 610 in the eye tracking mode will be described with reference to the flowchart shown in FIG. The eye tracking mode is performed subsequent to the processing in step S113 of the eye detection mode.
[0071]
First, the frame acquired in step S112 shown in FIG. 8 is converted into a grayscale image (step S201). Next, the detection area setting unit 611 sets the second detection area R2 on the current frame as shown in FIG. 3 based on the positions of the eyes detected in the past frame (step S202). Next, in steps S203 to S208, the same processing as the processing performed in steps S104 to S109 in the eye detection mode shown in FIG. 8 is performed on the second detection region R2.
[0072]
Next, the predicted position calculation unit 614 uses the eye position in the frame immediately before the current frame, the eye moving speed and the eye acceleration calculated from the eye position in the past frame, and performs Taylor expansion to calculate the current frame. The predicted position of the eye is calculated (step S209). Next, the determining unit 615 calculates a difference d between the position of the eye detected on the current frame by the position specifying unit 613 and the predicted position of the eye calculated in step S209, and based on the calculated difference d, The degree is calculated, the reliability is compared with a predetermined threshold, and the accuracy of the eye position detected by the position specifying unit 613 is determined (step S210). Here, when the calculated reliability is larger than the predetermined threshold, the position of the eye specified by the position specifying unit 613 is determined to be correct (YES in step S210), and the position is determined as the position of the eye. (Step S211). On the other hand, if the calculated reliability is smaller than the predetermined threshold, the position specifying unit 613 determines that the position of the eye has been erroneously specified, returns to step S103 illustrated in FIG. 8, and performs the eye detection process again ( (NO in step S210). In this case, as shown in FIG. 3, the display unit 300 displays a marker IM indicating the position of the eye detected by the position specifying unit 613 at the position of the center of the pupil so as to follow the movement of the face of the subject. You may.
[0073]
Next, the next frame is acquired by the image acquiring unit 100 (Step S212). Next, it is determined whether or not the processing of the last frame has been completed (step S213), and if the processing of the last frame has been completed (YES in step S213), the processing ends. If the processing of the last frame has not been completed (NO in step S213), the process returns to step S201, and the eye tracking mode processing is subsequently performed on the frame acquired in step S212.
[0074]
(Makeup processing)
Next, the makeup process will be described. FIG. 11 is a block diagram of the eye makeup area setting unit 621.
[0075]
The eye makeup area setting section 621 includes a noise removing section 6211, an edge detecting section 6212, an approximate function calculating section 6213, a control point creating section 6214, and a boundary line calculating section 6215.
[0076]
The noise removal unit 6212 removes unnecessary image components from the second detection region R2 set by the detection region setting unit 611, using a spatial frequency filter that passes image components in a predetermined frequency band. In the second detection region R2, there is a region darker than the surrounding skin color near the eyebrows or temples, and therefore, the edge detection unit 6212 may erroneously detect these dark regions as the outlines of the eyes and eyebrows. is there. Therefore, the noise removing unit 6212 removes image components that are unnecessary when detecting the contours of the eyes and eyebrows using the spatial frequency filter. As a filtering method using a spatial frequency filter, for example, a filtering method using a discrete Fourier transform, a discrete cosine transform, a wavelet transform, or the like may be adopted.
[0077]
The edge detection unit 6212 sets a position between the eyebrows and the eye in the y direction as a first start point based on the respective positions of both eyes detected by the position identification unit 613, and sets a second detection region R2 Of the image in the upper part (−y direction), the detection of a pixel darker than the surrounding skin color is performed by comparing with a predetermined density threshold, and the detected pixel is defined as the contour line below the eyebrows. A pixel (outline pixel) is detected, and a pixel darker than the surrounding skin color in a downward direction (+ y direction) is detected by comparing it with a predetermined density threshold. And The edge detection unit 6212 performs such processing on one line in the y direction from the substantially middle position of one side in the x-axis direction of the second detection region R2 toward the left side (+ x side) and the right side (−x side). Images are sequentially applied as one processing unit. As described above, since the edge detection unit 6212 detects the outline pixels of the eyes and eyebrows separately from the middle position in the x direction of the second detection region, the edge detection unit 6212 detects the position where the pixel darker than the density threshold is no longer detected. Are the positions of the inner and outer corners of the eye, the outer and inner corners of the eyebrow, and the outer corner of the eyebrow.
[0078]
Further, the edge detecting unit 6212 detects the outline pixel on the lower side of the eyebrow and the outline pixel on the upper side of the eyebrow based on the positions of both eyes detected by the position identification unit 613, in the same manner as described above. With the predetermined position on the upper side (x-axis side) as the second starting point, detection of pixels darker than the surrounding skin color in the downward direction (+ y direction) is performed, and the detected pixel is set as the upper contour pixel of the eyebrows. .
[0079]
Furthermore, the edge detecting unit 6212 detects a contour pixel above the eyebrows, based on the positions of both eyes detected by the position specifying unit 613, using a predetermined position below the eye as a third starting point. Similarly to the above, pixels darker than the surrounding skin color are detected in the upward direction (-y direction), and the detected pixels are set as contour pixels on the lower side of the eye.
[0080]
The approximation function calculation unit 6213 calculates an approximation function representing the upper contour of the eyebrow using, for example, the least squares method, based on the arrangement of the upper end pixels of the eyebrow detected by the edge detection unit 6212. Further, the approximation function calculating unit 6213 also uses the least squares approximation method based on the pixel arrangement for the contour pixels on the lower side of the eyebrow, the contour pixels on the upper side of the eye, and the contour pixels on the lower side of the eye. Approximate functions representing the outline on the side, the outline on the upper side of the eye, and the outline on the lower side of the eye are calculated. Here, if the contour is expressed using a higher-order approximation function, the shape of the contour becomes unnatural. Therefore, it is preferable to use a lower-order function, for example, a quadratic function as the approximation function.
[0081]
The control point creation unit 6214 translates the pixels of the eyebrow corner, eyebrows, corners of the eyes and the corners of the eyebrow detected by the edge detection unit 6212 in the y direction, and linearly interpolates these pixels to set an eye makeup area. Create a control point for. Further, when creating the control points, the control point creating unit 6214 creates the control points according to the eye makeup pattern selected by the user from the plurality of eye makeup patterns stored in the makeup pattern storage unit 400.
[0082]
The boundary line calculation unit 6215 calculates an approximate function representing the boundary line of the eye makeup region by using a constraint condition of passing through some control points among the plurality of control points generated by the control point generation unit 6214. . The area surrounded by the approximation function and the contour of the eye is the eye makeup area. Here, the approximation function is a function of one variable using the x coordinate as a variable. Therefore, the determination of the pixels in the eye makeup area can be easily performed by comparing the approximate function representing the boundary line with the approximate function representing the contour line above the eye. Further, since the eye makeup region can be specified by the approximation function, the pixel at the boundary of the eye makeup region can be easily specified, and the color of the eye makeup can be continuously changed near the boundary line.
[0083]
Next, processing performed by the eye makeup area setting unit 621 on one frame will be described with reference to the flowchart shown in FIG. It is assumed that this frame is converted into a grayscale image by the grayscale image conversion unit 616 shown in FIG. 4 in advance, and the second detection region R2 is set by the detection region setting unit 611. Until the second detection region R2 is specified, the eye makeup region setting unit 621 performs the following processing on the first detection region R1.
[0084]
First, the noise removal unit 6211 performs a filtering process using a spatial frequency filter on the second detection region R2 (step S301). This eliminates noise components that are unnecessary when the edge detection unit 6212 detects contour pixels of the eyes and eyebrows.
[0085]
Next, as shown in FIG. 13, the edge detection unit 6212 detects the upper contour pixel EP1 of the eye and the lower contour pixel MP2 of the eyebrow from the first start point S1 toward the upper and lower directions, respectively. Is performed (step S302). Next, the edge detection unit 6212 detects the contour pixel MP1 above the eyebrows downward from the second start point S2 and the contour pixel EP2 below the eye upward from the third start point S3. It is detected (step S303).
[0086]
Next, the approximation function calculating unit 6213 uses the least squares method based on the arrangement of the contour pixels EP1 on the upper side of the eye and the contour pixels MP2 on the lower side of the eyebrow detected in step S302 to determine the contour line of the upper eye and the eyebrow. Approximate functions representing the lower contour lines are calculated, and based on the arrangement of the upper contour pixels MP1 of the eyebrows and the lower contour pixels EP2 of the eye detected in step S303, the approximation of the eyebrows is performed by the least squares method. Approximate functions representing the upper contour and the lower contour of the eye are calculated (step S304).
[0087]
Next, the pixels of the inner corner of the eye, the outer corner of the eye, the inner corner of the eyebrow, and the outer corner of the eyebrow are detected by the edge detecting unit 6212 (step S305). Next, a control point is created by the control point creation unit 6214 based on the pixels at the inner and outer corners of the eye, the inner and outer eyebrows, and the outer edge of the eyebrow (step S306). In this case, as shown in FIG. 14, the inner corner ET is set as the control point CP, and the eyebrow corner ME is moved to, for example, the position ME 'in the + y direction and set as the control point CP. Further, the point HT at the upper end of the pupil is moved to the position HT ′ in the −y direction, and is set as the control point CP. Then, some of the control points CP are linearly interpolated to create a new control point CP ′.
[0088]
Next, in step S307 illustrated in FIG. 12, the boundary line calculation unit 6215 calculates an approximate function EL1 representing the boundary line of the eye makeup region based on the control points CP and CP ′ (step S307). In this case, as shown in FIG. 14, a region surrounded by the approximation function EL1 and the outline EL2 of the eye is an eye makeup region EMR.
[0089]
Further, in this case, the control points CP and CP 'are created so that the shape of the eye makeup region EMR matches the shape of the eye makeup pattern selected by the operator using the screen diagram shown in FIG. Is also good. The screen diagram shown in FIG. 15 includes an eye makeup pattern list display section PR for displaying a list of a plurality of eye makeup patterns, a face image display section FR for displaying a face as a moving image, and a plurality of eye color patterns to be applied to the eye makeup area EMR. An eye color pattern display section CR to be displayed in a list is displayed. The list image is created by the list image creating unit 700 combining the eye makeup pattern and the eye color pattern stored in the makeup pattern storage unit 400 and the color pattern storage unit 500 with the face image. When the operator positions the mouse pointer on the target eye makeup pattern and eye color pattern on the list image and clicks, the eye makeup pattern and the eye color pattern are applied to the face image displayed on the face image display section FR.
[0090]
Next, the eye color pattern selected by the user is colored in the EMR in the eye make-up area by the make-up processing unit 630 (step S308). Through the above processing, the eye image is subjected to eye makeup processing. Note that the processing from step S301 to step S308 is performed for each frame, so that an eye makeup process is performed on a moving image.
[0091]
Next, the lip makeup area setting unit 622 will be described with reference to FIG. The lip makeup area setting unit 622 individually detects the contour of the upper lip and the contour of the lower lip.
[0092]
The lip makeup area setting section 622 includes an RG image conversion section 6221, a grayscale image conversion section 6222, a lip detection area setting section 6223, a filter processing section 6224, an area calculation section 6225, and a lower lip image specification section 6226. , A lower lip contour detection section 6227, an upper lip image identification section 6228, an upper lip contour detection section 6229, an approximate function calculation section 6230, and a contour correction section 6231.
[0093]
6221 is used to detect the contour of the lower lip, extracts an R (red) component and a G (green) component from the face image, and subtracts a G component from the R component to obtain a difference (RG) component. Calculate and convert the face image into a difference (RG) image. In this case, the RG image conversion unit 6221 calculates a ratio (R / G) component of the R component and the G component extracted from the face image, and converts the face image into a ratio (R / G) component. You may.
[0094]
The grayscale image conversion unit 6222 is used to detect the contour of the upper lip, and combines the R, G, and B components included in the facial image to convert the facial image into a grayscale image representing only luminance.
[0095]
The lip detection area setting unit 6223 sets a predetermined area including the lips as a lip detection area for detecting the upper lip and the lower lip with reference to the position of the eye specified by the position specifying unit 613. As shown in FIG. 17, in setting the lip detection area, the lip detection area setting unit 6223 sets the average position RG of the center of gravity of the mouth with respect to the position EP of a general human eye in the x direction with the center of gravity R3G. A long rectangular area is set as the lip detection area R3.
[0096]
The filter processing unit 6224 applies a spatial frequency filter for extracting only a predetermined frequency band to the lip detection region R3, and removes an unnecessary image when detecting the upper lip and the lower lip.
[0097]
The area calculation unit 6225 binarizes the lip detection region R3, extracts a plurality of candidate images of the lower lip, labels each of the candidate images, and prints a dot included in each of the candidate images of the lower lip. The area of the lower lip candidate image is calculated by counting the number.
[0098]
The lower lip image specifying unit 6226 specifies, as the lower lip image, the image with the largest area among the plurality of lower lip candidate images.
[0099]
The lower lip contour detection unit 6227 detects a red pixel in the −y direction using the lower point of the lower lip image determined by the lower lip image identification unit 6226 as a fourth start point, and determines the detected pixel as a lower pixel. It is assumed that the contour pixel is on the lower side of the lips. The lower lip contour detection unit 6227 sequentially performs such processing in the lip detection region R3 while using the one line in the y direction as one processing unit and shifting the fourth start point in the x direction. Contour pixels are detected.
[0100]
As shown in FIG. 17, the upper lip image specifying unit 6228 sets the elliptical region R4 (the upper lip detecting region) having the center of gravity MG of the lower lip image determined by the lower lip image specifying unit 6226 as the center of gravity R3G in the lip detecting region R3. Is set in the lip detection region R3, and the image outside the elliptical region R4 is removed, and the image remaining in the lip detection region R3 is specified as the upper lip image.
[0101]
The upper lip contour detection unit 6229 detects a pixel darker (lower in brightness) in the + y direction than a predetermined density threshold, with a predetermined position above the upper lip image specified by the upper lip image specification unit 6228 as a fifth start point. Is performed, and the detected pixel is set as the upper contour pixel of the upper lip. The upper lip contour detection unit 6229 detects such contour pixels sequentially in the lip detection region R3 while shifting the fifth start point in the x direction using one line in the y direction as one processing unit. Go.
[0102]
The approximate function calculation unit 6230 calculates an approximate function representing the upper contour line of the upper lip by the least squares approximation method based on the arrangement of the contour pixels of the upper lip. Since the upper contour in the middle in the left-right direction of the upper lip has a cut portion, it is difficult to represent the contour of the upper lip by one approximation function. Therefore, the approximation function calculation unit 6230 calculates the approximation function piecewise in two regions, left and right. Here, in order to express a natural contour, it is preferable to use a low-order function, for example, a quadratic function as the approximation function.
[0103]
The contour correction unit 6231 calculates the approximation function representing the upper contour line of the upper lip and the approximation function representing the lower contour of the lower lip calculated by the approximation function calculation unit 6230, based on the shape of the lip makeup pattern selected by the operator. The approximation function is corrected so that Here, when controlling the thickness of the lips, the contour correction unit 6231 can control the thickness by moving the approximate function representing the contours of the upper lip and the lower lip up and down. Further, when changing the contour shape of the lip, the contour shape can be changed by combining a correction component for changing the contour shape with an approximation function representing the contour of the lip. Here, the correction component for correcting the contour shape can be represented by a function, so that the contour shape can be easily changed.
[0104]
Further, the contour correction unit 6231 detects and removes the tooth and mouth regions included in the lip region from the region (lip region) surrounded by the upper contour line and the lower contour line of the lip, and removes the lip makeup region. Set. In this case, the contour correction unit 6231 detects the teeth and the oral cavity region by detecting white pixels and black pixels inside the lip region.
[0105]
Next, a process performed by the lip makeup area setting unit 622 for detecting the contour of the lower lip for one frame will be described with reference to the flowchart shown in FIG.
[0106]
First, only the RG component is extracted from the face image by the RG image conversion unit 6221, and the face image is converted into the RG image (step S401). Next, the lip detection area setting unit 6223 sets the lip detection area R3 in the face image based on the position of the eye (step S402). Next, the filter processing unit 6224 performs a filtering process using a spatial frequency filter on the image of the lip detection region R3 (step S403), and removes noise components.
[0107]
Next, the image of the lip detection region R3 is binarized by the area calculation unit 6225 (step S403). In this case, the lip detection region R3 includes a plurality of lower lip candidate images. Next, the area of the lower lip candidate image is calculated by the area calculator 6225 based on the number of dots of each lower lip candidate image (step S405). Next, the lower lip image specifying unit 6226 specifies the lower lip candidate image having the largest area among the lower lip candidate images as the lower lip image (step S406).
[0108]
Next, as illustrated in FIG. 19, the lower lip contour detection unit 6227 detects a red pixel in the −y direction starting from the fourth start point S4, and determines the detected pixel as a lower contour of the lower lip. The pixel PR2 is set (step S407).
[0109]
Next, the approximate function calculating unit 6230 calculates an approximate function representing the lower contour of the lower lip based on the arrangement of the lower contour pixels of the lower lip (step S408).
[0110]
Next, a process performed by the lip makeup area setting unit 622 for detecting the contour of the upper lip performed on one frame will be described with reference to the flowchart illustrated in FIG.
[0111]
First, a face image is converted into a gray image by the gray image conversion unit 6222 (step S501). Next, the lip detection region setting unit 6223 sets the lip detection region R3 in the face image based on the position of the eye (step S502). Next, the filter processing unit 6224 performs a filtering process using a spatial frequency filter on the image of the lip detection region R3 (step S503). Thereby, the noise component is removed. Next, the elliptical region R4 having the center of gravity of the lower lip calculated by the lower lip image specifying unit 6226 is set in the lip detection region R3 by the upper lip image specifying unit 6228, and the image of the upper lip is specified (step S2). S504).
[0112]
Next, as shown in FIG. 19, the upper lip contour detection unit 6229 detects a pixel darker than a predetermined density threshold in the + y direction starting from the fifth start point S5 on the upper side of the upper lip, and detects the detected darkness. The pixel is set as the upper contour pixel RP1 of the upper lip (step S505). Next, the upper lip contour detection unit 6229 calculates an approximate function representing the upper contour line of the upper lip by the least squares method based on the array of the contour pixels RP1 (step S506). Next, the contour correction unit 6231 detects and removes the images of the teeth and the mouth in the lip region surrounded by the contour of the lower lip calculated in step 408 of FIG. 18 and the contour of the upper lip calculated in step S506. Is performed (step S507). As a result, a lip make area is set.
[0113]
Since the lip makeup area setting unit 622 performs such processing for each frame, it is possible to set a lip makeup area that follows the facial expression.
[0114]
Next, the makeup processing unit 630 will be described. FIG. 21 is a block diagram of the makeup processing unit 630. The make-up processing section 630 includes a coloring section 631, a highlight processing section 632, and a glitter processing section 633.
[0115]
The coloring unit 631 colors the lip color pattern selected by the operator on the lip makeup area set by the lip makeup area setting unit 622. It should be noted that the coloring unit 631 colors a predetermined lip color pattern in the lip make area when the lip color pattern has not been selected by the operator. Here, when the coloring unit 631 colors the lip color pattern in the lip makeup area, the color pattern is colored using the technique of Japanese Patent Application No. 2001-230528.
[0116]
The highlight processing unit 632 performs highlight processing for representing the gloss of the lower lip. In this case, the highlight processing unit 632 determines the center line of the highlight region by translating the lower contour line of the highlight lip region in the −y direction, and determines the luminance value of the pixel as approaching the center line. To perform highlight processing. As a result, the gloss of the lower lip is more realistically expressed.
[0117]
The glitter processing unit 633 performs a glitter process that expresses a visual effect when a lipstick containing glitter particles is applied to the lips.
[0118]
Next, the lip makeup process performed by the makeup process unit 630 on each frame will be described with reference to the flowchart shown in FIG.
[0119]
First, the contour lines on the upper and lower sides of the lip region are corrected by the contour correction unit 6231 based on the lip make pattern selected by the operator (step S601). Thereby, the shape of the lip make region matches the shape of the lip make pattern. In this case, a screen for selecting a lip makeup pattern as shown in FIG. 23 is displayed on display unit 300, and the operator can select a makeup pattern by operating this screen. This screen includes a lip makeup list display area AA showing a list of lip makeup patterns, a face image display area BA for displaying a face image as a moving image, and a color palette display area CA for displaying a list of colors to be colored in the lip makeup area. It is composed of A plurality (for example, six) of lip makeup patterns RP are displayed in the lip makeup list display area AA. When the operator positions the mouse pointer on one of the lip makeup patterns RP and clicks, the lip makeup pattern is applied to the face image displayed in the face image display area BA.
[0120]
Next, the lip color selected by the operator is colored in the lip makeup area by the makeup processing unit 630 (step S602). In this case, when the operator positions the mouse pointer on any one of the plurality of color pallets IP displayed in the color pallet display area CA and clicks on the screen shown in FIG. 23, the image displayed on the color pallet IP is displayed. The color is colored in the lip makeup area of the face image.
[0121]
Next, highlight processing is performed by the highlight processing unit 632 (step S603). In this case, as shown in FIG. 19, in the lower lip region, the luminance value of each pixel of the lower lip is adjusted around the center line HCL of the highlight region, and the highlight of the lower lip is expressed.
[0122]
Next, when the glitter processing unit 633 performs the glitter processing on the lip makeup area (YES in step S604), the process proceeds to step S605. On the other hand, when the glitter processing unit 633 does not perform the glitter processing on the lip makeup area (NO in step S604), the processing ends.
[0123]
Next, the pixels to be subjected to the lame processing in the lip make area are randomly extracted by the lame processing unit 633 (step S605). Next, the luminance of the extracted pixels is set high by the glitter processing unit 633, and the gloss generated on the lips due to the glitter is expressed (step S606).
[0124]
The make-up processing unit 630 performs such processing for each frame, so that lip makeup processing that follows the movement of the face image can be performed.
[0125]
The present invention can take the following modes.
[0126]
(1) In the above embodiment, the image acquisition unit 100 acquires the face image such that the head of the subject's face is on the upper side. However, the present invention is not limited to this. The face image may be acquired so as to be inclined. In this case, the direction may be parallel to a straight line connecting the center of the eye and the center of the mouth, with the head side being the upper side and the mouth side being the lower side. Furthermore, in this case, when the upper lip image specifying unit 6228 sets the ellipse region R4 in the lip detection region R3, the ellipse is set so that the minor axis of the ellipse overlaps a straight line connecting the center of the eye and the center of gravity RG of the mouth. The region R4 may be set.
[0127]
(2) In the above embodiment, the approximation function calculation unit 6213 calculates the approximation function representing the contour by the least squares method. However, the present invention is not limited to this. Good. However, in order to obtain a smooth contour, an approximation function calculated by the least square method is preferable.
[0128]
(3) Although not described in the above embodiment, the makeup area setting unit sets the makeup area on the cheek other than the eyes and the mouth, for example, and the makeup processing unit 630 colors the makeup area. May be used to perform the make-up processing. Here, when the makeup processing unit 630 colors the makeup area, the makeup area is colored using the technique of Japanese Patent Application No. 2001-230528.
[0129]
(4) The present makeup simulation program may be configured by a semiconductor integrated circuit, and may be built in, for example, a video camera.
[0130]
【The invention's effect】
As described above, according to the makeup simulation apparatus of the present invention, a makeup simulation can be performed on a moving image. Furthermore, since the makeup area is set based on the position of the eye from which the position can be accurately detected, the makeup simulation can be more realistic without the makeup area position shifting between frames.
[Brief description of the drawings]
FIG. 1 is a diagram showing a schematic configuration of a makeup simulation apparatus according to the present invention.
FIG. 2 is a block diagram of a makeup simulation apparatus according to the present invention.
FIG. 3 is a diagram illustrating an example of a face image.
FIG. 4 is a block diagram of a position detection unit.
FIG. 5 is a graph showing a pupil extraction filter.
FIG. 6 is a graph showing an eye contour extraction filter.
FIG. 7 is a graph showing an example of a reliability function.
FIG. 8 is a flowchart illustrating processing in an eye position detection mode performed by a position detection unit.
FIG. 9 is a diagram for explaining how an eye image is extracted.
FIG. 10 is a flowchart illustrating a process performed by a position detection unit in an eye tracking mode.
FIG. 11 is a block diagram of an eye makeup area setting unit.
FIG. 12 is a flowchart illustrating a process performed by the eye makeup area setting unit on one frame.
FIG. 13 is a diagram for explaining how an outline pixel of an eye and an eyebrow is detected.
FIG. 14 is a diagram for explaining how an eye makeup area is set.
FIG. 15 is a diagram showing a list image of an eye make pattern and an eye line pattern.
FIG. 16 shows a block diagram of a lip make setting unit.
FIG. 17 is a diagram for explaining how a lip detection area is set.
FIG. 18 is a flowchart illustrating a process of detecting a contour of a lower lip performed by a lip makeup area setting unit on one frame.
FIG. 19 is a diagram for explaining how a contour pixel of the lower lip is detected.
FIG. 20 is a flowchart illustrating a process of detecting the contour of the upper lip performed on one frame by the lip makeup area setting unit.
FIG. 21 is a block diagram of a makeup processing unit.
FIG. 22 is a flowchart illustrating a process performed by a make-up processing unit for each frame.
FIG. 23 is a diagram showing a list image of a lip make pattern and a lip color pattern.
[Explanation of symbols]
100 Image acquisition unit
200 image memory
300 Display
400 makeup pattern storage
500 color pattern storage
600 Program execution unit
610 Position detector
611 Detection area setting unit
612 Image extraction unit
6121 Density correction unit
6122 Subsampling unit
6123 Filter processing unit
6124 Area calculation unit
6125 Image judgment unit
613 Position Identification Unit
614 Prediction position calculation unit
615 Judgment unit
616 Grayscale image conversion unit
620 Makeup area setting unit
621 Eye Make Area Setting Unit
6211 Noise removal unit
6212 Edge detection unit
6212 Noise removal unit
6213 Approximate function calculator
6214 Control point creation unit
6215 Boundary line calculation unit
622 Lip Make Area Setting Part
6221 RG image conversion unit
6222 grayscale image conversion unit
6223 Lip detection area setting unit
6224 Filter processing unit
6225 Area calculation unit
6226 Lower lip image specifying unit
6227 Lower lip contour detector
6228 Upper lip image specifying unit
6229 Upper lip contour detector
6230 approximate function calculator
6231 Contour correction unit
630 Makeup processing unit
631 colored part
632 Highlight processing unit
633 lame processing unit
700 List image creation unit

Claims (10)

Image acquisition means for acquiring a subject's face image taken at a predetermined frame period,
Position detecting means for detecting the position of the eyes of the face image for each frame;
Makeup area setting means for setting a predetermined area of the face image as a makeup area based on the position of the eye detected for each frame,
A makeup simulation apparatus, comprising: makeup processing means for coloring the makeup area with a predetermined color and performing makeup processing on a face image. The makeup area setting means includes a lip detection area setting means for setting an area including the upper lip and the lower lip as a lip detection area based on the position of the eye, and a contour of the lower lip based on color information from the lip detection area. Lower lip contour detecting means for detecting the upper lip contour detecting means for detecting the contour of the upper lip based on the brightness from the lip detection area, and the area surrounded by the detected upper lip and lower lip contours is 2. The makeup simulation apparatus according to claim 1, wherein the makeup simulation area is set as a makeup area. The makeup area setting means sets an upper lip detection area around the center of gravity of the lower lip image in the lip detection area, and specifies the upper lip image from the image included in the set upper lip detection area. The makeup simulation apparatus according to claim 2, further comprising an image specifying unit. The makeup area setting means calculates an approximate function representing the contour of the upper lip from the array of pixels constituting the contour of the upper lip detected by the upper lip contour detecting means, and calculates the approximate function of the lower lip detected by the lower lip contour detecting means. 4. The makeup simulation apparatus according to claim 2, further comprising an approximation function calculating unit configured to calculate an approximation function representing an outline of the lower lip from an array of pixels constituting the outline. The method according to claim 4, wherein the approximate function calculating means divides the upper contour line of the upper lip into two regions at the center in the left-right direction, and calculates an approximate function representing the contour line in each region. Makeup simulation device. 6. The make-up processing unit according to claim 4, wherein the approximation function representing the contour of the lip calculated by the approximation function calculation unit performs highlight processing for representing gloss of the lips. Makeup simulation device. Storage means for storing a plurality of predetermined makeup patterns,
List image creating means for creating a list image of the plurality of makeup patterns,
Input means for receiving an operation command from an operator,
The makeup area setting means sets the makeup area based on a makeup pattern selected by an operator using the input means from the list image of the plurality of makeup patterns,
2. The makeup area according to claim 1, wherein the makeup area is colored based on a makeup pattern selected by an operator using the input means from a list image of the plurality of makeup patterns. The makeup simulation apparatus according to any one of claims 1 to 7. A computer acquires a subject's face image taken at a predetermined frame cycle,
The computer sets a predetermined area of the face image as a makeup area based on the position of the eye detected for each frame,
A makeup simulation method, wherein a computer colors the makeup area with a predetermined color and performs a makeup process on a face image. Image acquisition means for acquiring a subject's face image taken at a predetermined frame period,
Position detection means for detecting the position of the eyes of the face image for each frame,
Makeup area setting means for setting a predetermined area of the face image as a makeup area based on the position of the eye detected for each frame,
A makeup simulation program that causes a computer to function as makeup processing means for coloring the makeup area with a predetermined color and performing makeup processing on a face image. Image acquisition means for acquiring a subject's face image taken at a predetermined frame period,
Position detection means for detecting the position of the eyes of the face image for each frame,
Makeup area setting means for setting a predetermined area of the face image as a makeup area based on the position of the eye detected for each frame,
A computer-readable recording medium storing a makeup simulation program, wherein the makeup area is colored with a predetermined color and a computer functions as makeup processing means for performing makeup processing on a face image.

Images