JP4329398B2 - Face detection apparatus and method, program, and recording medium - Google Patents

Description

【００９１】
キャリアｓ（ｘ，ｙ）は、複数関数を用いて、下記式（２）のように表現される。ここで、座標値（ｕ_０，ｖ_０）は空間周波数を表し、またＰはコサイン成分の位相を表す。
【００９２】
【数２】

【００９３】
ここで、上記式（２）に示すキャリアは、下記式（３）に示すように、実数成分Ｒｅ（ｓ（ｘ，ｙ）と虚数成分Ｉｍ（ｓ（ｘ，ｙ））に分離することができる。
【００９４】
【数３】

【００９５】
一方、２次元ガウス分布からなるエンベローブは、下記式（４）のように表現される。
【００９６】
【数４】

【００９７】
ここで、座標軸（ｘ_０，ｙ_０）はこの関数のピークであり、定数ａ及びｂはガウス分布のスケール・パラメータである。また、添え字ｒは、下記式（５）に示すような回転操作を意味する。
【００９８】
【数５】

【００９９】
従って、上記式（２）及び（４）より、ガボア・フィルタは、下記式（６）に示すような空間関数として表現される。
【０１００】
【数６】

【０１０１】
本適用例に係る前処理部は、８種類の方向と３通りの周波数を採用して、合計２４個のガボア・フィルタを用いて顔抽出処理を行う。
【０１０２】
ガボア・フィルタのレスポンスは、Ｇ_ｉをｉ番目のガボア・フィルタとし、ｉ番目のガボアの結果（Gabor Jet）をＪ_ｉとし、入力イメージをＩとすると、下記式（７）で表される。この式（７）の演算は、実際には高速フーリエ変換を用いて高速化することができる。
【０１０３】
【数７】

【０１０４】
作成したガボア・フィルタの性能を調べるためには、フィルタリングして得られた画素を再構築することによって行う。再構築されたイメージＨは、下記式（８）のように表される。
【０１０５】
【数８】

【０１０６】
そして、入力画像Ｉと再構築された画像ＨとのエラーＥは、下記式（９）のように表される。
【０１０７】
【数９】

【０１０８】
このエラーＥを最小にするような最適なａを求めることにより再構築することができる。なお、ガボア・フィルタリングに関しては、認識タスクに応じてフィルタの種類を変更するようにしてもよい。
【０１０９】
低周波でのフィルタリングでは、フィルタリング後のイメージすべてをベクトルとして持っているのは冗長である。そこで、ダウンサンプリングして、ベクトルの次元を落とすようにしてもよい。ダウンサンプリングされた２４種類のベクトルを一列に並べ、長いベクトル（上述したパターンベクトル）にする。
【０１１０】
パターン識別部６４は、外部から供給される学習用のデータすなわち教師データを用いて、暫定的な識別関数を得た後、当該識別関数を前処理部６３からパターンベクトルとして得られた３６０画素分のスコア画像に試して顔の検出を行う。そして、検出に成功したものを顔データとして出力する。また検出に失敗したものを非顔データとして学習データに追加して、更に学習をし直す。
【０１１１】
本適用例では、パターン識別部６４における顔認識に関して、パターン認識の分野で最も学習汎化能力が高いとされるサポートベクタマシン（Support Vector Machine：ＳＶＭ）を用いて該当する顔か否かの識別を行う。
【０１１２】
サポートベクタマシン自体に関しては、例えばB.sholkopf外著の報告（B.Sholkopf、C.Burges、A.Smola、”Advance in Kernel Support Vector earning”、The MIT Press、1999.）を挙げることができる。本願出願人が行った予備実験の結果からは、サポートベクタマシンによる顔認識方法は、主成分分析（ＰＣＡ）やニューラル・ネットワークを用いる手法に比べ、良好な結果を示すことが判っている。
【０１１３】
サポートベクタマシンは、識別関数に線形識別器（パーセプトロン）を用いた学習機械であり、カーネル関数を使うことで非線形空間に拡張することができる。また識別関数の学習では、クラス間分離のマージンを最大にとるように行われ、その解は２次数理計画法を解くことで得られるため、グローバル解に到達できることを理論的に保証することができる。
【０１１４】
通常、パターン認識の問題は、テスト・サンプルｘ＝（ｘ１、ｘ２、・・・、ｘｎ）に対して、下記式（１０）で与えられる識別関数ｆ（ｘ）を求めることである。
【０１１５】
【数１０】

【０１１６】
ここで、サポートベクタマシンの学習用の教師ラベルを下記式（１１）のようにおく。
【０１１７】
【数１１】

【０１１８】
すると、サポートベクタマシンにおける顔パターンの認識を下記式（１２）に示す制約条件の下での重み因子ｗの２乗の最小化する問題としてとらえることができる。
【０１１９】
【数１２】

【０１２０】
このような制約のついた問題は、ラグランジュの未定定数法を用いて解くことができる。すなわち、下記式（１３）に示すラグランジュをまず導入し、次いで、下記式（１４）に示すように、ｂ、ｗの各々について偏微分する。
【０１２１】
【数１３】

【０１２２】
【数１４】

【０１２３】
この結果、サポートベクタマシンにおける顔パターンの識別を下記式（１５）に示す２次計画問題としてとらえることができる。
【０１２４】
【数１５】

【０１２５】
特徴空間の次元数が、訓練サンプルの数よりも少ない場合は、スクラッチ変数ξ≧０を導入して、制約条件を下記式（１６）のように変更する。最適化については、下記式（１７）の目的関数を最小化する。
【０１２６】
【数１６】

【０１２７】
【数１７】

【０１２８】
上記式（１７）において、Ｃは、制約条件をどこまで緩めるかを指定する係数であり、実験的に値を決定する必要がある。ラグランジュ定数ａに関する問題は下記式（１８）のように変更される。
【０１２９】
【数１８】

【０１３０】
しかし、上記式（１８）のままでは、非線型の問題を解くことはできない。そこで、本適用例では、カーネル関数Ｋ（ｘ，ｘ３）を導入して、一旦、高次元の空間に写像して（カーネル・トリック）、その空間で線形分離することにしている。従って、元の空間では非線型分離していることと同等となる。
【０１３１】
カーネル関数は、ある写像Φを用いて下記式（１９）のように表される。
【０１３２】
【数１９】

【０１３３】
また、上記（１０）式に示した識別関数も、下記式（２０）のように表すことができる。
【０１３４】
【数２０】

【０１３５】
また学習に関しても、下記式（２１）に示す２次計画問題としてとらえることができる。
【０１３６】
【数２１】

【０１３７】
カーネルとしては、下記式（２２）に示すガウシアン・カーネル（ＲＢＦ：Radius Basic Function）等を用いることができる。
【０１３８】
【数２２】

【０１３９】
このようにパターン識別部６４は、前処理部６３から与えられたスコア画像に基づくパターンベクトルについて、当該スコア画像内に顔データが存在するか否かを判断し、存在する場合のみ当該スコア画像の画像領域における左上位置（座標）及びその大きさ（縦横の画素数）と、当該スコア画像の切出し元となるスケール画像のフレーム画像に対する縮小率（すなわち上述の５段階のうちの該当する段階）とをリスト化し、これをリストデータとして内部メモリ３８１Ａに格納する。
【０１４０】
この後、パターン識別部６４は、ウィンドウ切出部６１に対して、第１のスケール画像のうち先頭のウィンドウ画像の顔検出が終了した旨を通知することにより、当該ウィンドウ切出部６１から第１のスケール画像のうち次にスキャンされたウィンドウ画像をテンプレートマッチング部６２に送出させる。
【０１４１】
そしてテンプレートマッチング部６２は、当該ウィンドウ画像がテンプレートにマッチングした場合のみスコア画像として前処理部６３に送出する。前処理部６３は、当該スコア画像をパターンベクトルに変換してパターン識別部６４に送出する。パターン識別部６４は、パターンベクトルから識別結果として得られた顔データに基づいてリストデータを生成して内部メモリ３８１Ａに格納する。
【０１４２】
このようにウィンドウ切出部６１おいて第１のスケール画像から切り出した全てのウィンドウ画像について、スキャン順にテンプレートマッチング部６２、前処理部６３及びパターン識別部６４の各処理を行うことにより、当該第１のスケール画像から撮像結果に存在する顔画像を含むスコア画像を複数検出することができる。
【０１４３】
この後、パターン識別部６４は、入力画像スケール変換部６０に対して、第１のスケール画像の顔検出が終了した旨を通知することにより、当該入力画像スケール変換部６０から第２のスケール画像をウィンドウ切出部６１に送出させる。
【０１４４】
そして第２のスケール画像についても、上述した第１のスケール画像と同様の処理を行って、当該第２のスケール画像から撮像結果に存在する顔画像を含むスコア画像を複数検出した後、第３〜第５のスケール画像についても同様の処理を順次行う。
【０１４５】
かくしてパターン識別部６４は、撮像画像であるフレーム画像を５段階に縮小した第１〜第５のスケール画像について、当該撮像画像内に存在する顔画像を含むスコア画像をそれぞれ複数検出した後、その結果得られたリストデータをそれぞれ内部メモリ３８１Ａに格納する。この場合、元のフレーム画像内での顔画像のサイズによっては、全くスコア画像が得られない場合もあるが、少なくとも１以上（２又は３以上でもよい）のスケール画像でスコア画像が得られれば、顔検出処理を続行することとする。
【０１４６】
ここで、各スケール画像において顔画像を含む複数のスコア画像は、ウィンドウ切出部６１におけるスキャンが２画素ずつすらして行われたため、実際に顔がある領域とその近傍領域とで高い相関性があり、隣接するスコア画像同士で相互に重なり合う画像領域を含むこととなる。
【０１４７】
そこで続く重なり判定部６５は、内部メモリ３８１Ａに格納されている第１〜第５のスケール画像ごとに複数のリストデータをそれぞれ読み出して、当該各リストデータに含まれるスコア画像同士を比較して、相互に重なり合う領域を含むか否かを判定する。
【０１４８】
その際、重なり判定部６５は、図７に示すように、２つのスコア画像Ｐ１、Ｐ２の左上角部の位置（座標）がそれぞれ（Ｘ_Ａ，Ｙ_Ａ）、（Ｘ_Ｂ，Ｙ_Ｂ）であり、その大きさ（縦横の画素数）がそれぞれＨ_Ａ×Ｌ_Ａ、Ｈ_Ｂ×Ｌ_Ｂであるとき、ｄＸ（＝Ｘ_Ｂ−Ｘ_Ａ）、ｄＹ（＝Ｙ_Ｂ−Ｙ_Ａ）として、下記式（２３）を満たすか否かによって、当該スコア画像Ｐ１、Ｐ２同士が重なり合うか否かを判定することができる。
【０１４９】
【数２３】

【０１５０】
重なり判定部６５は、当該判定結果に基づいて、スコア画像同士で重なり合う領域を除去することにより、各スケール画像において、最終的に複数のスコア画像を互いに重なることなく寄せ集めた単一の画像領域として得ることができ、当該画像領域を顔決定データとして新たに内部メモリ３８１Ａに格納する。
【０１５１】
また重なり判定部６５は、テンプレートマッチング部６２において顔画像でないと判断された場合には、そのまま何もすることなく、内部メモリ３８１Ａの格納も行わない。
【０１５２】
（２−１−３）第１の適用例における動作及び効果
以上の構成において、このロボット装置２００では、ＣＣＤカメラ５０により撮像したフレーム画像を縮小率が相異なる複数のスケール画像に変換した後、当該各スケール画像の中からそれぞれ所定サイズのウィンドウ画像を所定画素ずつずらすようにスキャンしながら１枚ずつ切り出す。
【０１５３】
このウィンドウ画像について、平均的な顔画像を表すテンプレートを用いてマッチングをとって大まかに顔画像であるか否かを判断するようにして、明らかに顔画像でないウィンドウ画像を除去することにより、後段の顔検出処理に要する演算量及び時間をその分減少させることができる。
【０１５４】
続いてテンプレートマッチングで顔画像であると判断されたウィンドウ画像（すなわちスコア画像）について、当該スコア画像の矩形領域の４隅部分を除去した後、濃淡補正及び続くコントラスト強調の平滑化を行い、更に１本のパターンベクトルに変換する。
【０１５５】
そして当該パターンベクトルについて、元のスコア画像内での顔検出を行って顔データ又は非顔データを判断し、顔データが存在するスコア画像の画像領域の位置（座標）及びその大きさ（画素数）と、当該スコア画像の切出し元となるスケール画像のフレーム画像に対する縮小率とをリスト化したリストデータを生成する。
【０１５６】
このように各スケール画像毎にそれぞれ全てのスコア画像についてリストデータを生成した後、当該各リストデータに含まれるスコア画像同士を比較して、相互に重なり合う領域を除去した顔決定データを求めることにより、元のフレーム画像から顔画像を検出することができる。
【０１５７】
このような顔検出タスク処理のうち特にテンプレートマッチング処理は、比較的構成が簡易な演算器にもたやすく実装できる上に、画像圧縮等で利用されるブロックマッチングの手法と類似する処理であることからＣＰＵを用いた高速処理を行うハードウェアが数多く存在する。従ってテンプレートマッチング処理に関してはさらなる高速化が可能である。
【０１５８】
以上の構成によれば、このロボット装置２００において、ＣＣＤカメラ５０により撮像したフレーム画像について顔画像を検出する顔検出タスク処理の際、当該フレーム画像を相異なる縮小率で縮小した各スケール画像の中からそれぞれ所定サイズのウィンドウ画像を所定画素ずつずらすようにスキャンしながら１枚ずつ切り出した後、平均的な顔画像を表すテンプレートを用いてマッチングをとって大まかに顔画像であるか否かを判断するようにして、明らかに顔画像でないウィンドウ画像を除去するようにしたことにより、当該テンプレートマッチングで顔画像であると判断されたスコア画像に対する種々の顔検出処理に要する演算量及び時間をその分減少させることができ、ロボット装置２００全体の制御を司る主制御部３８１の処理負担を軽減させることができ、かくしてリアルタイム性を格段と向上し得るロボット装置２００を実現できる。
【０１５９】
（２−２）第２の適用例
（２−２−１）ロボットの内部構成
第２の適用例において、胴体部ユニット２０２（図１）の胴体下部を形成する腰ベースの背面側には、図５との対応部分に同一符号を付した図８に示すように、上述した主制御部３８１に加えて、この主制御部３８１と接続されたバスを介して内部メモリ３８１Ａ、ＤＭＡ（Direct Memory Access）コントローラ７０及び演算処理部（ＤＳＰ（Digital Signal Processor）及び画像処理用エンジン等）７１が相互に接続されており、それ以外は第１の適用例と同様に構成されている。
【０１６０】
この場合、主制御部３８１のみが内部メモリ３８１Ａの管理を行うのではなく、ＤＭＡコントローラ７０が主制御部３８１又は演算処理部７１にデータを転送する。一方、当該主制御部３８１又は演算処理部７１から得られた演算結果をＤＭＡコントローラ７０を介して内部メモリ３８１Ａに格納するようになされている。
【０１６１】
このようにデータ転送をＤＭＡコントローラ７０が行うと共に、データ演算を主制御部３８１又は演算処理部７１に振り分けて互いに独立に行わせることにより、主制御部３８１にかかる負担を低減させ得るようになされている。
【０１６２】
（２−２−２）顔検出タスク機能に関する主制御部３８１の処理
第２の適用例の場合、顔検出タスク機能は、ＤＭＡコントローラ７０を介した主制御部３８１及び演算処理部７１における各種処理により実現されている。
【０１６３】
ここで、かかる顔検出タスク機能に関する主制御部３８１及び演算処理部７１の各処理内容を機能的に分類すると、図９に示すように、入力画像スケール変換部８０、ウィンドウ切出部８１、テンプレートマッチング部８２、重なり判定部８３、スケール変換及び切出部８４、前処理部８５及びパターン識別部８６に分けることができる。
【０１６４】
このうち入力画像スケール変換部８０、ウィンドウ切出部８１及びテンプレートマッチング部８２は、演算処理部７１が処理すると共に、重なり判定部８３、スケール変換及び切出部８４、前処理部８５及びパターン識別部８６は、主制御部３８１が処理するようになされている。演算処理部７１と主制御部３８１との処理の切り換えはＤＭＡコントローラ７０が行う。
【０１６５】
先ず、入力画像スケール変換部８０は、ＣＣＤカメラ５０（図８）からの画像信号Ｓ１Ａに基づくフレーム画像を内部メモリ３８１Ａから読み出して、当該フレーム画像を縮小率が相異なる複数のスケール画像（上述した第１〜第５のスケール画像）に変換する。
【０１６６】
続くウィンドウ切出部８１は、第１〜第５のスケール画像のうち、まず第１のスケール画像に対して、画像左上を起点として順に画像右下まで、適当な画素（例えば２画素）分を飛ばしながらスキャンするようにして、４００（＝20×20）画素の矩形領域でなるウィンドウ画像を順次切り出す。
【０１６７】
その際、ウィンドウ切出部８１は、第１のスケール画像から切り出した複数のウィンドウ画像のうち先頭のウィンドウ画像を後段のテンプレートマッチング部８２に送出する。
【０１６８】
テンプレートマッチング部８２は、ウィンドウ切出部８１から得られた先頭のウィンドウ画像について、平均的な顔画像をテンプレートとし、大まかなマッチングをとって、当該テンプレートとの相関度（テンプレートのうちマッチングする画素数の割合）を相関度データとして当該ウィンドウ画像と共にＤＭＡコントローラ７０を介して内部メモリ３８１Ａに格納する。
【０１６９】
具体的には、図１０に示すように、任意のスケール画像Ｆ１から切り出したウィンドウ画像Ｗ１について、平均的な顔画像であるテンプレートＴ１をとり、当該テンプレートＴ１との相関度をとった結果として、当該相関度を表す画像Ｒ１が検出され、これを相関度データとして内部メモリ３８１Ａに格納するようになされている。
【０１７０】
この第２の適用例の場合には、ウィンドウ画像について顔画像であると判断されたスコア画像以外をふるい落とす第１の適用例とは異なり、高速処理を実現すべく、当該ウィンドウ画像をスコア画像か否かにかかわりなく後段の重なり判定部８３に送出する。
【０１７１】
この後、テンプレートマッチング部８２は、ウィンドウ切出部８１に対して、第１のスケール画像のうち先頭のウィンドウ画像のテンプレートマッチングが終了した旨を通知することにより、当該ウィンドウ切出部８１から第１のスケール画像のうち次にスキャンされたウィンドウ画像をテンプレートマッチング部８２に送出させる。
【０１７２】
このようにテンプレートマッチング部８２は、ウィンドウ切出部８１において第１のスケール画像から切り出した全てのウィンドウ画像について、スキャン順にそれぞれ相関度データを生成する。
【０１７３】
この後、テンプレートマッチング部８２は、入力画像スケール変換部８０に対して、第１のスケール画像の顔検出が終了した旨を通知することにより、当該入力画像スケール変換部８０から第２のスケール画像をウィンドウ切出部８１に送出させる。
【０１７４】
そして第２のスケール画像についても、上述した第１のスケール画像と同様の処理を行って、当該第２のスケール画像から全てのウィンドウ画像に対応する相関度データを生成した後、第３〜第５のスケール画像についても同様の処理を順次行う。
【０１７５】
かくしてテンプレートマッチング部８２は、撮像画像であるフレーム画像を５段階に縮小した第１〜第５のスケール画像について、それぞれ切出した複数のウィンドウ画像に対応する相関度データを生成して、当該ウィンドウ画像と共にＤＭＡコントローラ７０を介して内部メモリ３８１Ａに格納する。
【０１７６】
以上が演算処理部７１における処理内容であり、ＤＭＡコントローラ７０の制御に応じて、主制御部３８１に切り換えられ、以下は当該主制御部３８１における処理内容となる。
【０１７７】
先ず、重なり判定部８３は、図１１の重なり判定処理手順Ｒ１１をステップＳＰ０から実行し、第１〜第５のスケール画像毎に、当該各スケール画像内のウィンドウ画像を切り出す際のスキャン順と同じ順番で、ウィンドウ画像をこれに付随する相関度データと共にＤＭＡコントローラ７０を介して内部メモリ３８１Ａから読み出す（ステップＳＰ１）。
【０１７８】
続いて重なり判定部８３は、該当するスケール画像における複数のウィンドウ画像について、スキャン順にウィンドウ画像同士を上述した式（２３）を用いて比較して、相互に重なり合う領域を含むか否かを判定する（ステップＳＰ２）。
【０１７９】
そして重なり判定部８３は、相互に重なり合う領域を含む場合のみ、当該ウィンドウ画像同士をそれぞれ対応する相関度の大小で比較して、相関度の大きいほうをスコア画像として、内部メモリ３８１Ａ内に設定する候補リストに追加して格納する（ステップＳＰ３）。
【０１８０】
そして重なり判定部８３は、該当するスケール画像内の全てのウィンドウ画像について、相関度を比較した場合の大きい方をスコア画像として候補リストに追加する（ステップＳＰ４）。一方、小さい方は候補リストに加えることなく又は既に候補リストに存在する場合には削除する（ステップＳＰ５）ようにして順次候補リストの書き換えを行う。
【０１８１】
やがて重なり判定部８３は、該当するスケール画像内の全てのウィンドウ画像について、比較的相関度が高い複数のスコア画像を候補リストとして決定した後、他のスケール画像についても比較的相関度が高い複数のスコア画像を候補リストとして決定する。なお、候補リストとして追加される比較的相関度が高いスコア画像の数は、各スケール画像毎にそれぞれ１０以下に絞られることが実験結として得ることができた。
【０１８２】
このようにして重なり判定部８３では、第１〜第５のスケール画像について、スコア画像を相関度の最も高いピーク値のみならず、そのピーク値の近傍領域をも対象として候補リストに加えるようにしたことにより、テンプレートマッチング部８２でのマッチング結果と後段のパターン識別部８６におけるサポートベクタマシンの処理結果とのずれをその分低下させることができる。
【０１８３】
実際上、かかる重なり判定部８３における候補リストの作成処理は、数回の加算及び乗算によって実行可能であるため、後段の前処理部８５及びパターン識別部８６による演算処理と比較すると、極めて高速に短時間に処理することができる。そして候補リストとして残ったスコア画像について前処理部８５及びパターン識別部８６による処理を行えば、当該前処理部８５及びパターン識別部８６においても処理負担を軽減させることができる。
【０１８４】
この後、スケール変換及び切出部８４は、重なり判定部８３において各スケール画像毎に候補リストとして内部メモリ３８１Ａに格納された複数のスコア画像について、当該内部メモリ３８１ＡからＤＭＡコントローラ７０を介して第１のスケール画像内でかつ候補リストの決定順に１つずつスコア画像を読み出して、前処理部８５に送出する。
【０１８５】
前処理部８５は、スケール変換及び切出部８４から得られたスコア画像について、矩形領域でなる当該スコア画像（４００画素）から人間の顔画像とは無関係である背景部分に相当する４隅の領域を除去するために、当該４隅の領域を切り取ったマスクを用いて３６０画素分を抽出する。
【０１８６】
そして前処理部８５は、撮像時の照明により濃淡で表される被写体の傾き条件を解消すべく、当該抽出した３６０画素分のスコア画像のうち顔画像として最適な部位を基準とする平面を形成するように画素単位で濃淡値に補正をかける。
【０１８７】
続いて前処理部８５は、当該３６０画素分のスコア画像のコントラストを強調した結果をヒストグラム平滑化処理を行うことにより、ＣＣＤカメラ５０のゲインや照明の強弱によらずに検出できるようにする。
【０１８８】
次いで、前処理部８５は、上述したガボア・フィルタリング処理を行うことにより、当該３６０画素分のスコア画像をベクトル変換し、得られたベクトル群を更に1本のパターンベクトルに変換する。
【０１８９】
パターン識別部８６は、上述したサポートベクタマシンを用いて、前処理部８５からパターンベクトルとして得られた３６０画素分のスコア画像に試して顔か否かの識別を学習しながら行う。
【０１９０】
このようにパターン識別部８６は、前処理部８５から与えられたスコア画像に基づくパターンベクトルについて、当該スコア画像内での顔データに相当する画像領域の位置（座標）及びその大きさ（画素数）と、当該スコア画像の切出し元となるスケール画像のフレーム画像に対する縮小率（すなわち上述の５段階のうちの該当する段階）とをリスト化し、これをリストデータとして内部メモリ３８１Ａに格納する。
【０１９１】
この後、パターン識別部８６は、スケール変換及び切出部８４に対して、第１のスケール画像のうち最初のスコア画像の顔検出が終了した旨を通知することにより、当該スケール変換及び切出部８４から第１のスケール画像のうち次のスコア画像を前処理部８５に送出する。前処理部８５は、当該スコア画像をパターンベクトルに変換してパターン識別部８６に送出する。パターン識別部８６は、パターンベクトルから得られた顔データに基づいてリストデータを生成して内部メモリ３８１Ａに格納する。
【０１９２】
このようにスケール変換及び切出部８４において第１のスケール画像内で候補リストにある全てのスコア画像について、順番に前処理部８５及びパターン識別部８６の各処理を行うことにより、当該第１のスケール画像から撮像結果に存在する顔画像を検出することができる。
【０１９３】
この後、パターン識別部８６は、スケール変換及び切出部８４に対して、第１のスケール画像の顔検出が終了した旨を通知することにより、当該第２のスケール画像についても、上述した第１のスケール画像と同様の処理を行って、当該第２のスケール画像から撮像結果に存在する顔画像を検出した後、第３〜第５のスケール画像についても同様の処理を順次行う。
【０１９４】
かくしてパターン識別部８６は、撮像画像であるフレーム画像を５段階に縮小した第１〜第５のスケール画像について、当該撮像画像内に存在する顔画像をそれぞれ検出した後、その結果得られた顔データに基づいてそれぞれリストデータを生成して内部メモリ３８１Ａに格納する。
【０１９５】
（２−２−３）第２の適用例における動作及び効果
以上の構成において、このロボット装置２００では、ＣＣＤカメラ５０により撮像したフレーム画像を縮小率が相異なる複数のスケール画像に変換した後、当該各スケール画像の中からそれぞれ所定サイズのウィンドウ画像を所定画素ずつずらすようにスキャンしながら１枚ずつ切り出す。
【０１９６】
このウィンドウ画像について、平均的な顔画像を表すテンプレートを用いてマッチングをとって当該テンプレートとの相関度を表す相関度データを生成する。このように各スケール画像毎にそれぞれ全てのウィンドウ画像についてスキャン順にそれぞれ相関度データを生成する。
【０１９７】
ここまでを演算処理部７１において実行すると共にこれ以降の処理を主制御部３８１において実効することにより、その分当該主制御部３８１の処理負担を軽減することができる。
【０１９８】
続いて各スケール画像における複数のウィンドウ画像について、スキャン順にウィンドウ画像同士を比較して、相互に重なり合う領域を含む場合には、更に相関度を比較して大きい方のみをスコア画像として残るように候補リストに追加する。この結果、各スケール画像毎に、比較的相関度が高い複数のスコア画像を候補リストとして決定することができ、後段の顔識別処理の際に識別精度を一層向上させることができる。
【０１９９】
続いて各スケール画像毎に候補リストとして内部メモリ３８１Ａに格納された複数のスコア画像について、当該スコア画像の矩形領域の４隅部分を除去した後、濃淡補正及び続くコントラスト強調の平滑化を行い、更に１本のパターンベクトルに変換する。
【０２００】
そして当該パターンベクトルについて、元のスコア画像内での顔検出を行って顔データ又は非顔データを判断し、顔データが存在するスコア画像の画像領域の位置（座標）及びその大きさ（画素数）と、当該スコア画像の切出し元となるスケール画像のフレーム画像に対する縮小率とをリスト化したリストデータを生成する。
【０２０１】
このように各スケール画像毎にそれぞれ全てのスコア画像についてリストデータを生成することにより、元のフレーム画像から顔画像を検出することができる。
【０２０２】
かくして顔検出タスク処理に関して、ロボット装置２００の全体の制御を司る主制御部３８１に加えて、演算処理部７１を設け、内部メモリ３８１Ａに対するデータの読み書きをＤＭＡコントローラ７０の制御を介して主制御部３８１及び演算処理部７１の双方が行い得るようにしたことにより、当該主制御部３８１及び演算処理部７１がそれぞれ処理を分配することができる分、主制御部３８１にかかる演算量及び演算時間の負担を格段と低減させることができる。
【０２０３】
以上の構成によれば、このロボット装置２００において、ＣＣＤカメラ５０により撮像したフレーム画像について顔画像を検出する顔検出タスク処理の際、当該フレーム画像を相異なる縮小率で縮小した各スケール画像の中からそれぞれ所定サイズのウィンドウ画像を所定画素ずつずらすようにスキャンしながら１枚ずつ切り出した後、平均的な顔画像を表すテンプレートを用いてマッチングをとって当該テンプレートとの相関度を表す相関度データをそれぞれ生成し、当該ウィンドウ画像同士で相互に重なり合う領域を含む場合には相関度を比較して大きい方のみを候補リストに残るようにしたことにより、当該候補リストにある比較的相関度が高い複数のスコア画像に対する種々の顔検出処理に要する演算量及び時間をその分減少させることができるのみならず、顔識別処理の際の識別精度を一層向上させることができる。
【０２０４】
これに加えて顔検出タスク処理に関して、ロボット装置２００の全体の制御を司る主制御部３８１と演算処理部７１とで処理を分担して行うようにしたことにより、当該主制御部３８１にかかる演算量及び演算時間の負担を格段と低減させることができ、かくしてリアルタイム性を格段と向上し得るロボット装置２００を実現できる。
【０２０５】
（３）実施の形態
次に、本発明の実施の形態について説明する。上述したように、テンプレートマッチングを行って顔候補を抽出し（第１の工程）、この顔候補の中からＳＶＭ等により顔領域を判定して（第２の工程）顔領域を検出する方法において、第１の工程においては、単純に正規化相関値の代償により顔候補を決定しているため、顔候補の見逃しを軽減しようとした場合、閾値を上げる方法又は間引きを減らす方法をとることができるものの、閾値を下げると演算量が増大してしまい、ロボット装置等のリソースの限られた環境においては好ましくない場合がある。一方、閾値を上げると、第２の工程において顔判定するための候補画像が減るため、演算量を減らすことができるものの、本来顔である画像も候補画像から取り除いてしまい、顔画像を見逃してしまう場合がある。
【０２０６】
そこで、本願発明者等は、テンプレートと同一サイズの顔領域（顔画像）が入力画像内に存在する場合、この顔画像とテンプレートとの相関をとれば、テンプレートサイズ近傍では最も相関値が大きくなることに着目し、顔領域の候補を絞り込む際に、局所的な絞り込みを行うアルゴリズムを使用することにより、本来顔である画像を見逃すことなく顔候補画像を低減して後段の第２の工程にて顔判定する計算量を低減する方法を見出した。具体的には、入力画像と所定サイズの平均顔のテンプレートとの正規化相関をとった相関値の集合であるマッチング結果における相関値の局所最大値に基づき候補となる顔領域を抽出するものである。以下、この顔検出方法について詳細に説明する。
【０２０７】
本実施の形態における顔検出装置は、上述の第１及び第２の適用例と同様の２足歩行のロボット装置等のＣＰＵ及びメモリ等のリソースが限られたシステムにおいて、性能を保持しつつ顔検出に要する計算量を削減する場合に好適である。本実施の形態においても、上述の第１及び第２の適用例と同様、図１乃至図４に示すロボット装置２００に搭載されるものとし、その内部構成は、上述の適用例と同様とし、その詳細な説明は省略する。
【０２０８】
（３−１）顔検出タスク機能に関する主制御部３８１の処理
本実施の形態におけるロボット装置２００に搭載された顔検出タスク機能について説明する。上述した如く、ロボット装置２００には、ＣＣＤカメラ５０を介して内部メモリ３８１Ａに記憶されるフレーム画像について、当該フレーム画像の中から人間の顔画像を検出する顔検出タスク機能が搭載されている。そしてこの顔検出タスク機能は、主制御部３８１における各種処理により実現されている。
【０２０９】
ここで、かかる顔検出タスク機能に関する主制御部３８１の処理内容を機能的に分類すると、図１２に示すように、テンプレートサイズ及び入力画像スケール変換決定部９０、ウィンドウ切り出し部９１、テンプレートマッチング部９２、スケール変換及び切り出し部９３、前処理部９４、及びパターン識別部９５に分けることができる。
【０２１０】
入力画像スケール変換決定部９０は、第１の適用例と同様に、ＣＣＤカメラ５０（図５）からの画像信号Ｓ１Ａに基づくフレーム画像を内部メモリ３８１Ａから読み出して、当該フレーム画像を縮小率が相異なる複数のスケール画像（第１〜第５のスケール画像）に変換すると共に、後段のテンプレートマッチング部９２で使用するテンプレートサイズ、即ち、顔画像のサイズを選択する（以下、これを第１のテンプレートサイズという）。
【０２１１】
続くウィンドウ切出部９１は、第１〜第５のスケール画像のうち、まず第１のスケール画像に対して、画像左上を起点として順に画像右下まで、適当な画素（例えば２画素）分を飛ばしながらスキャンするようにして、４００（＝20×20）画素の矩形領域でなるウィンドウ画像を順次切り出す。
【０２１２】
その際、ウィンドウ切出部９１は、第１のスケール画像から切り出した複数のウィンドウ画像のうち先頭のウィンドウ画像を、スケール変換決定部９１で選択した第１のテンプレートサイズのテンプレートと共に後段のテンプレートマッチング部９２に送出する。
【０２１３】
テンプレートマッチング部９２は、ウィンドウ切出部９１から得られた先頭のウィンドウ画像について、第１のテンプレートサイズを有する平均的な顔画像をテンプレートとして大まかなマッチングをとり、当該テンプレートとの相関値（テンプレートのうちマッチングする画素数の割合）の集合をマッチング結果として当該ウィンドウ画像と共にＤＭＡコントローラ７０を介して内部メモリ３８１Ａに格納する。
【０２１４】
即ち、図１３（ａ）に示すように、任意のスケール画像から切り出した、例えば高さ（ｙ軸方向の辺の長さ）hei_s×幅（ｘ軸方向の辺の長さ）weid_sのウィンドウ画像（スケール変換後の入力画像）Ｗ２について、図１３（ｂ）に示すように、例えば高さhei_t×幅wid_sである第１のテンプレートサイズを有する平均的な顔画像であるテンプレートＴ２を使用し、ウィンドウ画像Ｗ２をスキャンし、所定画素（例えば１画素）ずつずらしながら移動させたテンプレートＴ２と上記入力画像との相関値の集合であるマッチング結果を求める。このマッチング結果は、テンプレートＴ２の移動に伴い相関値が２次元に配列されたものであり、図１４に示すように、当該相関値を表す高さhei_r×幅wid_ｒのテンプレートマッチング結果画像Ｒ２が得られる。ここで、テンプレートレートマッチング結果画像Ｒ２の高さher_ｒは、hei_s−（hei_t＋１）であり、画像Ｒ２の幅wiｄ_ｓは、wid_s−（wid_t＋１）となる。
【０２１５】
次に、このテンプレートレートマッチング結果画像Ｒ２を所定のサイズ、例えば第１のテンプレートサイズと同一の大きさに分割し、各第１のテンプレートサイズに仕切られた分割領域毎に相関値の最大値を有する点（位置）を求め、これら各分割領域から得られた最大値を示す点のうち、所定の閾値以上のものを顔候補として抽出する。
【０２１６】
即ち、平均顔のテンプレートを使用して正規化相関をしようした場合、必ずしも任意のパターンより、顔画像の方が相関値が高くなるという保証はないものの、テンプレートと同一のサイズの顔画像が存在する場合は、テンプレートサイズ近傍の大きさでは相関値が最大値をとることから、相関値が分割領域内で最大値となり、且つ所定の閾値以上の点を顔候補として抽出することにより、単にテンプレートマッチングの結果、相関値が所定の閾値以上であるものを顔候補として抽出する場合に比して、顔候補をより有効に絞り込むことができる。
【０２１７】
次に、この顔候補として抽出された点を顔候補（第１の顔候補）とすると共に、この顔候補近傍の点も顔候補（第２の顔候補）として抽出する。後段のパターン認識部９５において、上述したＳＶＭを使用して顔判定する場合、ＳＶＭは１画素ずれるのみで顔検出ができなくなるほど敏感であるため、本実施の形態においては、顔候補の近傍も検索範囲（第２の顔候補）とすることにより、顔候補の近傍を重点的に検索して顔検出性能を向上することができる。ここで第１の顔候補として抽出された点の上下左右に隣接する８点全てを検索範囲とすると、後段の計算量が増加するため、所定の条件を満たす場合にのみ、検索範囲に指定することにより、後段の計算量の増加を抑えつつ検出性能を向上させることができる。
【０２１８】
即ち、第１の顔候補として抽出された点に隣接する点において、これらの点に対応するテンプレートサイズの入力画像領域において、肌色領域の占有率（肌色占有率）が所定の閾値以上である場合、又は、予め検出された（又は予め学習された）顔色情報を有し、この顔色領域の占有率（顔色占有率）が所定の閾値以上である場合にのみ、その点を検索範囲に設定することができる。ここで、肌色占有率は、例えば、肌色カラーテーブルを使用し、この肌色と比較することにより求めることができる。
【０２１９】
この後、テンプレートマッチング部９２は、ウィンドウ切出部９１に対して、第１のスケール画像のうち先頭のウィンドウ画像のテンプレートマッチングが終了した旨を通知することにより、当該ウィンドウ切出部９１から第１のスケール画像のうち次にスキャンされたウィンドウ画像をテンプレートマッチング部９２に送出させる。
【０２２０】
このようにテンプレートマッチング部９２は、ウィンドウ切出部９１において第１のスケール画像から切り出した全てのウィンドウ画像について、スキャン順にそれぞれ顔候補を検出する。
【０２２１】
以上が演算処理部７１における処理内容であり、ＤＭＡコントローラ７０の制御に応じて、主制御部３８１に切り換えられ、以下は当該主制御部３８１における処理内容となる。
【０２２２】
スケール変換及び切出部９３は、各スケール画像毎に候補リストとして内部メモリ３８１Ａに格納された複数のスコア画像について、当該内部メモリ３８１ＡからＤＭＡコントローラ７０を介して第１のスケール画像内でかつ候補リストの決定順に１つずつスコア画像を読み出して、前処理部９４に送出する。
【０２２３】
前処理部９４は、スケール変換及び切出部９３から得られたスコア画像について、矩形領域でなる当該スコア画像（４００画素）から人間の顔画像とは無関係である背景部分に相当する４隅の領域を除去するために、当該４隅の領域を切り取ったマスクを用いて３６０画素分を抽出する。
【０２２４】
そして前処理部９４は、撮像時の照明により濃淡で表される被写体の傾き条件を解消すべく、当該抽出した３６０画素分のスコア画像のうち顔画像として最適な部位を基準とする平面を形成するように画素単位で濃淡値に補正をかける。
【０２２５】
続いて前処理部９４は、当該３６０画素分のスコア画像のコントラストを強調した結果をヒストグラム平滑化処理を行うことにより、ＣＣＤカメラ５０のゲインや照明の強弱によらずに検出できるようにする。
【０２２６】
次いで、前処理部９４は、上述したガボア・フィルタリング処理を行うことにより、当該３６０画素分のスコア画像をベクトル変換し、得られたベクトル群を更に１本のパターンベクトルに変換する。
【０２２７】
パターン識別部（識別手段）９５は、上述したサポートベクタマシンを用いて、前処理部９４からパターンベクトルとして得られた３６０画素分のスコア画像に試して顔か否かの識別を学習しながら行う。
【０２２８】
このようにパターン識別部９５は、前処理部９４から与えられたスコア画像に基づくパターンベクトルについて、当該スコア画像内での顔データに相当する画像領域の位置（座標）及びその大きさ（画素数）と、当該スコア画像の切出し元となるスケール画像のフレーム画像に対する縮小率（すなわち上述の５段階のうちの該当する段階）と、テンプレートのサイズとをリスト化し、これをリストデータとして内部メモリ３８１Ａに格納する。
【０２２９】
この後、パターン識別部９５は、スケール変換及び切出部９３に対して、第１のスケール画像のうち最初のスコア画像の顔検出が終了した旨を通知することにより、当該スケール変換及び切出部９３から第１のスケール画像のうち次のスコア画像を前処理部９４に送出する。前処理部９５は、当該スコア画像をパターンベクトルに変換してパターン識別部９５に送出する。パターン識別部９５は、パターンベクトルから得られた顔データに基づいてリストデータを生成して内部メモリ３８１Ａに格納する。
【０２３０】
このようにスケール変換及び切出部９３において、第１のスケール画像内で候補リストにある全てのスコア画像について、順番に前処理部９４及びパターン識別部９５の各処理を行うことにより、当該第１のスケール画像から撮像結果に存在する顔画像を検出することができる。
【０２３１】
即ち、テンプレートマッチング部９２は、入力画像スケール変換部９０に対して、第１のスケール画像及び第１のテンプレートサイズのテンプレートを使用した顔検出が終了した旨を通知することにより、当該入力画像スケール変換部９０から、第１のスケール画像及び第２のテンプレートサイズのテンプレートをウィンドウ切り出し部９１に送出させる。第２のテンプレートサイズを使用した場合も、上述した第１のテンプレートサイズのテンプレートを使用した場合と同様の処理を行って全てのテンプレートサイズに対応する顔候補を検出した後、テンプレートマッチング部９２は、入力画像スケール変換部９０の第１のスケール画像に対して全てのテンプレートサイズを使用した顔検出が終了した旨を通知することにより、当該入力画像スケール変換決定部９０から第２のスケール画像をウィンドウ切出部９１に送出させる。
【０２３２】
そして第２のスケール画像についても、上述した第１のスケール画像と同様の処理を行って、当該第２のスケール画像から全てのウィンドウ画像に対応する顔候補を検出した後、第３〜第５のスケール画像についても同様の処理を順次行う。
【０２３３】
かくしてテンプレートマッチング部９２は、撮像画像であるフレーム画像を５段階に縮小した第１〜第５のスケール画像及び複数のテンプレートサイズのテンプレートについて、それぞれ切出した複数のウィンドウ画像に対応する顔候補を抽出して、当該ウィンドウ画像と共にＤＭＡコントローラ７０を介して内部メモリ３８１Ａに格納する。
【０２３４】
ここで、本実施の形態においては、任意の大きさのテンプレートを使用することができるが、使用するテンプレートサイズを切り替えて、テンプレートサイズを選択することにより、入力画像に対して準備できる全てのテンプレートサイズに対して演算をする場合に比して、演算量を減らして高効率化することができる。例えば、一度顔が検出された場合に、次に顔検出する際はそのテンプレートサイズを使用することができる。また、例えば、ロボット装置に設けられた距離センサを使用し、この距離センサからの距離情報に基づき入力画像に含まれる対象物との間の距離を認識することにより、対象物の顔領域の大きさを予測してテンプレートサイズを選択する対象距離切り替え手段を設ける等することができ、目的に応じてテンプレートサイズを切り替えることができる。
【０２３５】
（３−２）実施の形態における動作
以上の構成において、このロボット装置２００では、ＣＣＤカメラ５０により撮像したフレーム画像を縮小率が相異なる複数のスケール画像に変換した後、当該各スケール画像の中からそれぞれ所定サイズのウィンドウ画像を所定画素ずつずらすようにスキャンしながら１枚ずつ切り出す。
【０２３６】
このウィンドウ画像について、平均的な顔画像を表すテンプレートを用いてマッチングをとって当該テンプレートとの相関値の集合であるマッチング結果画像を生成する。このように各スケール画像毎にそれぞれ全てのウィンドウ画像についてスキャン順にそれぞれマッチング結果画像を生成する。以下、マッチング結果画像から顔候補を検出する工程について詳細に説明する。
【０２３７】
図１５は、テンプレートマッチング部９２において、テンプレートマッチング結果画像Ｒ２から顔候補となる画素を検出する各処理工程を示すフローチャートである。図１５に示すように、先ず、テンプレートマッチング結果画像Ｒ２が入力されると、マッチング結果画像Ｒ２をテンプレートサイズに分割し、その分割領域の１つ、例えば０≦ｘ≦wid_t−１、０≦ｙ≦hei_t−１において、最も相関値が高い点（座標）を抽出する（ステップＳＰ１１）。以下、マッチング結果画像Ｒ２をテンプレートサイズに分割した領域を分割領域ｒｎ、分割領域ｒｎにおいて、相関値が最も大きい点（座標）をlocal_max（ｘ，ｙ）という。ここでは、この各分割領域内において最も相関値が高い画素を抽出するが、本実施の形態においては、マッチング結果画像において分割された分割領域を左から右へ一行ずつ順に処理を行う場合について説明する。
【０２３８】
次に、local_max（ｘ，ｙ）が所定の閾値（ｔｈ１）より大きいか否かを判定し（ステップＳＰ１２）、大きい場合は、顔候補として追加する（ステップＳＰ１３）。上述した如く、入力画像スケール変換決定部９０においては、スケールと共に入力画像に含まれると想定される顔の大きさのテンプレートサイズを選択するが、テンプレートサイズは異なる大きさの複数種類あり、複数種類ある各テンプレートサイズ毎にマッチング結果画像Ｒ２を算出して顔候補を抽出すると、同一の点が抽出される場合がある。従って、ステップＳＰ１３において、顔候補として同一の点がある場合、即ち、異なるテンプレートサイズで顔候補を抽出した際に既に抽出されている場合はこの点は追加しない。
【０２３９】
次に、顔候補として抽出された点に対応するテンプレートサイズの入力画像領域において、この画像領域内に含まれる肌色画素の占有率を求める。本実施の形態においては、肌色画素の占有率を求める際に、肌色カラーテーブル１００を参照する。そして、この肌色画素占有率が所定の閾値（ｔｈ２）より大きいか否かを判定する（ステップＳＰ１４）。大きい場合は、このlocal_max（ｘ，ｙ）の周辺、例えば上下左右の８近傍点を顔候補として追加する（ステップＳＰ１５）。ここで、ステップＳＰ１３と同様に、既にこれらの８近傍点が既に顔候補として抽出されている場合は、候補に追加しない。
【０２４０】
ステップＳＰ１２でlocal_max（ｘ，ｙ）が閾値ｔｈ１未満だった場合、ステップＳＰ１４でlocal_max（ｘ，ｙ）に相当する入力画像における肌色画素占有率が閾値ｔｈ２未満であった場合、及びステップＳＰ１５で顔候補の追加が終了した後は、いずれもステップＳＰ１６に進み、次の顔候補を抽出するために次の分割領域に移り、処理を進める。
【０２４１】
先ず、マッチング結果画像Ｒ２において、ｘ方向にテンプレートサイズ分、即ち、wid_tだけずれた隣の分割領域に移る（ステップＳＰ１６）。次に、wid_tだけずれたｘ座標（ｘ＋wid_t）の分割領域において、そのｘ座標がマッチング結果画像の幅（ｘ方向の辺）wid_rより大きい場合は、分割領域がマッチング結果画像に含まれないことを示し、次の行に移り、０≦ｘ≦wid_t−１であって、ｙ方向にテンプレートサイズ分、即ち、hei_tだけずれた隣の分割領域に移る（ステップＳＰ１８）。次に、分割領域のｙ座標がマッチング結果画像の高さ（ｙ方向の辺）hei_rより大きいか否かを判定し（ステップＳＰ１９）、大きい場合は、マッチング結果画像における全ての分割領域の相関値の最大値を求めたことを示し、処理を終了する。
【０２４２】
一方、ステップＳＰ１７及びステップＳＰ１８において、分割領域がマッチング結果画像に含まれると判定された場合は、再びステップＳＰ１１に戻り、その分割領域内で最も高い相関値を有する点を抽出する。
【０２４３】
本実施の形態においては、マッチング結果画像Ｒ２をテンプレートサイズに区切った分割領域における相関値の最大値を求めているため、ステップＳＰ１６において、隣接する分割領域に移る場合は、ｘ方向にwid_tだけずれるものとしたが、マッチング結果画像Ｒ２は、テンプレートサイズ以下のサイズであれば、任意の大きさに分割することができる。その際、分割する画像の大きさの幅（ｘ方向の辺）wid_step、高さ（ｙ方向）hei_stepとすると、ステップＳＰ１６及びステップＳＰ１８において、夫々ｘ方向にwid_step、又はｙ方向にhei_step移動することにより、次の分割領域に進むことができる。
【０２４４】
こうして、テンプレートマッチング部９２において、顔候補が抽出される。ここまでを演算処理部７１において実行すると共にこれ以降の処理を主制御部３８１において実効することにより、その分当該主制御部３８１の処理負担を軽減することができる。以降のスケール変換及び切り出し部９３、前処理部９４及びパターン認識部９５を経て顔データを抽出する処理は、上述の第２の適用例と同様である。
【０２４５】
（３−３）本実施の形態の効果
図１６は、テンプレートマッチング部９２において、ウィンドウ画像Ｗ２から顔候補として検出された点を示す図である。図１６において、白で示す点が、図１４に示すマッチング結果画像Ｒ２から顔候補として抽出された点である。比較として、図１７は、マッチング結果画像Ｒ２において、閾値以上である点を全て顔候補として抽出した例を示す図である。図１７に示す図と比較すると、本実施の形態において、テンプレートマッチング部９２にて顔候補として抽出される点が飛躍的に少なくなっているのがわかる。これにより、後段の処理における計算量を飛躍的に削減することができる。
【０２４６】
本実施の形態においては、ウィンドウ画像について、平均的な顔画像を表すテンプレートを用いてマッチングをとって大まかに顔画像であるか否かを判断する際に、テンプレートマッチング結果画像を所定のサイズに仕切り、相関値の最大値を顔候補として抽出して明らかに顔画像でないウィンドウ画像を除去することにより、本来顔である領域を見逃すことなく、後段の顔検出処理に要する演算量及び時間を減少させることができ、かくしてリアルタイム性を格段と向上した顔検出装置及びこれを搭載したロボット装置を実現することができる。
【０２４７】
また、相関値が最大となる点と共にその周囲においても顔検索範囲とすることにより、顔検出精度を向上することができる。更に、所定の閾値以上の肌色占有率又は顔の色占有率を有する場合のみ、顔検索範囲として設定することにより、顔検出精度を保ちつつ顔候補を減らして後段の演算量を減らすことができる。更にまた、テンプレートのサイズを適宜切り替えることにより、更に演算量を減らすことができる。
【０２４８】
（４）他の実施の形態
なお上述の実施の形態においては、本発明を図１のように構成された２足歩行型のロボット装置２００に適用するようにした場合について述べたが、本発明はこれに限らず、この他種々のロボット装置及びロボット装置以外のこの他種々の装置に広く適用することができる。例えば、ロボット装置は、４足歩行であってもよく、更に、移動手段は、脚式移動方式に限定されない。
【０２４９】
また、ＣＣＤカメラ（撮像手段）５０による撮像結果として得られるフレーム画像を複数の異なる縮小率でなるスケール画像に縮小変換するスケール変換手段として、図１２に示す主制御部３８１の処理機能のうちの入力画像スケール変換部９０を適用するようにした場合について述べたが、本発明はこれに限らず、要は、フレーム画像を複数の異なる縮小率で縮小変換した複数のスケール画像を得ることができれば、この他種々の構成のものを適用するようにしてもよい。
【０２５０】
更に、１枚のフレーウ画像を０．８倍ずつ縮小変換した５種類のスケール画像を適用するようにしたが、主制御部３８１の処理能力の範囲内であれば、２〜４種類でも６種類以上でも良く、また縮小率も自由に設定するようにしてもよい。この場合、複数のスケール画像を基にして顔検出処理を行えば、撮像結果であるフレーム画像に実際に写っている顔のサイズにかかわりなく（すなわちロボット装置２００と人間との距離）、当該フレーム画像から直接顔画像を検出する場合よりも極めて高い確率で顔画像を検出することができる。
【０２５１】
また、ＣＣＤカメラ（撮像手段）５０による撮像結果として得られるフレーム画像内を、４００画素のウィンドウ画像を１画素ずつずらすようにスキャンしながら順次抽出する抽出手段を、図１２に示す主制御部３８１の処理機能のうちのウィンドウ切出部９１により構成するようにした場合について述べたが、本発明はこれに限らず、要は、フレーム画像又はスケール画像から所定サイズのウィンドウ画像を所定画素ずつずらすようにスキャンしながら順次抽出する得ることができれば、この他種々の構成のものを適用するようにしてもよい。
【０２５２】
この場合、スキャンの順番は矩形画像の左上位置から右下位置まで以外にも当該矩形画像全体を走査することができればどのように設定してもよい。またスキャン時にずらす画素数は１画素でも３画素以上でも良く、ウィンドウ画像のサイズも４００画素以外に種々の縦横比の所望の画素数に設定するようにしてもよい。
【０２５３】
更に、ウィンドウ画像から当該顔画像を識別する識別手段として、図１２に示す主制御部３８１の処理機能のうちの前処理部９５及びパターン識別部９６を適用するようにした場合について述べたが、本発明はこれに限らず、顔画像を検出することができればこの他種々の方法を用いた構成のものに広く適用することができる。
【０２５４】
更にまた、抽出されたウィンドウ画像を、平均的な顔画像を表すテンプレートを用いてマッチングをとり、当該テンプレートとの相関度を検出する相関度検出手段として、図１２に示す主制御部３８１の処理機能のうちのテンプレートマッチング部９２を適用するようにした場合について述べたが、本発明はこれに限らず、要は、当該テンプレートを基準として相関度を検出することができれば、この他種々の方法を用いた構成のものに広く適用することができる。
【０２５５】
更に、ロボット装置２００の制御全体を司る制御手段として主制御部３８１を適用すると共に、当該主制御部３８１と接続され、その制御に応じて情報を読み書きするメモリ手段として内部メモリ３８１Ａを適用し、更に図１２に示す主制御部３８１の処理機能のうち、入力画像スケール変換部９０、ウィンドウ切出部９１、テンプレートマッチング部９２による各処理を実行する演算処理手段として演算処理部７１（ＤＭＡコントローラ７０）を適用するようにした場合について述べたが、本発明はこれに限らず、主制御部３８１と演算処理部７１とで顔検出処理の処理機能（９０〜９５）を選択的に分担して内部メモリ３８１Ａを介して切り換えて実行することができれば、その際の分担内容は、主制御部３８１にかかる演算量及び演算時間の負担を低減させるように自由に設定することができる。また、主制御部３８１のみで処理を行ってもよい。
【０２５６】
【発明の効果】
以上詳細に説明したように本発明に係る顔検出装置は、入力画像から対象物の顔領域を抽出する顔検出装置において、撮像手段による撮像結果として得られるフレーム画像を入力画像とし、この入力画像と平均的な顔画像を示す所定サイズのテンプレートとの相関をとった相関値の集合であるマッチング結果を生成するマッチング結果生成手段と、上記マッチング結果から相関値の局所最大値を求めこの局所最大値に基づき顔候補を抽出する顔候補抽出手段と、上記顔候補として抽出された点に相当する入力画像領域から顔画像を識別する識別手段とを有するので、顔候補抽出手段により極めて効率よく且つ正確に顔候補を絞り込むことができるため、後段の顔識別手段における演算量を低減して、リアルタイムの顔検出を可能とする。
【０２５７】
本発明に係るロボット装置は、供給された入力情報に基づいて動作を行う自律型のロボット装置において、撮像手段と、上記撮像手段による撮像結果として得られるフレーム画像を入力画像とし該入力画像から人物の顔領域を抽出する顔検出手段を有し、上記顔検出手段は、上記入力画像と平均的な顔画像を示す所定サイズのテンプレートとの相関をとった相関値の集合であるマッチング結果を生成するマッチング結果生成手段と、上記マッチング結果から相関値の局所最大値を求めこの局所最大値に基づき顔候補を抽出する顔候補抽出手段と、上記顔候補として抽出された点に相当する入力画像領域から顔画像を識別する識別手段とを具備するので、顔候補抽出手段により極めて効率よく且つ正確に顔候補を絞り込み、顔検出の演算量を低減して、リアルタイムの顔検出を可能とすることにより、ロボット装置のエンタテーメント性を向上することができる。
【図面の簡単な説明】
【図１】本発明の実施の形態における人間型ロボット装置を前方からみた外観を示す斜視図である。
【図２】本発明の実施の形態における人間型ロボット装置を後方からみた外観を示す斜視図である。
【図３】本発明の実施の形態における人間型ロボット装置の自由度構成モデルを示す模式図である。
【図４】本発明の実施の形態における人間型ロボット装置の制御システム構成を示す模式図である。
【図５】第１の適用例におけるロボット装置の内部構成の説明に供するブロック図である。
【図６】第１の適用例におけるロボット装置の顔検出手段を示すブロック図である。
【図７】重なり判定処理の説明に供する略線的な平面図である。
【図８】第２の適用例におけるロボット装置の内部構成の説明に供するブロック図である。
【図９】第２の適用例におけるロボット装置の顔検出手段を示すブロック図である。
【図１０】テンプレートマッチングによる相関性の検出の説明に供する模式図である。
【図１１】重なり判定処理手順をその工程順に示すフローチャートである。
【図１２】本発明の実施の形態におけるロボット装置の顔検出タスク機能を示すブロック図である。
【図１３】（ａ）及び（ｂ）は、夫々入力画像（ウィンドウ画像）及びテンプレートを示す模式図である。
【図１４】入力画像（ウィンドウ画像）とテンプレートとから求めた相関値の集合であるマッチング結果画像を示す図である。
【図１５】本発明の実施の形態における顔検出手段のテンプレートマッチング部における処理工程を示すフローチャートである。
【図１６】本発明の実施の形態における顔検出タスク機能のテンプレートマッチング部がマッチング結果画像から顔候補を抽出した結果を示す図である。
【図１７】マッチング結果画像において、所定の閾値以上のものを顔候補として抽出した結果を示す図である。
【符号の説明】
１ ロボット装置、 ３８１ メイン制御部、 ３８１Ａ 内部メモリ、 ５０ ＣＣＤカメラ、 ６０、８０、９０ 入力画像スケール変換部、 ６１、８１、９１ ウィンドウ切出部、 ６２、８２、９２ テンプレートマッチング部、 ６３、８５、９４ 前処理部、 ６４、８６、９５ パターン識別部、 ６５、８３ 重なり判定部、 ７０ ＤＭＡコントローラ、 ７１ 演算処理部、８４、９３ スケール変換及び切出部、 ＲＴ１ 重なり判定処理手順、 Ｓ１Ａ 画像信号[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to a face detection apparatus and method for detecting a face of an object from an input image.,The present invention also relates to a program for causing a computer to perform an operation for detecting a face, and a recording medium on which the program is recorded.
[0002]
[Prior art]
A mechanical device that performs an action similar to that of a human (living body) using an electrical or magnetic action is called a “robot”. Robots started to spread in Japan from the end of the 1960s, but many of them are industrial robots such as manipulators and transfer robots for the purpose of automating and unmanned production work in factories. Met.
[0003]
Recently, practical robots that support life as a human partner, that is, support human activities in various situations in daily life such as the living environment, have been developed. Unlike industrial robots, such practical robots have the ability to learn how to adapt themselves to humans with different personalities or to various environments in various aspects of the human living environment. For example, a “pet-type” robot that mimics the body mechanism and movement of a four-legged animal such as a dog or cat, or a body mechanism and movement of a human that walks upright on two legs. Robot devices such as “humanoid” or “humanoid robots” are already in practical use.
[0004]
Since these robot devices can perform various operations with an emphasis on entertainment, for example, compared to industrial robots, they may be referred to as entertainment robots. In addition, such a robot apparatus is equipped with various external sensors such as a CCD (Charge Coupled Device) camera and a microphone, and recognizes an external situation based on the output of these external sensors, and provides information from outside and internal information. Some work autonomously depending on the state.
[0005]
[Problems to be solved by the invention]
By the way, in such an entertainment-type robot apparatus, a human face that is the opponent during conversation or a human face that enters the field of view while moving is detected, and the conversation or action is performed while watching the human face. If it is possible, it is most desirable in view of its naturalness as in the case where human beings usually perform, and it is considered that the entertainment property as an entertainment robot apparatus can be further improved.
[0006]
2. Description of the Related Art Conventionally, many methods have been proposed for detecting a human face using only a light and shade pattern based on an image signal from a complex image scene such as a moving image without using color or movement.
[0007]
As these face detection methods, there is a method of generating a discriminator by learning a face pattern in advance using a pattern recognition technique such as an eigenface, a neural network, and a support vector machine (SVM). Can be mentioned.
[0008]
However, according to the method for generating the pattern discriminator, the robot apparatus performs pattern recognition by learning on face image data, which has a huge amount of data, in response to changes in the environment and changes in its posture and expression. However, there is a problem that the time required for the arithmetic processing becomes enormous because the amount of calculation required for the pattern identification increases accordingly.
[0009]
In fact, in the process of detecting a human face image from the captured image (hereinafter referred to as a face detection task), the entire captured image is identified in order to identify the face image from the captured image. Scan at various scales. For this reason, it is extremely important to reduce the arithmetic processing required for pattern identification once at a time.
[0010]
For example, in the case of a face detection task using pattern recognition by a support vector machine, hundreds of types of support are provided for a 400-dimensional vector obtained from a clipped image of about 400 (= 20 × 20) pixels from a captured image. An inner product operation with a vector (400 dimensions) is required. If this is performed in the entire screen of size (W, H), the inner product operation must be repeated (W−20 + 1) × (H−20 + 1) times, resulting in a huge amount of calculation processing. .
[0011]
Further, when a face detection task is used for a robot apparatus, it is extremely difficult to feed back as a robot action that requires real-time characteristics unless a face image is detected sufficiently quickly from a moving image. In addition, since the CPU inside the robot apparatus has many tasks that are always executed in addition to the face detection task, the CPU capacity is fully consumed for the face detection task because it spends computing power on these tasks. It is extremely difficult to spend computing power.
[0012]
  The present invention has been proposed in view of such a conventional situation, and a face detection apparatus and method that can significantly improve real-time performance by reducing the amount of calculation.,An object of the present invention is to provide a program and a recording medium.
[0013]
[Means for Solving the Problems]
  In order to achieve the above-described object, a face detection apparatus according to the present invention uses a frame image obtained as an imaging result by an imaging unit as an input image in a face detection apparatus that extracts a face area of an object from an input image. Matching result generation means for generating a matching result that is a set of correlation values obtained by correlating the input image with a template of a predetermined size representing an average face image, and obtaining a local maximum value of the correlation value from the matching result Face candidate extraction means for extracting face candidates based on the local maximum value and identification means for identifying the face area from the input image area corresponding to the face candidate in the matching result.The matching result, which is a set of correlation values, is a set of correlation values between the template and the input image that are moved by scanning the input image while shifting by predetermined pixels. The face candidate extraction unit divides the matching result into a plurality of regions, and sets a point having at least a maximum correlation value for each of the divided regions as a first face candidate; The points near the first face candidate are extracted as second face candidates, and the first and second face candidates are extracted as face candidates.
[0014]
In the present invention, using the fact that the correlation value of the template matching becomes the largest when a face having the same size as the average face template exists in the input image, the face candidate is based on the local maximum value of the correlation value of the matching result. Therefore, the number of face candidates can be drastically reduced while maintaining face detection performance, compared to the case where face candidates are detected simply by setting a threshold value from the matching result. Therefore, it is possible to detect the face area from the input image at a very high speed.
[0015]
Further, the face candidate extracting means can divide the matching result into a plurality of areas having a size equal to or smaller than the template size, for example, and extract at least the maximum correlation value as a face candidate for each divided area.
[0016]
Furthermore, it has template size determination means for determining the size of the template to be input to the matching result generation means from templates of different sizes, and the template size determination means selects a template having the same size as the face area detected in advance. Alternatively, distance information with respect to the object in the input image is input, and a template size can be efficiently determined by selecting a template based on the distance information.
[0017]
Furthermore, the matching result generating means can generate a matching result corresponding to each template by sequentially inputting templates of different sizes, thereby detecting a face of any size included in the input image. Can do.
[0018]
In addition, the face candidate extraction unit can further narrow down the face candidates by extracting only the predetermined threshold value or more from the maximum correlation value of each of the divided areas as a face candidate.
[0019]
Further, the matching result, which is a set of correlation values, is a set of correlation values between the template and the input image that are moved by scanning the input image while shifting by predetermined pixels. It can be an arrayed two-dimensional array.
[0020]
Furthermore, the face candidate extraction means uses the point having the maximum correlation value of each of the divided regions as the first face candidate, and points near the first face candidate as the second face candidate. The face detection performance can be further improved by extracting the second face candidate as the face candidate.
[0021]
Further, the face candidate extracting means has a predetermined threshold value that occupies the skin color area in the input image area of the template size corresponding to the point near the first face candidate among the points near the first face candidate. The face color in the input image region having the template size corresponding to the point near the first face candidate among the points near the first face candidate and having the face color information learned in advance. A point whose area occupancy is equal to or greater than a predetermined threshold can be extracted as a second face candidate, thereby narrowing face candidates while improving face detection performance.
[0022]
  A face detection method according to the present invention is a face detection method for extracting a face area of an object from an input image.ComputerMatching result generation that generates a matching result that is a set of correlation values obtained by taking a correlation between the input image and a template of a predetermined size indicating an average face image as a frame image obtained as a result of imaging by the imaging means A face candidate extracting step of obtaining a local maximum value of a correlation value from the matching result and extracting a face candidate based on the local maximum value; and extracting the face area from an input image area corresponding to the face candidate in the matching result. An identification process to identify andThe matching result, which is a set of correlation values, is a set of correlation values between the template and the input image that are moved while shifting the predetermined image by scanning the input image, and the movement of the template In the face candidate extraction step, the matching result is divided into a plurality of regions, and a point having at least a maximum correlation value is determined for each divided region as the first face candidate. Then, the first and second face candidates are extracted as face candidates by using a point in the vicinity of the first face candidate as a second face candidate.
[0026]
  A program according to the present invention and a recording medium on which the program is recorded are a program for causing a computer to perform an operation of extracting a human face area from an input image, and a recording medium on which the program is recorded.On the computer,Matching result generation that generates a matching result that is a set of correlation values obtained by taking a correlation between the input image and a template of a predetermined size indicating an average face image as a frame image obtained as a result of imaging by the imaging meansprocedureAnd a face candidate extraction that obtains a local maximum value of the correlation value from the matching result and extracts a face candidate based on the local maximum valueprocedureAnd identifying the face area from the input image area corresponding to the point extracted as the face candidateprocedureWhenThe matching result, which is a set of correlation values, is a set of correlation values between the input image scanned with the input image and shifted by predetermined pixels and the input image. In the face candidate extraction procedure, the matching result is divided into a plurality of regions, and each of the divided regions has at least a maximum correlation value. Is a first face candidate, a point in the vicinity of the first face candidate is used as a second face candidate, and the first and second face candidates are extracted as face candidates, and a recording medium on which the program is recorded.
[0027]
DETAILED DESCRIPTION OF THE INVENTION
(1) Summary of the present invention
The face detection apparatus and the face detection method according to the present invention detect a human face (face area) from an input image (captured image). This face detection task is roughly divided into two steps. That is, it comprises a first step of extracting face candidates by performing template matching on the input image, and a second step of determining whether or not the face is a face by SVM or the like.
[0028]
In order to perform face detection in real time in a system with limited resources such as a CPU and memory such as a robot apparatus, it is necessary to reduce the amount of calculation in the face detection task. However, as described above, the face determination using SVM or the like performed in the second step requires a huge amount of calculation. Therefore, in the face detection task, it is effective to narrow down face candidates in the first step in order to reduce the amount of calculation required for face detection while maintaining face detection performance.
[0029]
Here, in the first step, when extracting a face candidate by performing template matching, for example, there is a method of extracting a face candidate by providing a threshold value for a correlation value of matching. In this method, when trying to reduce oversight of face candidates, a method of increasing the correlation value threshold or a method of reducing the thinning at the time of matching can be taken. Since candidates are extracted, the amount of calculation increases, which may not be preferable in an environment with limited resources such as a robot apparatus. On the other hand, when the threshold value is increased, the number of face candidates can be reduced, but an image that is originally a face is also removed from the candidates, and the face image may be missed.
[0030]
Therefore, the present invention proposes a method capable of preventing an increase in the amount of calculation without overlooking an image that is originally a face from face candidates, that is, effectively reducing the number of face candidates. Specifically, in the first step, a matching result that is a set of correlation values obtained by correlating the input image with a template indicating an average face image of a predetermined size is generated, and the correlation value in the matching result is generated. A face region is extracted based on the local maximum value of. For the face area, the matching result is divided into a plurality of areas having the same size as the template, for example, and at least the maximum correlation value is extracted as a face candidate for each divided area. In the second step, a face area (face image) is identified (determined) by SVM or the like from the input image area corresponding to the point extracted as the face candidate.
[0031]
(2) Robot configuration
Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In this embodiment, the present invention is applied to a biped walking robot device equipped with a face detection device. First, a biped humanoid robot apparatus will be described as an example of a robot apparatus having face detection means.
[0032]
1 and 2 show a state in which the humanoid robot device 200 is viewed from the front and the rear. Further, FIG. 3 schematically shows the joint degree-of-freedom configuration of the humanoid robot apparatus 200.
[0033]
As shown in FIGS. 1 and 2, the humanoid robot device 200 includes leg units 201R and 201L for left and right legs that perform legged movement, a torso unit 202, and left and right arm units 203R and 203L. And a head unit 204.
[0034]
Each of the left and right leg units 201R and 201L includes thighs 205R and 205L, knee joints 206R and 206L, shin parts 207R and 207L, ankles 208R and 208L, and feet 209R and 209L. 210R and 210L are connected at the substantially lowermost end of the body unit 202. The left and right arm units 203R and 203L are composed of upper arms 211R and 211L, elbow joints 212R and 212L, and forearms 213R and 213L, and left and right sides above the body unit 202 by the shoulder joints 214R and 214L. It is connected at the edge. The head unit 204 is connected to the substantially uppermost center of the body unit 202 by a neck joint 255.
[0035]
As shown in FIG. 3, the neck indirect between the head unit 204 and the trunk unit 202 includes a neck joint yaw shaft 302 that supports the head unit 204, a neck joint pitch shaft 303, and a neck joint roll shaft 304. 3 degrees of freedom.
[0036]
Further, the arm joint includes a shoulder joint pitch axis 308, a shoulder joint roll axis 309, an upper arm yaw axis 310, an elbow joint pitch axis 311, a forearm yaw axis 312, a wrist joint pitch axis 313, and a wrist joint roll. A ring 314 and a hand portion 315 are included. The hand portion 315 is actually a multi-joint / multi-degree-of-freedom structure including a plurality of fingers. However, since the movement of the hand portion 315 has little contribution or influence on the posture control or walking control of the humanoid robot apparatus 200, it is assumed in this specification that the degree of freedom is zero. Therefore, it is assumed that each arm portion has seven degrees of freedom.
[0037]
The body unit has three degrees of freedom: a body pitch axis 305, a body roll axis 306, and a body yaw axis 307.
[0038]
Each leg unit constituting the lower limb includes a hip joint yaw axis 316, a hip joint pitch axis 317, a hip joint roll axis 318, a knee joint pitch axis 319, an ankle joint pitch axis 320, and an ankle joint roll axis 321. And a foot portion 322. In the present specification, the intersection of the hip joint pitch axis 317 and the hip joint roll axis 318 defines the hip joint position of the humanoid robot apparatus 200. The foot 322 of the human body is actually a structure including a multi-joint / multi-degree-of-freedom sole, but the sole of the humanoid robot apparatus 200 has zero degrees of freedom. Accordingly, each leg is configured with 6 degrees of freedom.
[0039]
In summary, the humanoid robot apparatus 200 as a whole has a total of 3 + 7 × 2 + 3 + 6 × 2 = 32 degrees of freedom. However, the humanoid robot device 200 for entertainment is not necessarily limited to 32 degrees of freedom. Needless to say, the degree of freedom, that is, the number of joints, can be increased or decreased as appropriate in accordance with design / production constraints or required specifications.
[0040]
Each degree of freedom of the humanoid robot device 200 as described above is actually implemented using an actuator. The actuator must be small and light because of the need to eliminate the extra bulge on the exterior and approximate it to the shape of a human body, or to perform posture control on an unstable structure such as biped walking. preferable.
[0041]
FIG. 4 schematically shows a control system configuration of the humanoid robot apparatus 200. As shown in the figure, the humanoid robot apparatus 200 is configured by each of the mechanism units 330, 340, 350R / L, 360R / L representing human limbs, and adaptive control for realizing a cooperative operation between the mechanism units. (Where R and L are suffixes indicating the right and left, respectively, and so on).
[0042]
The overall operation of the humanoid robot apparatus 200 is controlled by the control unit 380 in an integrated manner. The control unit 380 is configured to store data and commands between a main control unit 381 including main circuit components (not shown) such as a CPU (Central Processing Unit) and a memory, and each component of the power supply circuit and the humanoid robot apparatus 200. It comprises a peripheral circuit 382 including an interface (not shown) for sending and receiving. The installation location of the control unit 380 is not particularly limited. Although it is mounted on the body unit 340 in FIG. 4, it may be mounted on the head unit 330. Alternatively, a control unit 380 may be provided outside the humanoid robot device 200 so as to communicate with the body of the humanoid robot device 200 by wire or wirelessly.
[0043]
Each degree of freedom of joint in the humanoid robot apparatus 200 shown in FIG. 4 is realized by a corresponding actuator. That is, the head unit 330 includes a neck joint yaw axis actuator A representing the neck joint yaw axis 302, the neck joint pitch 303, and the neck joint roll axis 304.₂, Neck joint pitch axis actuator A₃, Neck joint roll axis actuator A₄Is arranged.
[0044]
The head unit 330 is provided with a CCD (Charge Coupled Device) camera for imaging an external situation, a distance sensor for measuring the distance to an object located in front, and an external sound. A microphone for collecting sound, a speaker for outputting sound, a touch sensor for detecting pressure received by a physical action such as “stroking” or “striking” from a user, and the like are provided.
[0045]
The body unit 340 includes a body pitch axis actuator A that represents each of the body pitch axis 305, the body roll axis 306, and the body yaw axis 307.₅, Fuselage roll axis actuator A₆, Fuselage yaw axis actuator A₇Is arranged. In addition, the body unit 340 includes a battery serving as a starting power source for the humanoid robot apparatus 200. This battery is constituted by a chargeable / dischargeable battery.
[0046]
Further, the arm unit 350R / L is subdivided into an upper arm unit 351R / L, an elbow joint unit 352R / L, and a forearm unit 353R / L, but a shoulder joint pitch axis 308, a shoulder joint roll axis 309, an upper arm Shoulder joint pitch axis actuator A representing a yaw axis 310, an elbow joint pitch axis 311, a forearm yaw axis 312, a wrist joint pitch axis 313, and a wrist joint roll axis 314₈, Shoulder joint roll axis actuator A₉, Upper arm yaw axis actuator A₁₀, Elbow joint pitch axis actuator A₁₁, Elbow joint roll axis actuator A₁₂, Wrist joint pitch axis actuator A₁₃Wrist joint roll axis actuator A₁₄Is deployed.
[0047]
The leg unit 360R / L is subdivided into a thigh unit 361R / L, a knee unit 362R / L, and a shin unit 363R / L, but the hip joint yaw axis 316, hip joint pitch axis 317, hip joint Hip joint yaw axis actuator A representing roll axis 318, knee joint pitch axis 319, ankle joint pitch axis 320, and ankle joint roll axis 321₁₆Hip joint pitch axis actuator A₁₇, Hip joint roll axis actuator A₁₈, Knee joint pitch axis actuator A₁₉, Ankle joint pitch axis actuator A₂₀, Ankle joint roll axis actuator A₂₁Is deployed. Actuator A used for each joint₂, A₃.. Can be constituted by a small AC servo actuator of the type that is directly connected to the gear and that is mounted on the motor unit with the servo control system being integrated into a single chip.
[0048]
For each mechanism unit such as the head unit 330, the body unit 340, the arm unit 350, and each leg unit 360, sub-control units 335, 345, 355R / L, and 365R / L of the actuator drive control unit are provided. ing. Further, ground check sensors 391 and 392 for detecting whether or not the soles of the legs 360R and L have landed are mounted, and a posture sensor 393 for measuring the posture is provided in the body unit 340. ing.
[0049]
The ground contact confirmation sensors 391 and 392 are configured by, for example, proximity sensors or micro switches installed on the soles of the feet. The attitude sensor 393 is configured by a combination of an acceleration sensor and a gyro sensor, for example.
[0050]
Based on the outputs of the ground contact confirmation sensors 391 and 392, it is possible to determine whether the left and right legs are currently standing or swinging during an operation period such as walking or running. Further, the tilt and posture of the body part can be detected by the output of the posture sensor 393.
[0051]
The main control unit 381 can dynamically correct the control target in response to the outputs of the sensors 391 to 393. More specifically, adaptive control is performed for each of the sub-control units 335, 345, 355R / L, and 365R / L, and the arm, body, and legs of the humanoid robot device 200 are coordinated. A driving whole body movement pattern can be realized.
[0052]
The whole body movement on the human body of the humanoid robot device 200 sets the foot movement, the ZMP (Zero Moment Point) trajectory, the trunk movement, the upper limb movement, the waist height, and the like, and the movement according to these setting contents. The instructed command is transferred to each of the sub-control units 335, 345, 355R / L, 365R / L. Each of the sub-control units 335, 345,... Interprets the received command from the main control unit 381, and each actuator A₂, A₃... and a drive control signal is output. Here, “ZMP” is a point on the floor where the moment due to floor reaction force during walking is zero, and “ZMP trajectory” is, for example, the walking operation period of the humanoid robot device 200. It means the trajectory that ZMP moves inside.
[0053]
At the time of walking, gravity, inertia force, and these moments act on the road surface from the walking system due to gravity and acceleration caused by walking motion. According to the so-called “Dalambert principle”, they balance with the floor reaction force and the floor reaction force moment as a reaction from the road surface to the walking system. As a result of the dynamic reasoning, there is a point where the pitch and roll axis moment become zero, that is, “ZMP (Zero Moment Point)” on or inside the side of the support polygon formed by the sole contact point and the road surface.
[0054]
Many of the proposals related to posture stability control of legged mobile robots and prevention of falls during walking use this ZMP as a norm for determining the stability of walking. The biped walking pattern generation based on the ZMP norm has advantages such as being able to set the sole landing point in advance and easily considering the kinematic constraint conditions of the foot according to the road surface shape. In addition, using ZMP as a stability determination criterion means that a trajectory is treated as a target value in motion control instead of force, and thus technically feasible. Regarding the concept of ZMP and its application to the stability criteria for walking robots, Miomir Vukobratovic “LEGGED LOCOMOTION ROBOTS” (Ichiro Kato's “Walking Robots and Artificial Feet” (Nikkan Kogyo Shimbun)) It is described in.
[0055]
In general, a biped walking robot such as a humanoid has a higher center of gravity and a smaller ZMP stability area during walking than a quadruped walking. Therefore, the problem of posture change accompanying such a change in road surface condition is particularly important in a biped robot.
[0056]
As described above, in the humanoid robot apparatus 200, each of the sub-control units 335, 345,... Interprets the received command from the main control unit 381, and each actuator A₂, A₃... and a drive control signal is output to control the drive of each unit. As a result, the humanoid robot device 200 stably transitions to the target posture and can walk in a stable posture.
[0057]
Further, in the control unit 380 in the humanoid robot apparatus 200, in addition to the attitude control as described above, various sensors such as an acceleration sensor, a touch sensor, and a grounding confirmation sensor, image information from a CCD camera, and audio information from a microphone. And so on. In the control unit 380, although not shown, an acceleration sensor, a gyro sensor, a touch sensor, a distance sensor, various sensors such as a microphone and a speaker, each actuator, a CCD camera, and a battery are connected to the main control unit 381 via corresponding hubs. Has been.
[0058]
The main control unit 381 sequentially takes in sensor data, image data, and audio data supplied from the above-described sensors, and sequentially stores them in a predetermined position in the DRAM via the internal interface. The main control unit 381 sequentially takes in battery remaining amount data representing the remaining amount of battery supplied from the battery and stores it in a predetermined position in the DRAM. Each sensor data, image data, audio data, and battery remaining amount data stored in the DRAM is used when the main control unit 381 controls the operation of the humanoid robot device 200.
[0059]
The main control unit 381 reads out the control program and stores it in the DRAM when the power of the humanoid robot device 200 is initially turned on. In addition, the main control unit 381 determines from the user and the surrounding situation based on each sensor data, image data, audio data, and battery remaining amount data that are sequentially stored in the DRAM from the main control unit 381 as described above. Judgment of whether or not there is an instruction and action. Further, the main control unit 381 determines an action according to its own situation based on the determination result and a control program stored in the DRAM, and drives a necessary actuator based on the determination result to thereby make a humanoid robot. The apparatus 200 is caused to perform actions such as “gesture” and “hand gesture”.
[0060]
Accordingly, the humanoid robot device 200 can determine its own and surrounding conditions based on the control program, and can act autonomously according to instructions and actions from the user. In addition, the humanoid robot device 200 determines how to pronounce (read) the character extracted from the image captured by the CCD camera, and how to read the estimated character from the extracted character and the sound collected by the sound collecting microphone. Match and decide. Accordingly, the accuracy of speech recognition of the humanoid robot device 200 is improved, and new words can be registered in the speech recognition dictionary.
[0061]
(2) Application examples
Hereinafter, prior to the description of the embodiment of the present invention, in order to facilitate understanding, in the above-described biped walking robot apparatus, template matching is performed on the input image to extract face candidates, and the face candidates are extracted from the face candidates. Another example in which a face detection device for determining a face is mounted will be described (see Japanese Patent Application No. 2002-074907, hereinafter referred to as an application example).
[0062]
Also in the first and second application examples described below, after roughly extracting face candidates by template matching or the like as in the present invention, it is determined by SVM or the like whether or not the face is a face. As a result, the amount of calculation in the subsequent stage is reduced and the real-time property of the face detection of the robot apparatus is improved.
[0063]
(2-1) First application example
(2-1-1) Internal configuration of robot
In the first application example, as shown in FIGS. 4 and 5, the main control unit 381 that controls the operation of the entire robot apparatus 200 is provided on the back side of the waist base that forms the lower body part of the body unit 204. A control unit 42 in which a peripheral circuit 381 such as a power supply circuit and a communication circuit, a battery 45 (FIG. 5) and the like are housed in a box is disposed.
[0064]
The control unit 42 includes each sub-control unit 335 disposed in each component unit (body unit 202, head unit 204, arm units 203R and 203L, and leg units 209R and 209L). 350R, 350L, 365R, 365L are connected to supply necessary power supply voltage to these sub-control units 335, 350R, 350L, 365R, 365L, and these sub-control units 335, 350R, 350L, 365R, It is possible to communicate with 365L.
[0065]
In addition, each of the sub-control units 335, 350R, 350L, 365R, and 365L includes each actuator A in the corresponding component unit₂~ A₂₁And each actuator A in the component unit₂~ A₂₁Can be driven to a designated state based on various control commands given from the main control unit 381.
[0066]
Further, as shown in FIG. 5, the head unit 204 includes a CCD (Charge Coupled Device) camera 50 that functions as an “eye” of the robot apparatus 200, a microphone 51 that functions as an “ear”, a touch sensor 52, and the like. An external sensor unit 53 and a speaker 54 functioning as a “mouth” are arranged at predetermined positions, and an internal sensor unit 57 including a battery sensor 55 and an acceleration sensor 56 is arranged in the control unit 380. Has been.
[0067]
The CCD camera 50 of the external sensor unit 53 images the surrounding situation, sends the obtained image signal S1A to the main control unit 381, and sequentially stores it in the internal memory 381A in units of frames, while the microphone 51 is used by the user. Various command voices such as “walk”, “sit down”, and “follow the ball” given as voice inputs are collected, and the voice signal S1B thus obtained is sent to the main control unit 381.
[0068]
Further, the touch sensor is provided at the upper part of the head unit 3 and detects the pressure received by the physical action such as “blowing” or “striking” from the user, and the detection result is mainly used as the pressure detection signal S1C. The data is sent to the control unit 381.
[0069]
Furthermore, the battery sensor 55 of the internal sensor unit 57 detects the remaining energy of the battery 45 at a predetermined cycle, and sends the detection result to the main control unit 381 as a remaining battery level detection signal S2A. On the other hand, the acceleration sensor 56 detects accelerations in the three-axis directions (x-axis, y-axis, and z-axis) at a predetermined cycle, and sends the detection result to the main control unit 381 as an acceleration detection signal S2B.
[0070]
The main control unit 381 includes an image signal S1A, an audio signal S1B, and a pressure detection signal S1C supplied from the CCD camera 50, the microphone 51, the touch sensor 52, and the like of the external sensor unit 53 (hereinafter collectively referred to as external sensor signals). S1), the remaining battery level detection signal S2A and the acceleration detection signal S2B supplied from the battery sensor 55, the acceleration sensor, and the like of the internal sensor unit 57 (hereinafter collectively referred to as the internal sensor signal S2). Based on this, the situation around and inside the robot apparatus 200, the command from the user, the presence / absence of the action from the user, and the like are determined.
[0071]
The main control unit 381 determines the action to be continued based on the determination result, the control program stored in the internal memory 381A in advance, and the various control parameters stored in the external memory 58 loaded at that time. Then, the control command based on the determination result is sent to the corresponding sub-control units 335, 350R, 350L, 365R, 365L. As a result, based on this control command, the corresponding actuator A is controlled under the control of the sub-control units 335, 350R, 350L, 365R, 365L.₂~ A₂₁Thus, the robot apparatus 200 exhibits actions such as swinging the head unit 204 up and down, left and right, raising the arm units 203R and 203L, and walking.
[0072]
At this time, the main control unit 381 outputs a sound based on the sound signal S3 to the outside by giving a predetermined sound signal S3 to the speaker 54 as necessary, or functions as an “eye” in appearance. By outputting a drive signal to the LED provided at a predetermined position of the unit 204, it is blinked.
[0073]
In this manner, the robot apparatus 200 can act autonomously based on the surrounding and internal situations, the command from the user, the presence / absence of an action, and the like.
[0074]
(2-1-2) Processing of main control unit 381 regarding face detection task function
Next, the face detection task function installed in the robot apparatus 200 will be described. The robot apparatus 200 is equipped with a face detection task function for detecting a human face image from the frame image stored in the internal memory 381A via the CCD camera 50. The face detection task function is realized by various processes in the main control unit 381.
[0075]
Here, when the processing contents of the main control unit 381 related to the face detection task function are functionally classified, as shown in FIG. 6, the input image scale conversion unit 60, the window cutout unit 61, the template matching unit 62, the preprocessing It can be divided into the part 63, the pattern identification part 64, and the overlap determination part 65.
[0076]
The input image scale conversion unit 60 reads a frame image based on the image signal S1A from the CCD camera 50 (FIG. 5) from the internal memory 381A, and converts the frame image into a plurality of scale images having different reduction rates. In the case of this application example, a frame image composed of 25344 (= 176 × 144) pixels is sequentially reduced by 0.8 times to obtain 5 stages (1.0 times, 0.8 times, 0.64 times). , 0.51 times, 0.41 times) scale images (hereinafter referred to as first to fifth scale images).
[0077]
The subsequent window cut-out unit 61 first selects appropriate pixels (for example, two pixels) from the first scale image to the lower right of the image in order from the upper left of the first scale image. A rectangular area of 400 (= 20 × 20) pixels (hereinafter, this area is referred to as a window image) is sequentially cut out while scanning while shifting to the right side or the lower side.
[0078]
At that time, the window cutout unit 61 sends the first window image among the plurality of window images cut out from the first scale image to the template matching unit 62 at the subsequent stage.
[0079]
The template matching unit 62 converts the head window image obtained from the window cutout unit 61 into a function curve having a peak value by performing arithmetic processing such as a normalized correlation method and an error square method, A sufficiently low threshold is set to such a degree that the recognition performance does not deteriorate with respect to the curve, and it is determined whether or not the window image is a face image based on the threshold.
[0080]
In the case of this application example, the template matching unit 62 sets, for example, an average face image consisting of an average of about 100 persons as a template, and sets a threshold value as a criterion for determining whether or not the face image is such. Yes. As a result, the window image can be roughly matched with an average face image as a template.
[0081]
In this manner, the template matching unit 62 matches the window image obtained from the window cutout unit 61 with the template, and when it is determined that the image is a face image, the template image is used as a score image in the subsequent stage. On the other hand, when it is determined that the image is not a face image, the window image is directly transmitted to the subsequent overlap determination unit 65.
[0082]
The window image (score image) that has been determined to be a face image at this time actually includes a large amount of determination error images other than the face image, but in a daily scene, a background image similar to a face Therefore, it is determined that most window images are not face images, which is extremely effective.
[0083]
Actually, the calculation processing such as the normalized correlation method and the error square method described above requires a calculation amount of about 1/10 to 1/100 compared with the calculation processing in the pre-processing unit and the pattern identification unit in the subsequent stage. In the experiment, it was confirmed that 80% or more of images other than the face image can be screened out at this stage. Therefore, it can be seen that the main control unit 381 as a whole greatly reduces the calculation amount.
[0084]
For the score image obtained from the template matching unit 62, the preprocessing unit 63 removes the four corner regions corresponding to the background portion that are unrelated to the human face image from the score image that is a rectangular region. Using a mask obtained by cutting out the four corner areas, 360 pixels are extracted from a score image having 400 (= 20 × 20) pixels.
[0085]
Then, the preprocessing unit 63 forms a plane based on the most suitable part as the face image in the extracted score image for 360 pixels in order to eliminate the inclination condition of the subject expressed by shading by illumination at the time of imaging. As described above, for example, the grayscale value of the 360 pixels is corrected using a calculation method based on a mean square error (RSM).
[0086]
Subsequently, the preprocessing unit 63 performs the histogram smoothing process on the result of enhancing the contrast of the score image for 360 pixels so that it can be detected regardless of the gain of the CCD camera 50 or the intensity of illumination.
[0087]
Next, the preprocessing unit 63 performs vector conversion of the score image for 360 pixels by performing Gabor Filtering processing described later, and further converts the obtained vector group into one pattern vector. .
[0088]
The Gabor filtering process will be described below. First, it is already known that human visual cells have cells that are selective for a specific orientation. It consists of cells that respond to vertical lines and cells that respond to horizontal lines. Similarly, Gabor filtering is a spatial filter composed of a plurality of filters having orientation selectivity.
[0089]
A Gabor filter is spatially expressed by a Gabor function. The Gabor function g (x, y) includes a carrier s (x, y) composed of a cosine component and an envelope W of a two-dimensional Gaussian analysis as shown in the following formula (1)._r(X, y).
[0090]
[Expression 1]

[0091]
The carrier s (x, y) is expressed by the following equation (2) using a plurality of functions. Where the coordinate value (u₀, V₀) Represents the spatial frequency, and P represents the phase of the cosine component.
[0092]
[Expression 2]

[0093]
Here, the carrier shown in the above equation (2) can be separated into a real component Re (s (x, y) and an imaginary component Im (s (x, y)) as shown in the following equation (3). it can.
[0094]
[Equation 3]

[0095]
On the other hand, an envelope composed of a two-dimensional Gaussian distribution is expressed as the following equation (4).
[0096]
[Expression 4]

[0097]
Here, the coordinate axis (x₀, Y₀) Is the peak of this function, and constants a and b are the Gaussian distribution scale parameters. The subscript r means a rotation operation as shown in the following formula (5).
[0098]
[Equation 5]

[0099]
Therefore, from the above equations (2) and (4), the Gabor filter is expressed as a spatial function as shown in the following equation (6).
[0100]
[Formula 6]

[0101]
The preprocessing unit according to this application example employs eight types of directions and three frequencies, and performs face extraction processing using a total of 24 Gabor filters.
[0102]
The response of the Gabor filter is G_iIs the i th Gabor filter, and the i th Gabor result (Gabor Jet) is J_iWhen the input image is I, it is expressed by the following formula (7). The calculation of equation (7) can actually be speeded up using fast Fourier transform.
[0103]
[Expression 7]

[0104]
The performance of the created Gabor filter is examined by reconstructing the pixels obtained by filtering. The reconstructed image H is expressed as the following formula (8).
[0105]
[Equation 8]

[0106]
An error E between the input image I and the reconstructed image H is expressed by the following equation (9).
[0107]
[Equation 9]

[0108]
It can be reconstructed by obtaining an optimum a that minimizes the error E. As for Gabor filtering, the type of filter may be changed according to the recognition task.
[0109]
In low frequency filtering, it is redundant to have all the filtered images as vectors. Therefore, downsampling may be performed to reduce the vector dimension. The 24 types of downsampled vectors are arranged in a line to make a long vector (the above-described pattern vector).
[0110]
The pattern discriminating unit 64 obtains a provisional discriminant function using learning data supplied from the outside, that is, teacher data, and then obtains the discriminant function for 360 pixels obtained as a pattern vector from the preprocessing unit 63. Try to detect the face image and detect the face. Then, what has been successfully detected is output as face data. Further, the data that has failed to be detected is added to the learning data as non-face data, and further learning is performed.
[0111]
In this application example, regarding the face recognition in the pattern identification unit 64, identification of whether or not the face is an appropriate face using a support vector machine (Support Vector Machine: SVM) that has the highest learning generalization ability in the field of pattern recognition. I do.
[0112]
Regarding the support vector machine itself, for example, a report by B. sholkopf (B. Sholkopf, C. Burges, A. Smola, “Advance in Kernel Support Vector earning”, The MIT Press, 1999.) can be cited. From the result of the preliminary experiment conducted by the applicant of the present application, it is known that the face recognition method using the support vector machine shows a better result than the method using the principal component analysis (PCA) or the neural network.
[0113]
The support vector machine is a learning machine using a linear classifier (perceptron) as a discriminant function, and can be extended to a non-linear space by using a kernel function. In the learning of discriminant functions, the margin of separation between classes is maximized, and the solution can be obtained by solving quadratic mathematical programming, so that it is theoretically guaranteed that a global solution can be reached. it can.
[0114]
Usually, the problem of pattern recognition is to obtain the discriminant function f (x) given by the following equation (10) for the test sample x = (x1, x2,..., Xn).
[0115]
[Expression 10]

[0116]
Here, a teacher label for learning the support vector machine is set as shown in the following equation (11).
[0117]
## EQU11 ##

[0118]
Then, recognition of the face pattern in the support vector machine can be regarded as a problem of minimizing the square of the weighting factor w under the constraint condition shown in the following equation (12).
[0119]
[Expression 12]

[0120]
Problems with such constraints can be solved using Lagrange's undetermined constant method. That is, a Lagrangian represented by the following formula (13) is first introduced, and then a partial differentiation is performed for each of b and w as represented by the following formula (14).
[0121]
[Formula 13]

[0122]
[Expression 14]

[0123]
As a result, the identification of the face pattern in the support vector machine can be regarded as a quadratic programming problem expressed by the following equation (15).
[0124]
[Expression 15]

[0125]
When the number of dimensions of the feature space is smaller than the number of training samples, a scratch variable ξ ≧ 0 is introduced and the constraint condition is changed as in the following equation (16). For optimization, the objective function of the following equation (17) is minimized.
[0126]
[Expression 16]

[0127]
[Expression 17]

[0128]
In the above equation (17), C is a coefficient that specifies how much the constraint condition is relaxed, and it is necessary to determine the value experimentally. The problem relating to the Lagrange constant a is changed as shown in the following equation (18).
[0129]
[Formula 18]

[0130]
However, with the above equation (18), the nonlinear problem cannot be solved. Therefore, in this application example, a kernel function K (x, x3) is introduced, and once mapped to a high-dimensional space (kernel trick), linear separation is performed in that space. Therefore, it is equivalent to non-linear separation in the original space.
[0131]
The kernel function is expressed by the following formula (19) using a certain mapping Φ.
[0132]
[Equation 19]

[0133]
Further, the discriminant function shown in the above equation (10) can also be expressed as in the following equation (20).
[0134]
[Expression 20]

[0135]
Learning can also be considered as a quadratic programming problem expressed by the following equation (21).
[0136]
[Expression 21]

[0137]
As the kernel, a Gaussian kernel (RBF: Radius Basic Function) represented by the following formula (22) can be used.
[0138]
[Expression 22]

[0139]
As described above, the pattern identifying unit 64 determines whether or not face data exists in the score image with respect to the pattern vector based on the score image given from the preprocessing unit 63, and only in the case of the presence of the score image. The upper left position (coordinates) and size (number of vertical and horizontal pixels) in the image area, and the reduction ratio of the scale image from which the score image is cut out to the frame image (that is, the corresponding stage among the above five stages) Are stored in the internal memory 381A as list data.
[0140]
Thereafter, the pattern identifying unit 64 notifies the window cutout unit 61 that the face detection of the first window image in the first scale image has been completed, so that the window cutout unit 61 makes a first change. The window image scanned next out of one scale image is sent to the template matching unit 62.
[0141]
The template matching unit 62 transmits the score image to the preprocessing unit 63 only when the window image matches the template. The preprocessing unit 63 converts the score image into a pattern vector and sends the pattern vector to the pattern identification unit 64. The pattern identification unit 64 generates list data based on the face data obtained as the identification result from the pattern vector, and stores it in the internal memory 381A.
[0142]
As described above, the window matching unit 62, the preprocessing unit 63, and the pattern identification unit 64 perform the respective processes in the scan order for all window images cut out from the first scale image in the window cutting unit 61. A plurality of score images including a face image existing in the imaging result can be detected from one scale image.
[0143]
Thereafter, the pattern identification unit 64 notifies the input image scale conversion unit 60 that the face detection of the first scale image has been completed, so that the input image scale conversion unit 60 outputs the second scale image. Is sent to the window cutout 61.
[0144]
The second scale image is also subjected to the same processing as the first scale image described above, and after detecting a plurality of score images including face images existing in the imaging result from the second scale image, The same processing is sequentially performed for the fifth to fifth scale images.
[0145]
Thus, the pattern identification unit 64 detects a plurality of score images including face images existing in the captured image with respect to the first to fifth scale images obtained by reducing the frame image that is the captured image in five stages. The obtained list data is stored in the internal memory 381A. In this case, depending on the size of the face image in the original frame image, a score image may not be obtained at all. However, if the score image is obtained with a scale image of at least 1 (or 2 or 3). The face detection process is continued.
[0146]
Here, a plurality of score images including face images in each scale image are scanned even by two pixels at the window cutout unit 61. Therefore, there is a high correlation between the region where the face is actually located and its neighboring region. In other words, the adjacent score images include image areas that overlap each other.
[0147]
Then, the overlap determination unit 65 that follows reads out a plurality of list data for each of the first to fifth scale images stored in the internal memory 381A, compares the score images included in the respective list data, It is determined whether or not regions that overlap each other are included.
[0148]
At that time, as shown in FIG. 7, the overlap determination unit 65 determines that the positions (coordinates) of the upper left corners of the two score images P1 and P2 are (X_A, Y_A), (X_B, Y_B) And the size (number of vertical and horizontal pixels) is H_A× L_A, H_B× L_BDX (= X_B-X_A), DY (= Y_B-Y_A), It can be determined whether or not the score images P1 and P2 overlap each other depending on whether or not the following expression (23) is satisfied.
[0149]
[Expression 23]

[0150]
Based on the determination result, the overlap determination unit 65 removes an area where the score images overlap each other, so that in each scale image, a plurality of score images are finally collected without overlapping each other. The image area is newly stored in the internal memory 381A as face determination data.
[0151]
When the template matching unit 62 determines that the overlap determination unit 65 is not a face image, the overlap determination unit 65 does nothing and does not store the internal memory 381A.
[0152]
(2-1-3) Operations and effects in the first application example
In the above configuration, the robot apparatus 200 converts a frame image captured by the CCD camera 50 into a plurality of scale images having different reduction ratios, and then converts a window image of a predetermined size from the scale images to a predetermined pixel. Cut out one by one while scanning to shift each.
[0153]
For this window image, matching is performed using a template representing an average face image to roughly determine whether the image is a face image. The amount of computation and time required for the face detection process can be reduced accordingly.
[0154]
Subsequently, for the window image (that is, the score image) determined to be a face image by template matching, after removing the four corners of the rectangular area of the score image, density correction and subsequent smoothing of contrast enhancement are performed, and Convert to one pattern vector.
[0155]
Then, for the pattern vector, face detection in the original score image is performed to determine face data or non-face data, and the position (coordinates) and the size (number of pixels) of the image area of the score image where the face data exists. ) And the reduction ratio of the scale image to the frame image from which the score image is cut out.
[0156]
In this way, by generating list data for all score images for each scale image, by comparing the score images included in each list data, by obtaining face determination data from which the overlapping areas are removed The face image can be detected from the original frame image.
[0157]
Among these face detection task processes, the template matching process, in particular, can be easily implemented on an arithmetic unit having a relatively simple configuration, and is similar to a block matching method used for image compression or the like. There are many hardware that perform high-speed processing using a CPU. Accordingly, the template matching process can be further speeded up.
[0158]
According to the above configuration, in the robot apparatus 200, during the face detection task processing for detecting the face image for the frame image captured by the CCD camera 50, the scale image is reduced among the different scale images. After each window image of a predetermined size is scanned one by one while being scanned so as to be shifted by a predetermined pixel, matching is performed using a template representing an average face image to determine whether the image is roughly a face image. Thus, by removing window images that are clearly not face images, the amount of computation and time required for various face detection processes for score images determined to be face images by the template matching are reduced accordingly. The processing of the main control unit 381 that controls the entire robot apparatus 200 can be reduced. It is possible to reduce collateral, thus possible to realize a robot apparatus 200 capable of significantly enhancing real-time.
[0159]
(2-2) Second application example
(2-2-1) Internal configuration of robot
In the second application example, the back side of the waist base that forms the lower part of the torso of the torso unit 202 (FIG. 1) is described above, as shown in FIG. In addition to the main control unit 381, an internal memory 381A, a DMA (Direct Memory Access) controller 70, an arithmetic processing unit (DSP (Digital Signal Processor), an image processing engine, etc.) are connected via a bus connected to the main control unit 381. ) 71 are connected to each other, and the other configuration is the same as in the first application example.
[0160]
In this case, only the main control unit 381 does not manage the internal memory 381A, but the DMA controller 70 transfers data to the main control unit 381 or the arithmetic processing unit 71. On the other hand, the calculation result obtained from the main control unit 381 or the calculation processing unit 71 is stored in the internal memory 381A via the DMA controller 70.
[0161]
In this way, the DMA controller 70 performs the data transfer and distributes the data calculation to the main control unit 381 or the calculation processing unit 71 so as to be performed independently of each other, thereby reducing the burden on the main control unit 381. ing.
[0162]
(2-2-2) Processing of main control unit 381 regarding the face detection task function
In the case of the second application example, the face detection task function is realized by various processes in the main control unit 381 and the arithmetic processing unit 71 via the DMA controller 70.
[0163]
Here, when the processing contents of the main control unit 381 and the arithmetic processing unit 71 relating to the face detection task function are functionally classified, as shown in FIG. 9, an input image scale conversion unit 80, a window cutout unit 81, a template, It can be divided into a matching unit 82, an overlap determination unit 83, a scale conversion and extraction unit 84, a preprocessing unit 85, and a pattern identification unit 86.
[0164]
Among them, the input image scale conversion unit 80, the window cutout unit 81, and the template matching unit 82 are processed by the arithmetic processing unit 71, and the overlap determination unit 83, the scale conversion and cutout unit 84, the preprocessing unit 85, and the pattern identification. The unit 86 is configured to be processed by the main control unit 381. The DMA controller 70 switches processing between the arithmetic processing unit 71 and the main control unit 381.
[0165]
First, the input image scale conversion unit 80 reads a frame image based on the image signal S1A from the CCD camera 50 (FIG. 8) from the internal memory 381A, and converts the frame image into a plurality of scale images having different reduction ratios (described above). (First to fifth scale images).
[0166]
The subsequent window cut-out unit 81 first extracts appropriate pixels (for example, two pixels) from the first scale image to the lower right of the image in order from the upper left of the first scale image. Scanning while skipping, a window image consisting of a rectangular area of 400 (= 20 × 20) pixels is sequentially cut out.
[0167]
At that time, the window cutout unit 81 sends the first window image among the plurality of window images cut out from the first scale image to the template matching unit 82 at the subsequent stage.
[0168]
The template matching unit 82 uses the average face image as a template for the top window image obtained from the window cutout unit 81, performs rough matching, and the degree of correlation with the template (the matching pixel in the template). (Number ratio) is stored as correlation degree data in the internal memory 381A through the DMA controller 70 together with the window image.
[0169]
Specifically, as shown in FIG. 10, as a result of taking a template T1 that is an average face image for a window image W1 cut out from an arbitrary scale image F1, and taking a correlation with the template T1, An image R1 representing the degree of correlation is detected and stored as correlation degree data in the internal memory 381A.
[0170]
In the case of this second application example, unlike the first application example in which screen images other than the score image determined to be a face image are screened out, the window image is converted into a score image in order to realize high-speed processing. Regardless of whether or not, it is sent to the overlap determination unit 83 in the subsequent stage.
[0171]
Thereafter, the template matching unit 82 notifies the window cutout unit 81 that the template matching of the first window image in the first scale image has been completed. The window image scanned next out of one scale image is sent to the template matching unit 82.
[0172]
In this manner, the template matching unit 82 generates correlation degree data in the scan order for all window images cut out from the first scale image by the window cutout unit 81.
[0173]
Thereafter, the template matching unit 82 notifies the input image scale conversion unit 80 that face detection of the first scale image has been completed, whereby the second scale image is input from the input image scale conversion unit 80. Is sent to the window cutout section 81.
[0174]
The second scale image is processed in the same manner as the first scale image described above to generate correlation degree data corresponding to all window images from the second scale image, and then the third to third scale images. The same processing is sequentially performed for the scale image 5.
[0175]
Thus, the template matching unit 82 generates correlation data corresponding to a plurality of window images cut out for each of the first to fifth scale images obtained by reducing the frame image that is the captured image in five stages, and the window image At the same time, it is stored in the internal memory 381A via the DMA controller 70.
[0176]
The processing content in the arithmetic processing unit 71 has been described above, and is switched to the main control unit 381 in accordance with the control of the DMA controller 70. The processing content in the main control unit 381 is as follows.
[0177]
First, the overlap determination unit 83 executes the overlap determination processing procedure R11 of FIG. 11 from step SP0, and for each of the first to fifth scale images, the scan order is the same as when the window image in each scale image is cut out. In order, the window image is read out from the internal memory 381A via the DMA controller 70 together with the correlation data associated therewith (step SP1).
[0178]
Subsequently, the overlap determining unit 83 compares the window images with each other in the scan order using the above-described equation (23) for the plurality of window images in the corresponding scale image, and determines whether or not the regions include overlapping regions. (Step SP2).
[0179]
Then, the overlap determining unit 83 compares the window images with the corresponding correlation levels only when including overlapping regions, and sets the larger correlation level as a score image in the internal memory 381A. It is added to the candidate list and stored (step SP3).
[0180]
Then, the overlap determination unit 83 adds, as a score image, the larger one when the correlation degrees are compared for all window images in the corresponding scale image to the candidate list (step SP4). On the other hand, the smaller one is sequentially rewritten without being added to the candidate list or deleted if it already exists in the candidate list (step SP5).
[0181]
Eventually, the overlap determination unit 83 determines a plurality of score images having a relatively high correlation degree as a candidate list for all window images in the corresponding scale image, and then a plurality of relatively high correlation degrees for other scale images. Are determined as a candidate list. It has been experimentally obtained that the number of score images having a relatively high degree of correlation added as a candidate list can be reduced to 10 or less for each scale image.
[0182]
In this manner, the overlap determination unit 83 adds the score image of the first to fifth scale images to the candidate list not only for the peak value with the highest degree of correlation, but also for the area near the peak value. As a result, the difference between the matching result in the template matching unit 82 and the processing result of the support vector machine in the pattern identification unit 86 in the subsequent stage can be reduced accordingly.
[0183]
In practice, the candidate list creation process in the overlap determination unit 83 can be executed by several additions and multiplications, so that it is much faster than the calculation process performed by the pre-processing unit 85 and the pattern identification unit 86 in the subsequent stage. It can be processed in a short time. Then, if the score image remaining as a candidate list is processed by the preprocessing unit 85 and the pattern identification unit 86, the preprocessing unit 85 and the pattern identification unit 86 can also reduce the processing load.
[0184]
Thereafter, the scale conversion / cut-out unit 84 applies a plurality of score images stored in the internal memory 381A as a candidate list for each scale image in the overlap determination unit 83 from the internal memory 381A via the DMA controller 70. The score images are read one by one within the scale image and in the order of determination of the candidate list, and sent to the preprocessing unit 85.
[0185]
For the score image obtained from the scale conversion and cutout unit 84, the preprocessing unit 85 has four corners corresponding to a background portion that is irrelevant to the human face image from the score image (400 pixels) that is a rectangular area. In order to remove the region, 360 pixels are extracted using a mask obtained by cutting out the four corner regions.
[0186]
Then, the pre-processing unit 85 forms a plane based on the most suitable part as the face image among the extracted score images for 360 pixels in order to eliminate the inclination condition of the subject expressed by shading by the illumination at the time of imaging. In this way, the gray value is corrected in pixel units.
[0187]
Subsequently, the pre-processing unit 85 performs a histogram smoothing process on the result of enhancing the contrast of the score image for 360 pixels so that it can be detected regardless of the gain of the CCD camera 50 or the intensity of illumination.
[0188]
Next, the preprocessing unit 85 performs the above-described Gabor filtering process, thereby vector-converting the 360-pixel score image, and further converting the obtained vector group into one pattern vector.
[0189]
The pattern identification unit 86 uses the above-described support vector machine to perform the identification of whether or not it is a face by trying the 360-pixel score image obtained as a pattern vector from the preprocessing unit 85.
[0190]
As described above, the pattern identification unit 86, for the pattern vector based on the score image given from the preprocessing unit 85, the position (coordinates) of the image area corresponding to the face data in the score image and its size (number of pixels) ) And the reduction ratio of the scale image from which the score image is cut out to the frame image (that is, the corresponding stage among the above-described five stages) are stored in the internal memory 381A as list data.
[0191]
Thereafter, the pattern identification unit 86 notifies the scale conversion / cutout unit 84 that the face detection of the first score image in the first scale image has been completed, thereby performing the scale conversion / cutout. The next score image in the first scale image is sent from the unit 84 to the preprocessing unit 85. The preprocessing unit 85 converts the score image into a pattern vector and sends it to the pattern identification unit 86. The pattern identification unit 86 generates list data based on the face data obtained from the pattern vector and stores it in the internal memory 381A.
[0192]
As described above, the scale conversion / cutout unit 84 sequentially performs the respective processes of the preprocessing unit 85 and the pattern identification unit 86 on all score images in the candidate list in the first scale image, so that the first The face image present in the imaging result can be detected from the scale image.
[0193]
Thereafter, the pattern identifying unit 86 notifies the scale conversion and clipping unit 84 that the face detection of the first scale image has been completed, so that the second scale image is also described above. The same processing as that of the first scale image is performed to detect a face image existing in the imaging result from the second scale image, and then the same processing is sequentially performed for the third to fifth scale images.
[0194]
Thus, the pattern identification unit 86 detects the face images present in the captured image from the first to fifth scale images obtained by reducing the frame image that is the captured image in five stages, and then obtains the face obtained as a result. Each list data is generated based on the data and stored in the internal memory 381A.
[0195]
(2-2-3) Operations and effects in the second application example
In the above configuration, the robot apparatus 200 converts a frame image captured by the CCD camera 50 into a plurality of scale images having different reduction ratios, and then converts a window image of a predetermined size from the scale images to a predetermined pixel. Cut out one by one while scanning to shift each.
[0196]
The window image is matched using a template representing an average face image to generate correlation degree data representing the degree of correlation with the template. In this way, correlation data is generated for each scale image in the scan order for all window images.
[0197]
The processing up to this point is executed in the arithmetic processing unit 71 and the subsequent processing is executed in the main control unit 381, so that the processing load on the main control unit 381 can be reduced accordingly.
[0198]
Subsequently, for a plurality of window images in each scale image, the window images are compared with each other in the scan order, and if they include overlapping regions, the correlation is further compared and only the larger one remains as a score image. Add to list. As a result, a plurality of score images having a relatively high degree of correlation can be determined as a candidate list for each scale image, and the identification accuracy can be further improved during the subsequent face identification process.
[0199]
Subsequently, for a plurality of score images stored in the internal memory 381A as a candidate list for each scale image, after removing the four corners of the rectangular area of the score image, density correction and subsequent smoothing of contrast enhancement are performed, Further, it is converted into one pattern vector.
[0200]
Then, for the pattern vector, face detection in the original score image is performed to determine face data or non-face data, and the position (coordinates) and the size (number of pixels) of the image area of the score image where the face data exists. ) And the reduction ratio of the scale image to the frame image from which the score image is cut out.
[0201]
Thus, by generating list data for all the score images for each scale image, it is possible to detect a face image from the original frame image.
[0202]
Thus, regarding the face detection task processing, in addition to the main control unit 381 that controls the entire robot apparatus 200, the arithmetic processing unit 71 is provided, and the main control unit reads and writes data to and from the internal memory 381A via the control of the DMA controller 70. Since both the main control unit 381 and the arithmetic processing unit 71 can distribute the processing, the calculation amount and the calculation time required for the main control unit 381 can be reduced. The burden can be significantly reduced.
[0203]
According to the above configuration, in the robot apparatus 200, during the face detection task processing for detecting the face image for the frame image captured by the CCD camera 50, the scale image is reduced among the different scale images. Correlation data representing the degree of correlation with the template by using a template representing an average face image after cutting each window image of a certain size while scanning the window image so as to shift by a predetermined pixel. If the window images include regions that overlap each other, the degree of correlation is compared and only the larger one is left in the candidate list, so that the degree of correlation in the candidate list is relatively high. Reduce the amount of computation and time required for various face detection processes for multiple score images. DOO can not only, the identification accuracy in the face identification processing can be further improved.
[0204]
In addition to this, regarding the face detection task processing, the main control unit 381 that controls the entire robot apparatus 200 and the arithmetic processing unit 71 share the processing so that the calculation related to the main control unit 381 is performed. It is possible to realize a robot apparatus 200 that can remarkably reduce the burden of the amount and the calculation time, and thus can significantly improve the real-time property.
[0205]
(3) Embodiment
Next, an embodiment of the present invention will be described. As described above, in the method of extracting face candidates by performing template matching (first step), determining a face region from the face candidates by SVM or the like (second step), and detecting the face region In the first step, the face candidate is determined simply by the compensation of the normalized correlation value. Therefore, when it is attempted to reduce the oversight of the face candidate, a method of increasing the threshold or a method of reducing the thinning may be taken. Although it is possible, if the threshold value is lowered, the amount of calculation increases, which may not be preferable in an environment with limited resources such as a robot apparatus. On the other hand, if the threshold value is increased, the number of candidate images for face determination in the second step is reduced, so that the amount of calculation can be reduced. May end up.
[0206]
In view of this, when the face area (face image) having the same size as the template exists in the input image, the inventors of the present application have the largest correlation value in the vicinity of the template size if the correlation between the face image and the template is taken. In particular, when narrowing down face area candidates, an algorithm that performs local narrowing down is used to reduce face candidate images without missing the original face image, and to the second step in the subsequent stage. And found a method to reduce the amount of calculation to determine the face. Specifically, a candidate face area is extracted based on the local maximum value of the correlation value in the matching result, which is a set of correlation values obtained by taking a normalized correlation between the input image and the average face template of a predetermined size. is there. Hereinafter, this face detection method will be described in detail.
[0207]
The face detection device according to the present embodiment maintains the performance while maintaining the performance in a system with limited resources such as a CPU and a memory such as a biped robot device similar to the first and second application examples described above. This is suitable for reducing the amount of calculation required for detection. Also in the present embodiment, as in the first and second application examples described above, the robot apparatus 200 shown in FIGS. 1 to 4 is mounted, and the internal configuration thereof is the same as that in the application example described above. Detailed description thereof is omitted.
[0208]
(3-1) Processing of main control unit 381 regarding the face detection task function
The face detection task function mounted on the robot apparatus 200 in the present embodiment will be described. As described above, the robot apparatus 200 is equipped with a face detection task function for detecting a human face image from the frame image stored in the internal memory 381A via the CCD camera 50. The face detection task function is realized by various processes in the main control unit 381.
[0209]
Here, when the processing contents of the main control unit 381 regarding the face detection task function are functionally classified, as shown in FIG. 12, a template size and input image scale conversion determination unit 90, a window cutout unit 91, and a template matching unit 92 are obtained. , Scale conversion and cutout unit 93, preprocessing unit 94, and pattern identification unit 95.
[0210]
As in the first application example, the input image scale conversion determination unit 90 reads out a frame image based on the image signal S1A from the CCD camera 50 (FIG. 5) from the internal memory 381A, and compares the frame image with a reduction ratio. While converting to a plurality of different scale images (first to fifth scale images), the template size used in the template matching unit 92 in the subsequent stage, that is, the size of the face image is selected (hereinafter referred to as the first template). Called size).
[0211]
The subsequent window cut-out unit 91 first selects appropriate pixels (for example, two pixels) from the first scale image to the lower right of the image starting from the upper left of the first scale image. Scanning while skipping, a window image consisting of a rectangular area of 400 (= 20 × 20) pixels is sequentially cut out.
[0212]
At that time, the window cutout unit 91 selects the first window image among the plurality of window images cut out from the first scale image together with the template of the first template size selected by the scale conversion determination unit 91 in the subsequent template matching. To the unit 92.
[0213]
The template matching unit 92 roughly matches the first window image obtained from the window cutout unit 91 using an average face image having the first template size as a template, and a correlation value (template) with the template. Of the matching pixels) is stored as a matching result in the internal memory 381A via the DMA controller 70 together with the window image.
[0214]
That is, as shown in FIG. 13A, a window image of height (side length in the y-axis direction) hei_s × width (side length in the x-axis direction) weid_s cut out from an arbitrary scale image, for example. (Input image after scale conversion) For W2, as shown in FIG. 13B, for example, a template T2 that is an average face image having a first template size of height hei_t × width wid_s is used. The window image W2 is scanned, and a matching result which is a set of correlation values between the template T2 moved while shifting by predetermined pixels (for example, one pixel) and the input image is obtained. This matching result is that the correlation values are two-dimensionally arranged with the movement of the template T2, and as shown in FIG. 14, a template matching result image R2 of height hei_r × width wid_r representing the correlation value is obtained. It is done. Here, the height her_r of the template rate matching result image R2 is hei_s− (hei_t + 1), and the width wid_s of the image R2 is wid_s− (wid_t + 1).
[0215]
Next, this template rate matching result image R2 is divided into a predetermined size, for example, the same size as the first template size, and the maximum correlation value is determined for each divided region divided by each first template size. A point (position) is obtained, and points indicating the maximum value obtained from each of the divided regions are extracted as face candidates that have a predetermined threshold value or more.
[0216]
That is, when normalized correlation is performed using an average face template, there is no guarantee that the correlation value of the face image will be higher than that of an arbitrary pattern, but there is a face image of the same size as the template. In this case, since the correlation value has the maximum value in the vicinity of the template size, the correlation value becomes the maximum value in the divided region, and a point that is equal to or larger than a predetermined threshold is extracted as a face candidate. As a result of matching, face candidates can be narrowed down more effectively than when face values having a correlation value equal to or greater than a predetermined threshold are extracted as face candidates.
[0217]
Next, the points extracted as the face candidates are set as face candidates (first face candidates), and the points near the face candidates are also extracted as face candidates (second face candidates). When the pattern recognition unit 95 in the subsequent stage uses the above-described SVM to determine a face, the SVM is so sensitive that it cannot detect a face only by shifting one pixel. By setting the search range (second face candidate), it is possible to focus on the vicinity of the face candidate and improve the face detection performance. Here, if all the eight points adjacent to the top, bottom, left, and right of the point extracted as the first face candidate are set as the search range, the calculation amount of the subsequent stage increases. Therefore, the search range is specified only when a predetermined condition is satisfied. As a result, it is possible to improve the detection performance while suppressing an increase in the calculation amount in the subsequent stage.
[0218]
In other words, in the points adjacent to the points extracted as the first face candidates, the skin color area occupancy rate (skin color occupancy rate) is equal to or greater than a predetermined threshold in the input image area of the template size corresponding to these points. Or, if there is face color information detected in advance (or learned in advance) and the occupancy ratio (face color occupancy ratio) of the face color area is equal to or greater than a predetermined threshold, that point is set as the search range. be able to. Here, the skin color occupancy ratio can be obtained by using, for example, a skin color color table and comparing with the skin color.
[0219]
Thereafter, the template matching unit 92 notifies the window cutout unit 91 that the template matching of the first window image in the first scale image has been completed. The window image scanned next out of one scale image is sent to the template matching unit 92.
[0220]
As described above, the template matching unit 92 detects face candidates for all window images cut out from the first scale image by the window cutout unit 91 in the scan order.
[0221]
The processing content in the arithmetic processing unit 71 has been described above, and is switched to the main control unit 381 in accordance with the control of the DMA controller 70. The processing content in the main control unit 381 is as follows.
[0222]
The scale conversion / cut-out unit 93 selects a plurality of score images stored in the internal memory 381A as a candidate list for each scale image from the internal memory 381A via the DMA controller 70 and within the first scale image. The score images are read one by one in the order in which the list is determined, and sent to the preprocessing unit 94.
[0223]
For the score image obtained from the scale conversion and clipping unit 93, the preprocessing unit 94 has four corners corresponding to a background portion that is irrelevant to the human face image from the score image (400 pixels) formed of a rectangular area. In order to remove the region, 360 pixels are extracted using a mask obtained by cutting out the four corner regions.
[0224]
Then, the pre-processing unit 94 forms a plane based on an optimal part as a face image in the extracted score image for 360 pixels so as to eliminate the inclination condition of the subject expressed by shading by illumination at the time of imaging. In this way, the gray value is corrected in pixel units.
[0225]
Subsequently, the preprocessing unit 94 performs a histogram smoothing process on the result of enhancing the contrast of the score image for 360 pixels so that it can be detected regardless of the gain of the CCD camera 50 or the intensity of illumination.
[0226]
Next, the preprocessing unit 94 performs the Gabor filtering process described above, thereby vector-converting the score image for 360 pixels, and further converting the obtained vector group into one pattern vector.
[0227]
The pattern identification unit (identification means) 95 uses the above-described support vector machine to perform a 360-pixel score image obtained as a pattern vector from the preprocessing unit 94 while learning whether or not it is a face. .
[0228]
As described above, the pattern identifying unit 95, for the pattern vector based on the score image given from the preprocessing unit 94, the position (coordinates) of the image area corresponding to the face data in the score image and its size (number of pixels) ), The reduction ratio of the scale image from which the score image is cut out with respect to the frame image (that is, the corresponding stage among the above-described five stages), and the size of the template, and list them as list data in the internal memory 381A. To store.
[0229]
Thereafter, the pattern identification unit 95 notifies the scale conversion and extraction unit 93 that the face detection of the first score image in the first scale image has been completed, thereby performing the scale conversion and extraction. The next score image in the first scale image is sent from the unit 93 to the preprocessing unit 94. The preprocessing unit 95 converts the score image into a pattern vector and sends it to the pattern identification unit 95. The pattern identification unit 95 generates list data based on the face data obtained from the pattern vector and stores the list data in the internal memory 381A.
[0230]
As described above, the scale conversion and extraction unit 93 performs the respective processes of the preprocessing unit 94 and the pattern identification unit 95 in order on all score images in the candidate list in the first scale image. A face image existing in the imaging result can be detected from one scale image.
[0231]
That is, the template matching unit 92 notifies the input image scale conversion unit 90 that the face detection using the first scale image and the template of the first template size has been completed. The conversion unit 90 sends the first scale image and the template of the second template size to the window cutout unit 91. Even when the second template size is used, the template matching unit 92 performs processing similar to that when the template of the first template size described above is used to detect face candidates corresponding to all template sizes. By notifying the first scale image of the input image scale conversion unit 90 that face detection using all template sizes has been completed, the second scale image is received from the input image scale conversion determination unit 90. The window is cut out by the window cutting unit 91.
[0232]
The second scale image is also processed in the same manner as the first scale image described above, and face candidates corresponding to all window images are detected from the second scale image. The same processing is sequentially performed on the scale images.
[0233]
Thus, the template matching unit 92 extracts face candidates corresponding to the plurality of window images cut out from the first to fifth scale images and the templates having a plurality of template sizes obtained by reducing the frame image as the captured image in five stages. Then, it is stored in the internal memory 381A via the DMA controller 70 together with the window image.
[0234]
Here, in this embodiment, a template of any size can be used, but all templates that can be prepared for the input image by switching the template size to be used and selecting the template size. Compared with the case of calculating the size, the amount of calculation can be reduced and the efficiency can be improved. For example, once a face is detected, the template size can be used for the next face detection. Further, for example, by using a distance sensor provided in the robot apparatus and recognizing the distance to the object included in the input image based on distance information from the distance sensor, the size of the face area of the object is determined. It is possible to provide target distance switching means for predicting the size and selecting the template size, and the template size can be switched according to the purpose.
[0235]
(3-2) Operation in the embodiment
In the above configuration, the robot apparatus 200 converts a frame image captured by the CCD camera 50 into a plurality of scale images having different reduction ratios, and then converts a window image of a predetermined size from the scale images to a predetermined pixel. Cut out one by one while scanning to shift each.
[0236]
The window image is matched using a template representing an average face image, and a matching result image that is a set of correlation values with the template is generated. In this manner, matching result images are generated in the scan order for all the window images for each scale image. Hereinafter, the process of detecting face candidates from the matching result image will be described in detail.
[0237]
FIG. 15 is a flowchart showing each processing step in which the template matching unit 92 detects pixels as face candidates from the template matching result image R2. As shown in FIG. 15, first, when a template matching result image R2 is input, the matching result image R2 is divided into template sizes, and one of the divided areas, for example, 0 ≦ x ≦ wid_t−1, 0 ≦ y. In ≦ hei_t−1, a point (coordinate) having the highest correlation value is extracted (step SP11). Hereinafter, the region obtained by dividing the matching result image R2 into the template size is referred to as a divided region rn, and the point (coordinate) having the largest correlation value in the divided region rn is referred to as local_max (x, y). Here, the pixel having the highest correlation value is extracted in each of the divided regions. In the present embodiment, a case will be described in which the divided regions divided in the matching result image are sequentially processed from left to right one row at a time. To do.
[0238]
Next, it is determined whether or not local_max (x, y) is larger than a predetermined threshold (th1) (step SP12). If it is larger, it is added as a face candidate (step SP13). As described above, the input image scale conversion determination unit 90 selects the template size of the face size that is assumed to be included in the input image together with the scale. If the face candidate is extracted by calculating the matching result image R2 for each template size, the same point may be extracted. Therefore, in step SP13, when there is the same point as the face candidate, that is, when the face candidate has already been extracted with a different template size, this point is not added.
[0239]
Next, in the input image area of the template size corresponding to the point extracted as the face candidate, the occupation rate of the skin color pixels included in this image area is obtained. In the present embodiment, the skin color table 100 is referred to when obtaining the skin color pixel occupation ratio. Then, it is determined whether or not the skin color pixel occupancy is larger than a predetermined threshold (th2) (step SP14). If it is larger, the vicinity of this local_max (x, y), for example, up, down, left and right, 8 neighboring points are added as face candidates (step SP15). Here, as in step SP13, when these eight neighboring points have already been extracted as face candidates, they are not added to the candidates.
[0240]
If local_max (x, y) is less than the threshold th1 in step SP12, if the skin color pixel occupancy in the input image corresponding to local_max (x, y) is less than the threshold th2 in step SP14, and face in step SP15 After the addition of candidates is completed, in either case, the process proceeds to step SP16, and the process proceeds to the next divided area to extract the next face candidate.
[0241]
First, in the matching result image R2, the process proceeds to the adjacent divided region shifted by the template size in the x direction, that is, wid_t (step SP16). Next, in the divided region of x coordinate (x + wid_t) shifted by wid_t, if the x coordinate is larger than the width (side in the x direction) wid_r of the matching result image, the divided region is not included in the matching result image. In the next line, 0 ≦ x ≦ wid_t−1, and the next divided region shifted by the template size in the y direction, that is, hei_t (step SP18). Next, it is determined whether or not the y coordinate of the divided area is larger than the height (side in the y direction) hei_r of the matching result image (step SP19). If it is larger, the correlation values of all the divided areas in the matching result image are determined. Indicates that the maximum value has been obtained, and the process ends.
[0242]
On the other hand, when it is determined in step SP17 and step SP18 that the divided area is included in the matching result image, the process returns to step SP11 again, and a point having the highest correlation value in the divided area is extracted.
[0243]
  In the present embodiment, the matching result image R2 is set to the template size.SeparatedSince the maximum value of the correlation value in the divided area is obtained, when moving to the adjacent divided area in step SP16, it is assumed that it is shifted by wid_t in the x direction. However, the matching result image R2 is smaller than the template size. If there is, it can be divided into arbitrary sizes. At this time, if the width (side in the x direction) wid_step and the height (y direction) hei_step of the size of the image to be divided are moved at wid_step in the x direction or hei_step in the y direction in step SP16 and step SP18, respectively. Thus, it is possible to proceed to the next divided area.
[0244]
In this way, face candidates are extracted in the template matching unit 92. The processing up to this point is executed in the arithmetic processing unit 71 and the subsequent processing is executed in the main control unit 381, so that the processing load on the main control unit 381 can be reduced accordingly. The subsequent process of extracting face data via the scale conversion and cutout unit 93, the preprocessing unit 94, and the pattern recognition unit 95 is the same as in the second application example described above.
[0245]
(3-3) Effects of the present embodiment
FIG. 16 is a diagram illustrating points detected by the template matching unit 92 as face candidates from the window image W2. In FIG. 16, the points shown in white are points extracted as face candidates from the matching result image R2 shown in FIG. As a comparison, FIG. 17 is a diagram illustrating an example in which all points that are equal to or greater than the threshold are extracted as face candidates in the matching result image R2. Compared with the diagram shown in FIG. 17, it can be seen that the number of points extracted as face candidates by the template matching unit 92 is drastically reduced in the present embodiment. As a result, the amount of calculation in the subsequent processing can be drastically reduced.
[0246]
  In the present embodiment, when matching is performed using a template representing an average face image for a window image to determine whether or not it is roughly a face image, the template matching result image is set to a predetermined size. By dividing and extracting the maximum correlation value as a face candidate and removing window images that are not clearly face images, the amount of computation and time required for the subsequent face detection processing can be reduced without missing the original face region. Thus, real-time characteristics are remarkably improved.ImprovedA face detection device and a robot device equipped with the face detection device can be realized.
[0247]
Further, the face detection accuracy can be improved by setting the face search range around the point where the correlation value is maximized. Furthermore, only when the skin color occupancy rate or the face color occupancy rate is equal to or greater than a predetermined threshold, by setting the face search range, the number of face candidates can be reduced while maintaining the face detection accuracy, and the subsequent calculation amount can be reduced. . Furthermore, the amount of calculation can be further reduced by appropriately switching the template size.
[0248]
(4) Other embodiments
In the above-described embodiment, the case where the present invention is applied to the biped robot device 200 configured as shown in FIG. 1 has been described. However, the present invention is not limited to this, and other configurations are also possible. The present invention can be widely applied to various robot devices and various other devices other than robot devices. For example, the robot apparatus may be a quadruped walking, and the moving means is not limited to the legged moving system.
[0249]
Further, as a scale conversion means for reducing and converting a frame image obtained as a result of imaging by the CCD camera (imaging means) 50 into a scale image having a plurality of different reduction ratios, among the processing functions of the main control unit 381 shown in FIG. Although the case where the input image scale conversion unit 90 is applied has been described, the present invention is not limited to this, and the point is that a plurality of scale images obtained by reducing and converting a frame image at a plurality of different reduction ratios can be obtained. Various other configurations may be applied.
[0250]
Furthermore, five kinds of scale images obtained by reducing and converting one frame image by 0.8 times are applied. However, within the range of the processing capability of the main control unit 381, 2 to 4 kinds are also available. The above may be sufficient, and the reduction ratio may be set freely. In this case, if face detection processing is performed on the basis of a plurality of scale images, the frame does not matter regardless of the size of the face actually shown in the frame image that is the imaging result (that is, the distance between the robot apparatus 200 and the human). A face image can be detected with a much higher probability than when a face image is directly detected from an image.
[0251]
In addition, an extraction unit that sequentially extracts the frame image obtained as a result of imaging by the CCD camera (imaging unit) 50 while scanning the window image of 400 pixels so as to be shifted one pixel at a time, main control unit 381 shown in FIG. In the above processing function, the window cutout unit 91 is used. However, the present invention is not limited to this, and the point is that a window image of a predetermined size is shifted by a predetermined pixel from the frame image or the scale image. As long as it can be sequentially extracted while scanning, various other configurations may be applied.
[0252]
In this case, the scanning order may be set in any way as long as the entire rectangular image can be scanned other than from the upper left position to the lower right position of the rectangular image. The number of pixels to be shifted during scanning may be one pixel or three or more, and the size of the window image may be set to a desired number of pixels having various aspect ratios other than 400 pixels.
[0253]
Furthermore, the case where the pre-processing unit 95 and the pattern identification unit 96 of the processing functions of the main control unit 381 shown in FIG. 12 are applied as identification means for identifying the face image from the window image has been described. The present invention is not limited to this, and can be widely applied to configurations using various other methods as long as a face image can be detected.
[0254]
Furthermore, the processing of the main control unit 381 shown in FIG. 12 is performed as correlation degree detection means for matching the extracted window image with a template representing an average face image and detecting the degree of correlation with the template. Although the case where the template matching unit 92 of the functions is applied has been described, the present invention is not limited to this, and the point is that various methods can be used as long as the degree of correlation can be detected based on the template. It can apply widely to the thing using the structure.
[0255]
Further, the main control unit 381 is applied as a control unit that controls the entire control of the robot apparatus 200, and the internal memory 381A is applied as a memory unit that is connected to the main control unit 381 and reads / writes information according to the control. Further, among the processing functions of the main control unit 381 shown in FIG. 12, an arithmetic processing unit 71 (DMA controller 70) is used as an arithmetic processing means for executing each processing by the input image scale conversion unit 90, the window cutout unit 91, and the template matching unit 92. However, the present invention is not limited to this, and the main control unit 381 and the arithmetic processing unit 71 selectively share the processing functions (90 to 95) of the face detection processing. If it can be switched and executed via the internal memory 381A, the shared content at that time is the amount of calculation and the calculation applied to the main control unit 381. It can be freely set to reduce the burden between. Further, the processing may be performed only by the main control unit 381.
[0256]
【The invention's effect】
As described above in detail, the face detection apparatus according to the present invention uses a frame image obtained as a result of imaging by an imaging unit as an input image in the face detection apparatus that extracts a face area of an object from an input image. And a matching result generating means for generating a matching result that is a set of correlation values obtained by correlating the template with a predetermined size representing an average face image, and obtaining a local maximum value of the correlation value from the matching result. Since the face candidate extracting means for extracting the face candidate based on the value and the identifying means for identifying the face image from the input image area corresponding to the point extracted as the face candidate, the face candidate extracting means is extremely efficient and Since face candidates can be narrowed down accurately, the amount of calculation in the face identification means in the subsequent stage is reduced, and real-time face detection is possible.
[0257]
The robot apparatus according to the present invention is an autonomous robot apparatus that performs an operation based on supplied input information. An imaging unit and a frame image obtained as a result of imaging by the imaging unit are used as an input image and a person is obtained from the input image. Face detection means for extracting a face area, and the face detection means generates a matching result that is a set of correlation values obtained by correlating the input image with a template of a predetermined size indicating an average face image. Matching result generating means, a face candidate extracting means for obtaining a local maximum value of the correlation value from the matching result, and extracting a face candidate based on the local maximum value, and an input image region corresponding to the point extracted as the face candidate Identification means for discriminating face images from the face candidates, the face candidate extraction means narrows down face candidates extremely efficiently and accurately, reducing the amount of face detection computation. And, by allowing real-time face detection, it is possible to improve the entertainment property of the robot apparatus.
[Brief description of the drawings]
FIG. 1 is a perspective view showing an external appearance of a humanoid robot apparatus according to an embodiment of the present invention as viewed from the front.
FIG. 2 is a perspective view showing an external appearance of a humanoid robot apparatus according to an embodiment of the present invention as viewed from the rear.
FIG. 3 is a schematic diagram showing a freedom degree configuration model of the humanoid robot device according to the embodiment of the present invention.
FIG. 4 is a schematic diagram showing a control system configuration of a humanoid robot apparatus according to an embodiment of the present invention.
FIG. 5 is a block diagram for explaining an internal configuration of a robot apparatus according to a first application example;
FIG. 6 is a block diagram showing face detection means of the robot apparatus in the first application example.
FIG. 7 is a schematic plan view for explaining overlap determination processing;
FIG. 8 is a block diagram for explaining an internal configuration of a robot apparatus according to a second application example.
FIG. 9 is a block diagram showing face detection means of a robot apparatus in a second application example.
FIG. 10 is a schematic diagram for explaining correlation detection by template matching.
FIG. 11 is a flowchart showing an overlap determination processing procedure in the order of steps.
FIG. 12 is a block diagram showing a face detection task function of the robot apparatus according to the embodiment of the present invention.
FIGS. 13A and 13B are schematic diagrams showing an input image (window image) and a template, respectively.
FIG. 14 is a diagram showing a matching result image that is a set of correlation values obtained from an input image (window image) and a template.
FIG. 15 is a flowchart showing processing steps in the template matching unit of the face detection means in the embodiment of the present invention.
FIG. 16 is a diagram illustrating a result of extracting face candidates from a matching result image by a template matching unit of the face detection task function according to the embodiment of the present invention.
FIG. 17 is a diagram illustrating a result of extracting a face result candidate that is equal to or higher than a predetermined threshold in a matching result image;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Robot apparatus, 381 Main control part, 381A Internal memory, 50 CCD camera, 60, 80, 90 Input image scale conversion part, 61, 81, 91 Window cutting part, 62, 82, 92 Template matching part, 63, 85 , 94 Pre-processing unit, 64, 86, 95 Pattern identification unit, 65, 83 Overlap determination unit, 70 DMA controller, 71 Arithmetic processing unit, 84, 93 Scale conversion and clipping unit, RT1 overlap determination processing procedure, S1A image signal

Claims (14)

In a face detection device that extracts a face area of an object from an input image,
Matching result generation that generates a matching result that is a set of correlation values obtained by taking a correlation between the input image and a template of a predetermined size indicating an average face image as a frame image obtained as a result of imaging by the imaging means Means,
Face candidate extraction means for obtaining a local maximum value of a correlation value from the matching result and extracting a face candidate based on the local maximum value;
Have a identification means for identifying the face region from an input image region corresponding to the face candidate in the matching result,
The matching result, which is a set of correlation values, is a set of correlation values between the template and the input image moved by shifting the input image by scanning the input image, and is arranged as the template moves. A two-dimensional array
The face candidate extraction unit divides the matching result into a plurality of areas, sets a point having at least a maximum correlation value for each divided area as a first face candidate, and points near the first face candidate. A face detection apparatus that extracts the first and second face candidates as face candidates using the first face candidate as a face candidate . The size of the divided region, a face detecting device according to claim 1, wherein Ru der below the size of the template. Face detecting device according to claim 1, wherein that having a template size determination means for determining the size of the template input from different sized template to the matching result generating means. It said template size determination means, previously detected you select a template of the face region of the same size according to claim 3 face detecting apparatus according. It said template size determination means, the distance information between the target object in the input image is input, a face detecting device according to claim 3, wherein you select the template on the basis of the distance information. The matching result generating means, the face detection device of the different size of the template is sequentially inputted according to claim 3, wherein that generates the matching result corresponding to each template. The face candidate extraction means, the one of the maximum value of the correlation values for each divided region, the face detecting apparatus according to claim 1, wherein you extract only more than a predetermined threshold value as a face candidate. The face candidate extraction means has an occupancy rate of the skin color area in the input image area of the template size corresponding to the point in the vicinity of the first face candidate among the points in the vicinity of the first face candidate being a predetermined threshold value or more. face detecting device according to claim 1, wherein you extract a certain point as the second face candidate. The face candidate extraction means has face color information learned in advance, and among the points near the first face candidate, the face color area in the input image area of the template size corresponding to the point near the first face candidate occupancy face detecting device according to claim 1, wherein you extracted points is above a predetermined threshold value as a second face candidate. In a face detection method for extracting a face area of an object from an input image,
Computer
Matching result generation that generates a matching result that is a set of correlation values obtained by taking a correlation between the input image and a template of a predetermined size indicating an average face image as a frame image obtained as a result of imaging by the imaging means Process,
A face candidate extraction step of obtaining a local maximum value of the correlation value from the matching result and extracting a face candidate based on the local maximum value;
An identification step of identifying the face area from an input image area corresponding to the face candidate in the matching result;
Run
The matching result, which is a set of correlation values, is a set of correlation values between the template and the input image moved by shifting the input image by scanning the input image, and is arranged as the template moves. A two-dimensional array
In the face candidate extraction step, the matching result is divided into a plurality of regions, a point having at least a maximum correlation value for each divided region is set as a first face candidate, and points in the vicinity of the first face candidate The face detection method for extracting the first and second face candidates as face candidates using the first face candidate as a face candidate . In the face candidate extraction step, the skin color area occupancy rate in the input image area of the template size corresponding to the point in the vicinity of the first face candidate out of the points in the vicinity of the first face candidate is greater than or equal to a predetermined threshold value. face detection method of claim 10, wherein you extract a certain point as the second face candidate. In the face candidate extraction step, the face color area in the input image area having the template size corresponding to the point near the first face candidate among the points near the first face candidate, having face color information learned in advance 10. face detection method according occupancy of that to extract points is above a predetermined threshold value as a second face candidate. In a program for causing a computer to execute an operation of extracting a human face area from an input image,
On the computer,
Matching result generation that generates a matching result that is a set of correlation values obtained by taking a correlation between the input image and a template of a predetermined size indicating an average face image as a frame image obtained as a result of imaging by the imaging means Procedure and
A face candidate extraction procedure for obtaining a local maximum value of the correlation value from the matching result and extracting a face candidate based on the local maximum value;
An identification procedure for identifying the face area from the input image area corresponding to the point extracted as the face candidate;
A program for executing
The matching result, which is a set of correlation values, is a set of correlation values between the template and the input image moved by shifting the input image by scanning the input image, and is arranged as the template moves. A two-dimensional array
In the face candidate extraction procedure, the matching result is divided into a plurality of regions, a point having at least a maximum correlation value for each divided region is set as a first face candidate, and points in the vicinity of the first face candidate A program for extracting the first and second face candidates as face candidates . In a computer-readable recording medium on which a program for causing a computer to execute an operation of extracting a human face area from an input image is recorded,
On the computer,
Matching result generation that generates a matching result that is a set of correlation values obtained by taking a correlation between the input image and a template of a predetermined size indicating an average face image as a frame image obtained as a result of imaging by the imaging means Procedure and
A face candidate extraction procedure for obtaining a local maximum value of the correlation value from the matching result and extracting a face candidate based on the local maximum value;
An identification procedure for identifying the face area from the input image area corresponding to the point extracted as the face candidate;
A recording medium recording a program for executing
The matching result, which is a set of correlation values, is a set of correlation values between the template and the input image moved by shifting the input image by scanning the input image, and is arranged as the template moves. A two-dimensional array
In the face candidate extraction procedure, the matching result is divided into a plurality of regions, and a point having at least a maximum correlation value for each divided region is set as a first face candidate, and points in the vicinity of the first face candidate A recording medium in which a program for extracting the first and second face candidates as face candidates is recorded as a second face candidate .

Images