JP2005018654A - Looking-off state detection device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a looking-off state detection device improved in detecting precision. <P>SOLUTION: This looking-off state detection device comprises a face image taking part 10, an optical flow detection part 20 and a looking-off state detection part 30. The face image taking part 10 takes an image of the face of an object person. The optical flow detection part 20 detects an optical flow from the face image taken by the face image taking part 10. At this time, the optical flow detection part 20 detects the optical flow from at least two face images continued in time series. The looking-off state detection part 30 detects the looking-off state of the object person from the optical flow detected by the optical flow detection part 20. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

所定の式は、以下に示すものである。
【００５１】
【数２】

なお、ｄ（ｘ，ｙ）は、顔画像内の任意位置の画素値であり、ｄ’（ｘ，ｙ）は変換後の画素値である。
【００５２】
その後、マイコン３は、現在の顔画像の探索領域内から、前回の顔画像内の参照領域に最も類似する位置を求めて、移動量（ｘｄ，ｙｄ）、すなわちオプティカルフローを算出する（ＳＴ１１）。
【００５３】
具体的には、マイコン３は、まず、探索領域内から参照領域に最も類似する領域を求め、最も類似する領域の中心点を、参照領域に最も類似する位置とする。そして、マイコン３は、求められた最も類似する領域の中心点と、探索領域の中心点とから移動量（ｘｄ，ｙｄ）を算出し、オプティカルフローとする。
【００５４】
ここで、ステップＳＴ１１について詳細に説明する。上述したように、顔画像上には予め参照領域が設定されている。また、探索領域は参照領域を取り囲むように設定される。また、参照領域と探索領域とは時間を異にして設定される。具体的には、図７に示すように、参照領域は時刻ｔにおいて設定され、探索領域は時刻ｔ後の時刻（ｔ＋１）において設定される。
【００５５】
図７は、図６に示すステップＳＴ１１における移動量（ｘｄ，ｙｄ）の算出方法の説明図である。ステップＳＴ１１の処理において、マイコン３は、まず、時刻（ｔ＋１）の探索領域内に候補領域を作成する。この候補領域は、参照領域と同じ大きさを有する領域である。具体的には、探索領域内の任意の位置を（ｘｄ，ｘｙ）とした場合、−（ｓｗ−ｔｗ）／２＜ｘｄ＜（ｓｗ−ｔｗ）／２、且つ、−（ｓｈ−ｔｈ）／２＜ｘｙ＜（ｓｈ−ｔｈ）の範囲から切り出した領域である。
【００５６】
マイコン３は、探索領域内の所定箇所に上記のような候補領域を設定し、設定した候補領域と参照領域とを比較等して、類似度を求める。次に、マイコン３は、候補領域を他の位置に動かし、動かした位置の候補領域と参照領域とを比較等して類似度を求める。
【００５７】
その後、マイコン３は、候補領域を順次移動させていき、探索領域内での各箇所において参照領域との類似度を算出する。類似度は、例えば、濃淡データを基準に判断される。ここで、濃淡データを基準に類似度を算出する場合において、類似度をｃｏｓθとすると、類似度は以下の式にて表される。
【００５８】
【数３】

上式においては、参照領域の濃淡データをＴとし、候補領域の濃淡データをＳとしている。また、ｘｄは、探索領域内のＸ座標値を示し、ｙｄは、探索領域内のＹ座標値を示している。
【００５９】
以上から、マイコン３は、類似度が最大となる位置Ｓを定め、点Ｓと点Ｏとの座標値の差を移動量（ｘｄ，ｙｄ）として取得し、これをオプティカルフローとする。
【００６０】
再度、図６を参照して説明する。移動量（ｘｄ，ｙｄ）の算出後、マイコン３は、類似度の範囲が閾値以上か否かを判断する（ＳＴ１２）。すなわち、マイコン３は、まず、候補領域によって探索領域内を走査していき、探索領域内の各箇所の類似度を算出する。その後、マイコン３は、得られた類似度の分散を求め、この分散により類似度の範囲が閾値以上か否かを判断する。
【００６１】
ここで、類似度の範囲が小さい場合とは、探索領域内の各箇所において、同じような類似度が検出される場合である。例えば、参照領域が真っ白な画像である場合など、特徴が少ない場合には探索領域内のどの箇所と比較しても似たような類似度の結果が得られることとなる。
【００６２】
そして、このような場合、それぞれ類似度の差が小さいことから、類似度が最大となる点Ｓの検出が不正確になりやすい。このため、図６のステップＳＴ１２の処理では、所定の閾値と比較し、好適なものと不適なものとの選別するようにしている。
【００６３】
類似度の範囲が閾値以上であると判断した場合（ＳＴ１２：ＹＥＳ）、マイコン３は、参照領域を有効な領域とし、ｆｄに「１」を代入する（ＳＴ１３）。そして、処理はステップＳＴ１５に移行する。
【００６４】
一方、類似度の範囲が閾値以上でないと判断した場合（ＳＴ１２：ＮＯ）、マイコン３は、参照領域を無効な領域とし、ｆｄに「０」を代入する（ＳＴ１４）。そして、処理はステップＳＴ１５に移行する。このように、マイコン３は、特徴量としての類似度の変化量と、予め設定される閾値とを比較することにより、動きベクトルの計算に用いるか否かを判断している。
【００６５】
ステップＳＴ１５において、マイコン３は、領域の数だけ上記のステップＳＴ１１〜ＳＴ１４を行ったか否かを判断する（ＳＴ１５）。すなわち、マイコン３は、すべての参照領域について、探索領域内から類似する位置を特定したか否かを判断している。
【００６６】
いずれかの参照領域について、探索領域内から類似する位置を特定していないと判断した場合（ＳＴ１５：ＮＯ）、処理はステップＳＴ１１に戻り、類似する位置を特定していない参照領域について、上記ステップＳＴ１１〜ＳＴ１４の処理を繰り返すこととなる。
【００６７】
一方、すべての参照領域について、探索領域内から類似する位置を特定したと判断した場合（ＳＴ１５：ＹＥＳ）、マイコン３は、図５に示すステップＳＴ６の処理を実行する。なお、参照領域が１つだけしか設定されていない場合、ステップＳＴ１５の処理を省略するようにしてもよい。
【００６８】
また、オプティカルフローの計算方法は本実施形態の他に、八木信行監修， ”ディジタル映像処理”， 映像情報メディア学会編， ｐｐ．１２９−１３９， ２０００， オーム社 などにて動画像から動きを検出する手法が複数紹介されていおり、それらを用いることもできる。
【００６９】
ここで、オプティカルフローの検出例を説明する。図８は、運転者が左方向に顔の向きを変える場合のオプティカルフローの例を示す説明図であり、（ａ）は時刻ｔにおけるオプティカルフローの例を示し、（ｂ）は時刻（ｔ＋１）におけるオプティカルフローの例を示し、（ｃ）は時刻（ｔ＋２）におけるオプティカルフローの例を示し、（ｄ）は時刻（ｔ＋３）におけるオプティカルフローの例を示している。
【００７０】
まず、図８を参照して説明する。時刻ｔにおいて運転者は前方を視認している（図８（ａ））。その後、時刻（ｔ＋１）において、運転者は交差点の確認等を行うべく、顔を左方に向ける。このとき、オプティカルフローが検出される（図８（ｂ））。ここで、画像中の四角で表示されている領域は、参照領域であり、各参照領域から伸びる線分は、各部位の移動量、すなわちオプティカルフローを示している。
【００７１】
その後、時刻（ｔ＋２）において、運転者は顔をさらに左方に向ける。このときも同様に、オプティカルフローが検出される（図８（ｃ））。そして、時刻（ｔ＋３）において運転者は顔を左上方に向けると、同様にオプティカルフローが検出される（図８（ｄ））。
【００７２】
なお、図８中において、参照領域を示す四角枠が実線にて図示されているものは、図６のステップＳＴ１２にて「ＮＯ」と判断され、無効領域とされた参照領域であり、四角枠が破線にて図示されているものは、図６のステップＳＴ１２にて「ＹＥＳ」と判断され、有効領域とされた参照領域である。
【００７３】
次に、顔の動きベクトルの検出処理（図５：ＳＴ６）を説明する。図９は、図５に示した動きベクトルの検出処理（ＳＴ６）の詳細なフローチャートである。
【００７４】
まず、マイコン３は、領域グループが複数存在する場合には、複数の領域グループのうち処理の対象となるものを選択し、その領域グループの移動量に関する数値ｘ，ｙ，ｃを「０」に初期化する（ＳＴ２０）。その後、マイコン３は、選択した領域グループ内の参照領域のうちいずれか１つを選択する。
【００７５】
そして、マイコン３は、選択した参照領域が有効領域であるか否か、すなわちｆｄが「１」であるか否かを判断する（ＳＴ２１）。ｆｄが「１」であると判断した場合（ＳＴ２１：ＹＥＳ）、マイコン３は、移動量を積算する（ＳＴ２２）。具体的に、マイコン３は、「ｘ」を「ｘ＋ｘｄ」とし、「ｙ」を「ｙ＋ｙｄ」とし、「ｃ」を「ｃ＋１」とする。そして、処理はステップＳＴ２３に移行する。
【００７６】
一方、ｆｄが「１」でないと判断した場合（ＳＴ２１：ＮＯ）、マイコン３は、移動量を積算することなく、処理はステップＳＴ２３に移行する。
【００７７】
ステップＳＴ２３において、マイコン３は、選択した領域グループ内のすべての参照領域について処理したか否かを判断する（ＳＴ２３）。いずれかの参照領域について処理をしてないと判断した場合（ＳＴ２３：ＮＯ）、処理はステップＳＴ２１に戻り、上記ステップＳＴ２１，ＳＴ２２を繰り返すこととなる。すなわち、マイコン３は、すべての参照領域について有効領域か否かを判断し、有効領域である場合には、移動量を積算するという処理を行っていく。
【００７８】
そして、順次移動量の積算等が行われ、すべての参照領域について処理した場合（ＳＴ２３：ＹＥＳ）、マイコン３は、ｃが「０」であるか否かを判断する（ＳＴ２４）。
【００７９】
「ｃ」が「０」であると判断した場合（ＳＴ２４：ＹＥＳ）、処理はステップＳＴ２６に移行する。一方、「ｃ」が「０」でないと判断した場合（ＳＴ２４：ＮＯ）、マイコン３は、積算した「ｘ」「ｙ」についての平均を求める（ＳＴ２５）。すなわち、マイコン３は、「ｘ＝ｘ／ｃ」及び「ｙ＝ｙ／ｃ」を実行し、平均移動量を求める。
【００８０】
平均移動量の算出後、マイコン３は、求めた平均移動量について、移動平均（ａｘ，ａｙ）を求める（ＳＴ２６）。ここで得られた移動平均が顔の動きベクトルとなり、移動平均のｘ方向成分ａｘが動きベクトルのｘ方向成分（画像横方向成分）となり、移動平均のｙ方向成分ａｙが動きベクトルのｙ方向成分（画像縦方向成分）となる。なお、移動平均を求める範囲は任意に定められている。また、移動平均（ａｘ，ａｙ）を求めるのは、微細なノイズ等を除去するためである。
【００８１】
ここで、動きベクトルの一例を説明すると、例えば、図８（ｂ）〜（ｄ）の画像の右隅に示すものが挙げられる。図８に示す動きベクトルは、顔画像全体に領域グループが設定されたときに得られるものであり、各画像について１つずつ得られている。同図に示すように、動きベクトルは、運転者の顔の動きを的確に反映していることがわかる。
【００８２】
ステップＳＴ２６の後、マイコン３は、すべての領域グループについて処理したか否かを判断する（ＳＴ２７）。いずれかの領域グループについて処理をしてないと判断した場合（ＳＴ２７：ＮＯ）、処理は再度ステップＳＴ２０に戻り、同様の処理を行っていくこととなる。一方、すべての領域グループについて処理したと判断した場合（ＳＴ２７：ＹＥＳ）、処理は終了する。
【００８３】
次に、脇見状態検出処理（図５：ＳＴ７）について詳細に説明する。図１０は、時系列的に得られた動きベクトルの大きさ、一方向閾値、逆方向閾値、及び脇見判断の様子を示す説明図である。図１０において縦軸は動きベクトルの大きさを示し、横軸は時刻を示している。
【００８４】
脇見状態の検出に際しては、まず、マイコン３は過去に検出された動きベクトルの時系列データに基づいて、閾値の設定を行う。このとき、実験データにより予め閾値を設定しておき、図５のステップＳＴ７にて、閾値の設定を新たに行わなくともよい。
【００８５】
同図に示すように、マイコン３は、閾値設定機能により、例えば、右方向を負とし、左方向を正とした場合、右方向閾値（一方向閾値）を負の値に設定すると共に、左方向閾値（逆方向閾値）を正の値に設定する。また、図１０に示すものと、正及び負を逆にして設定してもよい。
【００８６】
設定後、マイコン３は、顔方向判断機能により、顔の動きベクトルの大きさが右方向閾値以下の値を示したときに、被検出者の顔が右方向に動作したと判断し、左方向閾値以上の値を示したときに、被検出者の顔が左方向に動作したと判断する。
【００８７】
まず、図１０ａの期間では、右方向閾値以下及び左方向閾値以上の双方について条件を満たさない。この場合、運転者には顔の動作がないととなる。すなわち、運転者は正面を視認しているといえる。その後、図１０ｂの期間中に、顔の動きベクトルの大きさ（ａｘのみ）が右方向閾値以下の値を示す。すなわち、運転者は右方向に顔を動かしたこととなる。そして、顔の動きベクトルの大きさが右方向閾値以下の値を示した段階で、計時が行われる。
【００８８】
その後、図１０ｃの期間中に動きベクトルの大きさが「０」付近の値に戻る。すなわち、動きが検出されていないことを示しており、運転者は右方向に顔を動かしたままの状態となっている。
【００８９】
そして、図１０ｄの期間中に顔の動きベクトルの大きさ（ａｘのみ）が左方向閾値以上の値を示す。すなわち、運転者は左方向に顔を動かしたこととなる。その後、図１０ｅの期間のように、動きベクトルの大きさが右方向閾値以下及び左方向閾値以上の双方の条件を満たさないため、運転者は顔を動かしておらず、正面方向を視認しているといえる。
【００９０】
上記の場合において、時系列的に被検出者の顔の方向を判断した結果、右方向に動作したと判断してから所定時間（図１０ｆ）経過するまでに、左方向に動作したと判断されていない。すなわち、運転者は比較的長い間右方向を視認したままでいたことになる。このため、マイコン３は、所定時間経過後の期間（図１０ｇ）に、運転者は脇見をしていると検出する。
【００９１】
このようにして、本実施形態に係る脇見状態検出装置１によれば、時系列に連続する少なくとも２枚の顔画像からオプティカルフローを求めている。このオプティカルフローは、画像内の何らかの物体等に動きがあったときに、その動きを表すものである。このため、被検出者が顔を動かした場合、特定の顔部位（例えば眼や鼻や口）が撮像されない状況にあっても、オプティカルフローが得られる。そして、このオプティカルフローから顔の向きを検出することができる。
【００９２】
従って、検出精度の向上を図ることができる。
【００９３】
また、顔画像撮像部１０は、被検出者の顔全体を一画像内に収めることができる画角を有し、固定して設置されている。すなわち、顔画像撮像部１０は、一画像内に被検出者の顔が収まった状態で撮像することができる。そして、一画像内に顔が収まることで顔全体の動きに基づいて脇見状態を検出できることとなり、顔の一部だけに基づいたことによって、脇見状態の検出精度が低下してしまうことを防止することができる。従って、一層の検出精度の向上を図ることができる。
【００９４】
なお、本実施形態では、顔全体に基づいて脇見検出を行う方が望ましいというだけで、上記したように顔全体が撮像されなくとも脇見検出は可能である。そして、たとえ顔の一部に基づいて脇見状態の検出したとしても、従来装置に比べると、検出精度は向上されている。
【００９５】
また、被検出者の顔が一の方向に動作したこと、及び一の方向と逆側の方向に動作したことを、閾値を用いて判断している。そして、時系列的に顔の方向を判断し、一の方向に動作してから所定時間中に被検出者の顔が逆側の方向に向かなかったときに、脇見を検出している。
【００９６】
このように、動きの大きさと閾値とを比較しているので、被検出者の小さな動きに基づいて、顔の方向が変わったと誤認することを防止している。例えば、凹凸のある道路を車両が走行している場合などには、運転者の顔はオフセット移動（平行移動）することがある。このオフセット移動は、わずかな移動量であることが多く、このような場合に閾値判断を行わないと、脇見検出するうえで、誤認を招く結果となる。ところが、本実施形態では、閾値判断によりこのような誤認を排除することができる。従って、一層検出精度の向上を図ることができる。
【００９７】
さらには、過去の動きベクトルに基づいて閾値を設定する場合には、オフセット移動が多く検出されるときなど、閾値を高めに設定することも可能となり、より誤検出してしまうことを防止することができる。
【００９８】
また、顔画像に対して所定の大きさ及び位置で定められる少なくとも１つの参照領域からオプティカルフローを求めている。このため、顔画像のよりも小さい領域からオプティカルフローを求めていることになり、詳細な検出が行われることとなる。
【００９９】
そして、検出したオプティカルフローに基づいて、参照領域を少なくとも１つ含んで顔画像上に定められる領域グループから顔の動きベクトルを求めている。すなわち、詳細に検出したオプティカルフローに基づいて、顔の動きベクトルが求められていることとなる。
【０１００】
このため、領域グループ全体から直接にオプティカルフローを求める場合に比して、詳細に画像内の物体等の移動を検出している分、精度の高い検出を行うことができる。
【０１０１】
また、参照領域は、顔画像のサイズに基づいて、顔部位程度の大きさに設定されている。このため、大き過ぎる参照領域を設定して計算量が増大していしまうことを防止すると共に、１つの参照領域内に同時に複数の特徴的な部位が入る可能性を少なくすることができる。さらに、小さ過ぎる領域を設定して特徴的な部位がない領域となることを防ぐことができる。
【０１０２】
次に、本発明の第２実施形態を説明する。第２実施形態に係る脇見状態検出装置１ａは、第１実施形態のものとほぼ同様であるが、脇見状態検出部３０が有する機能が異なっている。より詳しくは、脇見状態検出部３０は、第１実施形態と同様の機能を有しているが、顔方向判断機能及び閾値設定機能の詳細が異なっている。
【０１０３】
以下、第１実施形態との相違点について説明する。まず、第２実施形態の脇見状態検出部３０は、顔方向判断機能により、顔の動きベクトルの累積値と、閾値とを比較する。ここで、動きベクトルの累積値は、過去から現在に掛けて、動きベクトルの大きさ及び向きを考慮して積算したものである。すなわち、累積値は、動き量及び動いた方向に基づいて積算されることとなるため、被検出者の顔の現在位置を示すものとなる。
【０１０４】
具体的に本実施形態では、累積値を以下の式により算出する。
【０１０５】
【数４】

また、累積値に基づいて顔の方向を判断するため、閾値の設定についても多少異なっている。すなわち、第２実施形態において、一方向閾値は顔が一の方向に移動したことを検出するためのものとして設定される。また、逆方向閾値に代えて、被検出者の顔が一の方向から元の位置に復帰したことを検出するための復帰方向閾値が設定される。
【０１０６】
このため、脇見状態検出部３０は、時系列的に被検出者の顔の移動を判断した結果、一の方向に移動したと判断してから所定時間経過したにもかかわらず、元の位置に復帰したと判断されなかった場合に、被検出者が脇見をしていると検出する。
【０１０７】
なお、本装置１ａについても第１実施形態のものと同様に、自動車、鉄道車両、船舶の被検出者やプラントのオペレータ等に用いることができるが、以下、本実施形態においては、車両運転者の脇見状態を検出する場合を例に説明することとする。
【０１０８】
次に、上記脇見状態検出装置１ａの動作を説明する。図１１は、第２実施形態に係る脇見状態検出装置１ａの基本動作を示すフローチャートである。なお、図１１のステップＳＴ３０〜ＳＴ３５の処理は、図５に示すＳＴ１〜ＳＴ６の処理と同様であるため、説明を省略する。
【０１０９】
ステップＳＴ３６において、マイコン３は、顔の動きベクトルの累積値を算出する（ＳＴ３６）。すなわち、動きベクトル（ａｘ，ａｙ）の値を随時累積していく。
【０１１０】
その後、マイコン３は、脇見状態検出処理を実行し、脇見状態を検出する（ＳＴ３７）。ここでの脇見検出に際しては以下の処理が行われる。まず、マイコン３は、閾値設定機能により、閾値を設定する。具体的には、被検出者の顔が一の方向（例えば右方向）に移動したことを検出するための一方向閾値と、被検出者の顔が元の位置に復帰したことを検出するための復帰方向閾値とを設定する。なお、第２実施形態においても第１実施形態と同様に、実験により得られたデータに基づいて、予め上記閾値を設定しおいてもよい。
【０１１１】
閾値設定後、マイコン３は、顔方向判断機能により、被検出者の顔の位置がどの方向に存するかを判断する。具体的には、ステップＳＴ３６にて算出された累積値と、上記一方向閾値及び復帰方向閾値とを比較して、被検出者の顔が一の方向に移動したか否か、及び元の位置に復帰したか否かを判断する。
【０１１２】
なお、この処理において、累積値と一方向閾値のみとを比較して、被検出者の顔が一の方向に移動したか否かだけを判断し、一の方向に移動したことが判断された段階で、累積値と復帰方向閾値のみとを比較するようにしてもよい。すなわち、被検出者の顔が一の方向に移動していない場合には、元の位置に復帰という概念自体生じないからである。
【０１１３】
そして、マイコン３は、顔方向判断機能によって得られる判断結果を、過去分も含めて取得し、脇見状態を検出する。すなわち、マイコン３は、過去において、被検出者の顔が一の方向に移動したと判断した場合、その判断時点からの時間が所定時間に達しているか否かを判断する。そして、所定時間に達している場合には、被検出者が脇見をしていると判断する。一方、マイコン３は、所定時間に達する前に、顔が元の位置に復帰したと判断した場合には、運転者が脇見をしていると判断しない。
【０１１４】
図１２は、時系列的に得られた動きベクトルの大きさの累積値、一方向閾値、復帰方向閾値、及び脇見判断の様子を示す説明図である。なお、図１２において縦軸は動きベクトルの累積値を示し、横軸は時刻を示している。また、図１２において画像横方向の累積値をｓｘとし、画像縦方向の累積値をｓｙとする。
【０１１５】
脇見状態の検出に際しては、まず、マイコン３は第１実施形態と同様に閾値の設定を行う。すなわち、マイコン３は、閾値設定機能により、例えば、一の方向を負とした場合、向きはじめ閾値（一方向閾値）を負の値に設定すると共に、むき終わり閾値（復帰方向閾値）を向きはじめ閾値よりも小さい値に設定する。また、正及び負を逆にして設定してもよい。
【０１１６】
設定後、マイコン３は、顔方向判断機能により、累積値が向きはじめ閾値以下の値を示したときに、被検出者の顔がいずれかの方向に移動したと判断し、向き終わり閾値以上の値を示したときに、被検出者の顔が元の位置に復帰したと判断する。
【０１１７】
まず、図１２ａの期間では、向きはじめ閾値以下という条件を満たさない。この場合、運転者は顔を移動させておらず正面方向を視認していることとなる。その後、図１２ｂの期間中に、累積値（ｓｘのみ）が向きはじめ閾値以下（具体的には−２００）の値を示す。すなわち、運転者はいずれかの方向に顔を移動させたこととなる。そして、顔の動きベクトルの累積値が向きはじめ閾値以下の値を示した段階で、計時が行われる。
【０１１８】
その後、図１２ｃの期間中に累積値は「−２００」付近の値を維持する。すなわち、復帰動作が行われていないことを示しており、運転者の顔は移動したままの状態となっている。
【０１１９】
そして、図１２ｄの期間中に累積値（ｓｘのみ）が向き終わり閾値以上の値を示す。すなわち、運転者は元の位置に顔を戻したこととなる。その後、図１２ｅの期間のように、累積値が向きはじめ閾値以下という条件を満たさないため、運転者は顔を移動させておらず正面方向を視認していることとなる。
【０１２０】
上記の場合において、時系列的に被検出者の顔の移動を判断した結果、いずれか方向に移動したと判断してから所定時間（図１２ｆ）経過するまでに、元の位置に復帰したと判断されていない。すなわち、運転者は比較的長い間正面とは異なる方向を視認したままでいたことになる。このため、マイコン３は、所定時間経過後の期間（図１２ｇ）に、運転者は脇見をしていると検出する。
【０１２１】
このようにして、本実施形態に係る脇見状態検出装置１によれば、第１実施形態と同様に検出精度の向上を図ることができ、詳細に画像内の物体等の移動を検出している分、精度の高い検出を行うことができる。また、大き過ぎる参照領域を設定して計算量が増大していしまうことを防止すると共に、１つの参照領域内に同時に複数の特徴的な部位が入る可能性を少なくすることができる。さらに、小さ過ぎる領域を設定して特徴的な部位がない領域となることを防ぐことができる。
【０１２２】
また、本実施形態によれば、累積値を得ているので、被検出者の顔の現在位置を取得することができる。このため、累積値から、浅い角度の脇見しているのか、大きく脇見をしているか等について、詳細に検出することが可能となる。従って、一層の検出精度の向上を図ることができる。（請求項５，６）
次に、本発明の第３実施形態を説明する。第３実施形態に係る脇見状態検出装置１ｂは、第２実施形態のものとほぼ同様であるが、脇見状態検出部３０が有する機能が異なっている。より詳しくは、脇見状態検出部３０は、第２実施形態と同様の機能を有しているが、閾値設定機能を備えていない点で異なっている。また、顔方向判断機能の詳細が異なっている。
【０１２３】
以下、第２実施形態との相違点について説明する。まず、第３実施形態に係る脇見状態検出部３０は、顔方向判断機能により、顔の動きベクトルの累積値と閾値との比較を行うことなく顔の方向を判断している。このため、閾値設定機能は不要となっており、顔方向判断機能のみで顔の方向を判断するようにされている。
【０１２４】
顔方向判断機能は、被検出者の顔の動きが上下左右均等に発生するという考えに基づいて顔の方向を判断している。すなわち、顔方向判断機能は、被検出者が最終的には顔を正面に戻すことから、顔の動きは上下左右均等に発生するとしている。
【０１２５】
一方、顔の動きベクトルの累積値と実際の顔の現在位置とは一致する関係にある。ところが、実際の顔の動きは複雑であることから、長く計測していくほど累積値に誤差が蓄積され、累積値は実際の顔の現在位置を示さないようになる。
【０１２６】
そこで、本実施形態における顔方向判断機能は、顔の動きが上下左右均等に発生するという考えて基づいて、誤差分の影響を緩和し、正確に顔の方向を検出するようにしている。
【０１２７】
図１３は、第３実施形態に係る脇見状態検出装置１ｂの基本動作を示すフローチャートである。なお、図１３のステップＳＴ４０〜ＳＴ４６の処理は、図１１に示すＳＴ３０〜ＳＴ３６の処理と同様であるため、説明を省略する。
【０１２８】
ステップＳＴ４６において、マイコン３は、顔の向き算出処理を実行する（ＳＴ４６）。なお、この処理は顔方向判断機能によって行われる。この処理では、まず、顔の動きが上下左右均等に発生するという考えて基づいて、誤差分の影響を緩和するための補正値を算出する。
【０１２９】
図１４は、画像横方向の顔の動きベクトル、累積値、及び補正値を示す説明図である。なお、図１４において縦軸は動きベクトル、累積値、補正値の大きさを示し、横軸は時刻を示している。
【０１３０】
同図に示すように、顔の動きベクトルに基づいて得られる累積値は、徐々にマイナス方向へシフトしている。ここで、累積値は、運転者の顔の現在位置を示すため、マイナス方向を右側とした場合、運転者の顔の位置が徐々に右方向へシフトしていることを示している。しかし、上記したように、通常、顔の動きは上下左右均等に発生するものである。このため、同図に示すシフトは、本来発生するものではなく、誤差が蓄積されたものといえる。
【０１３１】
そこで、ステップＳＴ４７の処理では、所定時間帯における累積値の移動平均を求め、その移動平均値を中立位置としている。そして、累積値と中立位置との差を顔の現在位置としている。すなわち、累積値の移動平均を求めて中立位置とすることにより、当初の中立位置であった箇所（図１４でいえば時刻が「０」の箇所）から中立位置を変更していき、累積値に誤差を含んでいたとしても正確な中立位置を求めるようにしている。そして、正確な中立位置と累積値との差により正確に顔の現在位置を判断できるようにしている。なお、本実施形態では、例えば３秒間の累積値から移動平均を得るようにしている。
【０１３２】
図１５は、補正値により補正した後の累積値を示す説明図である。なお、図１５において縦軸は累積値の大きさを示し、横軸は時刻を示している。
【０１３３】
同図に示すように、補正後の累積値は、顔の動きが上下左右均等に発生するという理論に適するものとなっている。すなわち、当初の中立位置であった箇所（図１５でいえば時刻が「０」の箇所）に対して、図１４に示すようなシフトが取り除かれた状態で累積値が示されていることがわかる。
【０１３４】
そして、同図の縦軸の正方向が運転者にとって右方向に相当する場合、正の累積値が得られているときには、運転者が右方向を向いていると判断し、負の累積値が得られているときには、運転者が左方向を向いていると判断する。このように、第３実施形態では、顔方向判断機能により補正を行って現在位置を正確に取得するため、正確に顔の方向を判断できることとなる。
【０１３５】
再度、図１３を参照して説明する。顔の方向を判断した後、マイコン３は、脇見状態の検出処理を実行する（ＳＴ４８）。すなわち、マイコン３は、運転者が正面以外の他の方向を向いたと判断すると、計時を開始する。そして、時系列的に運転者の顔の方向を判断した結果、計時開始から所定時間経過したにもかかわらず、運転者の顔が正面方向に復帰したと判断できない場合には、運転者が脇見をしていると検出する。
【０１３６】
図１６は、累積値と補正値との関係を示す説明図である。なお、図１６において縦軸は動きベクトルの累積値及び補正値を示し、横軸は時刻を示している。同時に示すように、累積値は時刻１６付近においてマイナス方向にシフトしている。このとき、補正値についても同様にマイナスの値を示している。すなわち、シフトした分を補正する値となっている。
【０１３７】
以上のように、本実施形態では、正確に顔の方向を判断して、脇見検出の精度の向上を図っている。なお、本実施形態では、閾値設定機能を有していない構成を説明したが、有していてもよい。この場合、顔の向きの判断は、図１７に示すようにして行われる。
【０１３８】
図１７は、補正後の累積値と、閾値との関係を示す説明図である。なお、図１７において縦軸は補正後の累積値を示し、横軸は時刻を示している。また、図１７では、第１実施形態と同様に右向き閾値及び左向き閾値を設定するものとする。
【０１３９】
まず、図１７ａの期間では、右向き閾値以下という条件を満たさない。この場合、運転者は正面方向を視認していることとなる。その後、図１７ｂの期間中に、累積値は、右向き閾値以下の値を示した後に、「−１５０」以下の値まで到達する。すなわち、運転者は右方向に顔を移動させたこととなる。そして、顔の動きベクトルの累積値が右向き閾値以下の値を示した段階で、計時が行われる。
【０１４０】
その後、図１７ｃの期間中に累積値は「−１５０」以下の値を維持する。すなわち、左方向への動作が行われていないことを示しており、運転者の顔は右方向へ移動したままの状態となっている。
【０１４１】
そして、図１７ｄの期間中に累積値が右向き閾値以上の値を示す。すなわち、運転者の顔は正面方向に復帰したこととなる。その後、図１７ｅの期間のように、累積値が右向き閾値以下という条件を満たさず、さらには左向き閾値以上という条件についても満たさない。このため、運転者は顔を移動させておらず正面方向を視認していることとなる。
【０１４２】
上記の場合において、時系列的に被検出者の顔の方向を判断した結果、いずれか方向に移動したと判断してから所定時間（図１７ｆ）経過するまでに、正面方向に復帰したと判断されていない。すなわち、運転者は比較的長い間正面とは異なる方向を視認したままでいたことになる。このため、マイコン３は、所定時間経過後の期間（図１７ｇ）に、運転者は脇見をしていると検出する。
【０１４３】
なお、図１７では、運転者が右を向く場合を例に説明したが、右に限らず他の方向であっても構わない。また、上記したように、本実施形態では、閾値を設定することなく、脇見検出を行ってもよい。
【０１４４】
このようにして、本実施形態に係る脇見状態検出装置１ｂによれば、第２実施形と同様に、検出精度の向上を図ることができ、詳細に画像内の物体等の移動を検出している分、精度の高い検出を行うことができる。また、大き過ぎる参照領域を設定して計算量が増大していしまうことを防止すると共に、１つの参照領域内に同時に複数の特徴的な部位が入る可能性を少なくすることができる。さらに、小さ過ぎる領域を設定して特徴的な部位がない領域となることを防ぐことができる。
【０１４５】
また、本実施形態によれば、オプティカルフローから顔の方向を判断し、顔の方向及び時間に基づいて、脇見状態を検出している。このため、一層の検出精度の向上を図ることができる。（請求項７）特に本実施形態では、顔の動きベクトルから累積値を求め、所定の演算により累積値から補正値を求め、補正値により累積値を補正して、顔の向きを判断するようにしている。これにより、顔の向きが正確に判断されることとなり、一層の検出精度の向上を図ることができる。
【図面の簡単な説明】
【図１】本発明の第１実施形態に係る脇見状態検出装置の構成図である。
【図２】参照領域及び探索領域の説明図である。
【図３】顔画像に規則的に配置される参照領域の説明図であり、（ａ）は顔画像上の横方向に参照領域が複数（例えば７つ）配置される例を示し、（ｂ）は顔画像上に格子状に参照領域が複数（例えば５行７列）配置される例を示し、（ｃ）は横方向に且つ格子状に参照領域が複数（例えば３行５列に加え、さらに横方向に２つの計１７）配置される例を示している。
【図４】本発明の第１実施形態に係る脇見状態検出装置１を車両に搭載した場合のハード構成図である。
【図５】図１に示した脇見状態検出装置１の基本動作を示すフローチャートである。
【図６】図５に示したオプティカルフロー検出処理（ステップＳＴ５）の詳細なフローチャートである。
【図７】図６に示すステップＳＴ１１における移動量（ｘｄ，ｙｄ）の算出方法の説明図である。
【図８】運転者が左方向に顔の向きを変える場合のオプティカルフローの例を示す説明図であり、（ａ）は時刻ｔにおけるオプティカルフローの例を示し、（ｂ）は時刻（ｔ＋１）におけるオプティカルフローの例を示し、（ｃ）は時刻（ｔ＋２）におけるオプティカルフローの例を示し、（ｄ）は時刻（ｔ＋３）におけるオプティカルフローの例を示している。
【図９】図５に示した動きベクトルの検出処理（ＳＴ６）の詳細なフローチャートである。
【図１０】時系列的に得られた動きベクトルの大きさ、一方向閾値、逆方向閾値、及び脇見判断の様子を示す説明図である。
【図１１】第２実施形態に係る脇見状態検出装置１ａの基本動作を示すフローチャートである。
【図１２】時系列的に得られた動きベクトルの大きさの累積値、一方向閾値、復帰方向閾値、及び脇見判断の様子を示す説明図である。
【図１３】第３実施形態に係る脇見状態検出装置１ｂの基本動作を示すフローチャートである。
【図１４】画像横方向の顔の動きベクトル、累積値、及び補正値を示す説明図である。
【図１５】補正値により補正したとき後の累積値を示す説明図である。
【図１６】累積値と補正値との関係を示す説明図である。
【図１７】補正後の累積値と、閾値との関係を示す説明図である。
【符号の説明】
１，１ａ，１ｂ…脇見状態検出装置
１０…顔画像撮像部（顔画像撮像手段）
２０…オプティカルフロー検出部（オプティカルフロー検出手段）
３０…脇見状態検出部（脇見状態検出手段）[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an armpit state detection device.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, there has been known an armpit detection device that captures a face image as a moving image, sequentially detects eye coordinates by image processing, and detects an armpit from eye movement (see Patent Document 1). Further, a method for detecting a driver's line of sight (see Patent Document 2) is known, which uses reflection of the pupil of the eye to detect the coordinates of the eye and detect a look (Patent Document 3). reference).
[0003]
As described above, in the above-described apparatus, a specific facial part called the eye is detected from the face image to estimate the driver's aside action.
[0004]
[Patent Document 1]
Japanese Patent Laid-Open No. 06-262959
[0005]
[Patent Document 2]
JP 2002-83400 A
[0006]
[Patent Document 3]
Japanese Patent Application Laid-Open No. 07-61257
[0007]
[Problems to be solved by the invention]
However, the conventional apparatus cannot detect the face orientation or the like when a face image including the face part of the driver cannot be obtained, such as when the driver meditates. When the face orientation or the like cannot be detected as described above, it is impossible to detect the driver's looking aside.
[0008]
[Means for Solving the Problems]
According to the present invention, in the side-viewing state detection device, the face image imaging unit images the face of the person to be detected, and the optical flow detection unit is continuous in time series obtained by imaging by the face image imaging unit. An optical flow is detected from at least two face images, and the look-ahead state detecting means detects a look-ahead state of the detected person from the optical flow detected by the optical flow detection means.
[0009]
【The invention's effect】
According to the present invention, the optical flow is obtained from at least two face images continuous in time series. This optical flow represents the movement of any object or the like in the image. For this reason, when the person to be detected moves his / her face, an optical flow can be obtained even when a specific face part (for example, eyes, nose, or mouth) is not captured. The direction of the face can be detected from this optical flow.
[0010]
Accordingly, detection accuracy can be improved.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the drawings.
[0012]
FIG. 1 is a configuration diagram of an armpit state detection apparatus according to the first embodiment of the present invention. As shown in the figure, the look-ahead state detection device 1 according to the present embodiment detects whether the subject is looking aside, that is, whether the subject is looking aside, and includes a face image capturing unit (face image capturing unit). ) 10. The looking-aside state detection apparatus 1 includes an optical flow detection unit (optical flow detection unit) 20 and a side-aid state detection unit (side-aid state detection unit) 30.
[0013]
The face image capturing unit 10 captures the face of the person to be detected, and is configured to send face image data obtained by the imaging to the optical flow detection unit 20. In addition, the face image capturing unit 10 is fixedly installed, and a sufficient angle of view is secured so that the entire face of the detected person can be contained in one image.
[0014]
The optical flow detection unit 20 detects an optical flow from a face image obtained by imaging by the face image imaging unit 10. The optical flow detection unit 20 detects an optical flow from at least two face images continuous in time series based on a density difference or the like.
[0015]
Here, the optical flow represents a movement amount or the like in each area in the image. Specifically, the optical flow indicates how fast and in what direction the face portion or the like existing at a predetermined location of the image has moved in the subsequent image. That is, the optical flow includes size and direction information. The optical flow is detected based on a reference area (calculation area) and a search area set at a predetermined position and size on the face image.
[0016]
FIG. 2 is an explanatory diagram of the reference area and the search area. If the reference area is set on an image at a certain time, the search area is set on an image after that time. In FIG. 2, for the sake of convenience, the width is w pixels and the height is set. This will be described on one image of h pixels.
[0017]
As shown in the figure, the reference area is an area having a width tw pixels and a height th pixels set at the position (tx, ty). The search area is an area having a width sw pixel and a height sh pixel set at the position (sx, sy). This search area is set so as to surround each reference area, and is set in the same number as the reference area. Further, although the reference area and the search area are different in size, they coincide at a specific point O with respect to the center position.
[0018]
In this way, these two regions are set to have the same center and have a relationship of sw> tw and sh> sw. Here, the reference area and the search area are set at a predetermined position and size without depending on the position of the face of the person to be detected.
[0019]
Further, it is desirable that the reference areas are regularly arranged. FIG. 3 is an explanatory diagram of reference areas regularly arranged in the face image. For example, as shown in FIG. 3A, a plurality (for example, seven) of reference areas are arranged in the horizontal direction on the face image. Also, as shown in FIG. 3B, a plurality of reference areas (for example, 5 rows and 7 columns) are arranged in a grid pattern on the face image. Furthermore, as shown in FIG. 3C, a plurality of reference regions may be arranged in the horizontal direction and in a lattice shape (for example, in addition to 3 rows and 5 columns, two in total 17 in the horizontal direction).
[0020]
Further, at least one of the reference areas is fixed in advance to the size of the face part such as the eyes, nose or mouth based on the installation position and angle of view of the face image capturing unit 10 and the size of the face image. It is desirable to be set to.
[0021]
The look-ahead state detection unit 30 detects the look-ahead state of the person to be detected from the optical flow detected by the optical flow detection unit 20, and has various functions necessary for detection.
[0022]
Specifically, the look-ahead state detection unit 30 includes a motion vector detection function (motion vector detection means), a threshold setting function (threshold setting means), and a face direction determination function (face direction determination means).
[0023]
The motion vector detection function is a function for detecting a face motion vector from the optical flow detected by the optical flow detection unit 20. Here, the motion vector is obtained for each area group according to a predetermined calculation from the optical flow, and indicates the movement of an object or the like in the area group.
[0024]
There are various kinds of predetermined calculations, which will be described later. Hereinafter, the area group will be described with reference to FIG.
[0025]
As shown in the figure, the region group is defined on the face image including one or more reference regions. Here, one optical flow is detected for each reference region. Therefore, one or more optical flows belong to the region group, and the armpit state detection unit 30 detects a motion vector based on one or more optical flows by the motion vector detection function.
[0026]
Further, the region group only needs to include one or more reference regions, and may include all of the reference regions set on the face image (FIG. 3C), or the image edge. It may include only the reference region located in the part (FIG. 3B). Furthermore, the area groups may overlap with each other (FIG. 3A).
[0027]
The threshold setting function is a function for setting a threshold for detecting the face direction of the detection subject. The threshold value here may be set in advance based on, for example, data obtained through an experiment, or may be sequentially set according to the detection state during the side-by-side detection.
[0028]
The face direction determination function is a function for determining the face motion based on both information from the motion vector detection function and the threshold setting function. Specifically, this is a function for judging the motion of the face by comparing the magnitude of the motion vector detected by the motion vector detection function with the threshold set by the threshold setting function.
[0029]
Since the above functions are provided, the look-ahead state detection unit 30 collects the determination results from the face direction determination function in time series, and detects the look-ahead state based on the time-series determination results. . Specifically, the looking-aside state is detected based on the movement direction of the face and the time of facing the direction.
[0030]
Next, an outline of the operation of the looking-ahead state detection device 1 will be described. First, in the look-ahead state detection device 1, the face image capturing unit 10 captures the face of the person to be detected and transmits the obtained face image data to the optical flow detection unit 20. Receiving this, the optical flow detection unit 20 detects an optical flow from at least two face images continuous in time series obtained by the face image capturing unit 10 based on the reference region and the search region.
[0031]
Specifically, when detecting an optical flow from a face image at time t and time (t + 1), the optical flow detection unit 20 sets a reference region on the face image at time t. The optical flow detection unit 20 sets a search area on the face image at time (t + 1).
[0032]
Thereafter, the optical flow detection unit 20 obtains the position most similar to the reference area from within the search area. Then, the difference between the position of the reference region and the position that is most similar is detected as an optical flow. For this reason, an optical flow is obtained for each reference area, and the reference area functions as an optical flow calculation area.
[0033]
After detection of the optical flow, the aside look detection unit 30 obtains a face motion vector by a motion vector detection function. Here, the look-ahead state detection unit 30 detects a motion vector for each region group according to a predetermined calculation from the optical flow.
[0034]
After detecting the motion vector, the aside look state detection unit 30 sets a threshold value by the threshold value setting function. Specifically, a one-way threshold for detecting that the face of the detected person has moved in one direction (for example, the right direction), and a direction opposite to the one direction (for example, the detected person's face) A reverse direction threshold value for detecting the movement in the left direction) is set. The looking-aside state detection unit 30 may set the threshold value in advance based on data obtained through experiments.
[0035]
After setting the threshold, the aside look detection unit 30 determines the face motion of the person to be detected by the face direction determination function. Specifically, the magnitude of the detected face motion vector is compared with the one-way threshold value and the reverse-direction threshold value to determine whether the face of the person to be detected has moved in one direction, and one direction. It is determined whether or not it has moved in the opposite direction.
[0036]
After that, when the aside look state detection unit 30 repeats the processing by the above function, acquires the determination result by the face direction determination function in time series, and determines that the face of the detected person has moved in one direction , Start timing from the time of judgment. Then, when it cannot be determined that the movement has been performed in the opposite direction after the predetermined time has elapsed, it is determined that the detected person is looking aside.
[0037]
In addition, although this apparatus 1 can be used for detection of a look-aside state, such as a to-be-detected person of a car, a railway vehicle, a ship, and an operator of a plant, hereinafter, in this embodiment, it detects a look-a-side state of a vehicle driver. This will be described as an example.
[0038]
FIG. 4 is a hardware configuration diagram when the armpit state detection device 1 according to the first embodiment of the present invention is mounted on a vehicle. As shown in the figure, a TV camera 2 is provided on the instrument of the automobile as the face image capturing unit 10. The TV camera 2 has an angle of view that can fit the entire face of the driver in one image, and is fixedly installed. Further, the TV camera 2 is installed at a position where the driver can be imaged from substantially the front.
[0039]
The input image acquired by the TV camera 2 is input as image data to a microcomputer 3 (hereinafter referred to as a microcomputer) installed inside the vehicle body such as the back side of the instrument.
[0040]
The microcomputer 3 is programmed with program logic that constitutes the optical flow detection unit 20 and the aside look state detection unit 30. Note that the program logic of the look-aside state detection unit 30 includes logic of various functions such as a motion vector detection function.
[0041]
Next, the operation of the looking-ahead state detection device 1 will be described. FIG. 5 is a flowchart showing the basic operation of the looking-ahead state detection apparatus 1 shown in FIG. As shown in the figure, first, when the process is started, the microcomputer 3 executes an initial value input process (ST1). In this initial value input process, various constants such as sampling time are read.
[0042]
After initialization, the microcomputer 3 executes an end determination process (ST2). At this time, the microcomputer 3 makes a determination based on, for example, whether the engine is activated. Then, the microcomputer 3 determines whether it is “STOP” (ST3). For example, if it is determined that the engine is not activated, the microcomputer 3 determines that it is “STOP” (ST3: YES), and the process ends.
[0043]
On the other hand, when it is determined that it is not “STOP” due to the engine being started and running (ST3: NO), the microcomputer 3 executes the entire face image capturing process (ST4). Thereby, the TV camera 2 images the face of the person to be detected. The microcomputer 3 stores face image data obtained by imaging in an image memory (not shown).
Thereafter, the microcomputer 3 executes an optical flow detection process (ST5). The process of step ST5 is a process performed by the optical flow detection unit 20 described with reference to FIG. That is, the microcomputer 3 executes a program corresponding to the optical flow detection unit 20. In this process, the microcomputer 3 stores the obtained optical flow in a memory (not shown) in time series.
[0044]
Thereafter, the microcomputer 3 executes face motion vector detection processing (ST6). The motion vector detection process is performed based on the optical flow detected in step ST5. Further, the process of step ST6 is a process performed by the motion vector detection function. That is, the microcomputer 3 executes a program corresponding to the motion vector detection function.
[0045]
After detecting the motion vector, the microcomputer 3 detects the looking-aside state based on the face motion vector (ST7). The process of step ST7 is a process performed in the aside look state detection unit 30 described with reference to FIG. That is, the microcomputer 3 executes a program corresponding to the look-ahead state detection unit 30.
[0046]
In the process of step ST7, first, the microcomputer 3 executes programs corresponding to the threshold setting function and the face direction determination function. Thereafter, the microcomputer 3 performs a look-a-side detection based on the direction and time of the face, which is the determination result obtained.
[0047]
Thereafter, the process returns to step ST2. Then, the processes in steps ST2 to ST8 are repeated until the power is turned off. That is, the movement of the face of the detected person is determined in time series.
[0048]
Next, the optical flow detection process (step ST5) will be described in detail. FIG. 6 is a detailed flowchart of the optical flow detection process (step ST5) shown in FIG.
[0049]
First, the microcomputer 3 applies a smoothing filter to the face image and converts the pixel value using a predetermined formula (ST10). Here, the smoothing filter is a filter having 5 rows and 5 columns as shown below.
[0050]
[Expression 1]

The predetermined formula is shown below.
[0051]
[Expression 2]

Note that d (x, y) is a pixel value at an arbitrary position in the face image, and d ′ (x, y) is a pixel value after conversion.
[0052]
Thereafter, the microcomputer 3 obtains the position most similar to the reference area in the previous face image from the current face image search area, and calculates the movement amount (xd, yd), that is, the optical flow (ST11). .
[0053]
Specifically, the microcomputer 3 first obtains an area most similar to the reference area from the search area, and sets the center point of the most similar area as the position most similar to the reference area. Then, the microcomputer 3 calculates an amount of movement (xd, yd) from the center point of the most similar area obtained and the center point of the search area to obtain an optical flow.
[0054]
Here, step ST11 will be described in detail. As described above, a reference area is set in advance on the face image. The search area is set so as to surround the reference area. The reference area and the search area are set at different times. Specifically, as shown in FIG. 7, the reference area is set at time t, and the search area is set at time (t + 1) after time t.
[0055]
FIG. 7 is an explanatory diagram of a method of calculating the movement amount (xd, yd) in step ST11 shown in FIG. In the process of step ST11, the microcomputer 3 first creates a candidate area within the search area at time (t + 1). This candidate area is an area having the same size as the reference area. Specifically, when an arbitrary position in the search region is (xd, xy), − (sw−tw) / 2 <xd <(sw−tw) / 2 and − (sh−th) / This is an area cut out from the range of 2 <xy <(sh−th).
[0056]
The microcomputer 3 sets a candidate area as described above at a predetermined location in the search area and compares the set candidate area with a reference area to obtain a similarity. Next, the microcomputer 3 moves the candidate area to another position and compares the candidate area at the moved position with the reference area to obtain the similarity.
[0057]
Thereafter, the microcomputer 3 sequentially moves the candidate areas and calculates the similarity to the reference area at each location in the search area. The similarity is determined based on, for example, grayscale data. Here, in the case where the similarity is calculated based on the density data, assuming that the similarity is cos θ, the similarity is expressed by the following equation.
[0058]
[Equation 3]

In the above formula, the density data of the reference area is T, and the density data of the candidate area is S. Xd represents an X coordinate value in the search area, and yd represents a Y coordinate value in the search area.
[0059]
From the above, the microcomputer 3 determines the position S at which the degree of similarity is maximum, acquires the difference between the coordinate values of the point S and the point O as the movement amount (xd, yd), and uses this as the optical flow.
[0060]
Again, a description will be given with reference to FIG. After calculating the movement amount (xd, yd), the microcomputer 3 determines whether the similarity range is equal to or greater than a threshold value (ST12). That is, the microcomputer 3 first scans the search area with the candidate area, and calculates the similarity of each part in the search area. Thereafter, the microcomputer 3 obtains the variance of the obtained similarity and determines whether or not the similarity range is equal to or greater than a threshold value based on this variance.
[0061]
Here, the case where the range of similarity is small is a case where similar similarity is detected at each location in the search area. For example, when there are few features, such as when the reference region is a pure white image, a similar similarity result is obtained even when compared with any part in the search region.
[0062]
In such a case, since the difference in the degree of similarity is small, the detection of the point S at which the degree of similarity is maximum tends to be inaccurate. For this reason, in the process of step ST12 of FIG. 6, it compares with a predetermined threshold value and sorts a suitable thing and an unsuitable thing.
[0063]
If it is determined that the similarity range is equal to or greater than the threshold (ST12: YES), the microcomputer 3 sets the reference area as an effective area and substitutes “1” for fd (ST13). Then, the process proceeds to step ST15.
[0064]
On the other hand, if it is determined that the similarity range is not equal to or greater than the threshold value (ST12: NO), the microcomputer 3 sets the reference area as an invalid area and substitutes “0” for fd (ST14). Then, the process proceeds to step ST15. As described above, the microcomputer 3 determines whether or not to use for calculation of the motion vector by comparing the amount of change of the similarity as the feature amount with the preset threshold value.
[0065]
In step ST15, the microcomputer 3 determines whether or not the above steps ST11 to ST14 have been performed by the number of areas (ST15). That is, the microcomputer 3 determines whether or not a similar position is specified from the search area for all reference areas.
[0066]
If it is determined that a similar position is not specified from within the search area for any reference area (ST15: NO), the process returns to step ST11, and the above steps are performed for the reference area that does not specify a similar position. The processing of ST11 to ST14 will be repeated.
[0067]
On the other hand, when it is determined that similar positions are specified from the search area for all the reference areas (ST15: YES), the microcomputer 3 executes the process of step ST6 shown in FIG. If only one reference area is set, the process in step ST15 may be omitted.
[0068]
In addition to this embodiment, the optical flow calculation method is supervised by Nobuyuki Yagi, “Digital Video Processing”, edited by the Institute of Image Information Media, pp. 129-139, 2000, Ohmsha, etc. have introduced a plurality of techniques for detecting motion from moving images, and these can also be used.
[0069]
Here, an example of optical flow detection will be described. FIG. 8 is an explanatory diagram showing an example of an optical flow when the driver changes the face direction to the left, (a) shows an example of the optical flow at time t, and (b) shows time (t + 1). (C) shows an example of an optical flow at time (t + 2), and (d) shows an example of an optical flow at time (t + 3).
[0070]
First, a description will be given with reference to FIG. At time t, the driver visually recognizes the front (FIG. 8A). Thereafter, at time (t + 1), the driver turns his face to the left in order to confirm the intersection. At this time, an optical flow is detected (FIG. 8B). Here, a square area in the image is a reference area, and a line segment extending from each reference area indicates a movement amount of each part, that is, an optical flow.
[0071]
Thereafter, at time (t + 2), the driver turns his face further to the left. At this time, similarly, an optical flow is detected (FIG. 8C). Then, when the driver turns his face to the upper left at the time (t + 3), the optical flow is similarly detected (FIG. 8D).
[0072]
In FIG. 8, the square frame indicating the reference area shown by the solid line is the reference area determined as “NO” in step ST12 of FIG. Is indicated by a broken line, which is a reference area determined as “YES” in step ST12 of FIG.
[0073]
Next, face motion vector detection processing (FIG. 5: ST6) will be described. FIG. 9 is a detailed flowchart of the motion vector detection process (ST6) shown in FIG.
[0074]
First, when there are a plurality of area groups, the microcomputer 3 selects a target of processing from among the plurality of area groups, and sets numerical values x, y, and c relating to the movement amounts of the area groups to “0”. Initialization is performed (ST20). Thereafter, the microcomputer 3 selects any one of the reference areas in the selected area group.
[0075]
Then, the microcomputer 3 determines whether or not the selected reference area is an effective area, that is, whether or not fd is “1” (ST21). If it is determined that fd is “1” (ST21: YES), the microcomputer 3 integrates the movement amount (ST22). Specifically, the microcomputer 3 sets “x” to “x + xd”, “y” to “y + yd”, and “c” to “c + 1”. Then, the process proceeds to step ST23.
[0076]
On the other hand, if it is determined that fd is not “1” (ST21: NO), the microcomputer 3 proceeds to step ST23 without integrating the movement amount.
[0077]
In step ST23, the microcomputer 3 determines whether or not all the reference areas in the selected area group have been processed (ST23). If it is determined that any one of the reference areas has not been processed (ST23: NO), the process returns to step ST21, and steps ST21 and ST22 are repeated. That is, the microcomputer 3 determines whether or not all the reference areas are valid areas. If the area is the valid area, the microcomputer 3 performs a process of integrating the movement amount.
[0078]
When the movement amounts are sequentially accumulated and processed for all the reference areas (ST23: YES), the microcomputer 3 determines whether c is “0” (ST24).
[0079]
If it is determined that “c” is “0” (ST24: YES), the process proceeds to step ST26. On the other hand, when it is determined that “c” is not “0” (ST24: NO), the microcomputer 3 obtains an average for the accumulated “x” and “y” (ST25). In other words, the microcomputer 3 executes “x = x / c” and “y = y / c” to obtain the average movement amount.
[0080]
After calculating the average moving amount, the microcomputer 3 calculates a moving average (ax, ay) for the calculated average moving amount (ST26). The moving average obtained here becomes the motion vector of the face, the x-direction component ax of the moving average becomes the x-direction component (image horizontal direction component) of the motion vector, and the y-direction component ay of the moving average becomes the y-direction component of the motion vector. (Image vertical direction component). In addition, the range which calculates | requires a moving average is defined arbitrarily. The reason why the moving average (ax, ay) is obtained is to remove fine noise and the like.
[0081]
Here, an example of the motion vector will be described, for example, the one shown in the right corner of the images of FIGS. 8B to 8D. The motion vector shown in FIG. 8 is obtained when an area group is set for the entire face image, and one motion vector is obtained for each image. As shown in the figure, it can be seen that the motion vector accurately reflects the movement of the driver's face.
[0082]
After step ST26, the microcomputer 3 determines whether all area groups have been processed (ST27). If it is determined that any region group is not processed (ST27: NO), the process returns to step ST20 and the same process is performed. On the other hand, if it is determined that all region groups have been processed (ST27: YES), the processing ends.
[0083]
Next, the aside look detection process (FIG. 5: ST7) will be described in detail. FIG. 10 is an explanatory diagram showing the magnitude of the motion vector obtained in time series, the one-direction threshold value, the reverse-direction threshold value, and the state of the side-by-side determination. In FIG. 10, the vertical axis indicates the magnitude of the motion vector, and the horizontal axis indicates time.
[0084]
When detecting the looking-aside state, first, the microcomputer 3 sets a threshold based on time-series data of motion vectors detected in the past. At this time, a threshold value may be set in advance using experimental data, and the threshold value may not be newly set in step ST7 of FIG.
[0085]
As shown in the figure, the microcomputer 3 sets the right threshold value (one-way threshold value) to a negative value and sets the left threshold value to a negative value when the right direction is negative and the left direction is positive. The direction threshold (reverse direction threshold) is set to a positive value. Moreover, you may set to reverse what is shown in FIG.
[0086]
After the setting, the microcomputer 3 determines that the face of the person to be detected has moved in the right direction when the size of the face motion vector indicates a value equal to or smaller than the right direction threshold by the face direction determination function, and the left direction When a value equal to or greater than the threshold value is indicated, it is determined that the face of the detected person has moved leftward.
[0087]
First, in the period of FIG. 10a, the condition is not satisfied for both the right direction threshold value and the left direction threshold value. In this case, the driver has no facial motion. That is, it can be said that the driver is viewing the front. After that, during the period of FIG. 10b, the magnitude of the face motion vector (only ax) shows a value equal to or less than the right direction threshold. That is, the driver moves his face in the right direction. Then, timing is performed when the size of the face motion vector indicates a value equal to or smaller than the right direction threshold.
[0088]
Thereafter, the magnitude of the motion vector returns to a value near “0” during the period of FIG. That is, it indicates that no movement is detected, and the driver keeps moving his face in the right direction.
[0089]
Then, during the period of FIG. 10d, the size of the face motion vector (only ax) shows a value equal to or larger than the left direction threshold. That is, the driver has moved his face to the left. Thereafter, as shown in the period of FIG. 10e, the size of the motion vector does not satisfy both the right threshold value and the left threshold value, so the driver does not move his face and looks at the front direction. It can be said that.
[0090]
In the above case, as a result of determining the face direction of the person to be detected in time series, it is determined that the face has moved leftward until a predetermined time (FIG. 10f) elapses after it is determined that the face has moved rightward. Not. That is, the driver has been viewing the right direction for a relatively long time. For this reason, the microcomputer 3 detects that the driver is looking aside during the period after the predetermined time has elapsed (FIG. 10g).
[0091]
As described above, according to the aside look detection device 1 according to the present embodiment, the optical flow is obtained from at least two face images that are continuous in time series. This optical flow represents the movement of any object or the like in the image. For this reason, when the person to be detected moves his / her face, an optical flow can be obtained even when a specific face part (for example, eyes, nose, or mouth) is not captured. The direction of the face can be detected from this optical flow.
[0092]
Accordingly, detection accuracy can be improved.
[0093]
The face image capturing unit 10 has an angle of view that can fit the entire face of the person to be detected in one image, and is fixedly installed. That is, the face image capturing unit 10 can capture an image with the face of the person to be detected within one image. When the face fits in one image, the look-ahead state can be detected based on the movement of the entire face, and the detection accuracy of the look-ahead state is prevented from being lowered due to being based on only a part of the face. be able to. Therefore, the detection accuracy can be further improved.
[0094]
In the present embodiment, it is desirable to perform the look-ahead detection based on the entire face, and the look-ahead detection is possible even if the entire face is not imaged as described above. Even if a side-by-side state is detected based on a part of the face, the detection accuracy is improved as compared with the conventional device.
[0095]
Further, it is determined using a threshold value that the face of the person to be detected has moved in one direction and has moved in a direction opposite to the one direction. Then, the direction of the face is determined in time series, and when the face of the person to be detected does not turn in the opposite direction within a predetermined time after operating in one direction, a side look is detected.
[0096]
Thus, since the magnitude of the movement is compared with the threshold value, it is possible to prevent misidentification that the face direction has changed based on the small movement of the person to be detected. For example, when a vehicle is traveling on an uneven road, the driver's face may move offset (translate). This offset movement is often a small movement amount. In such a case, if threshold judgment is not performed, a misleading result may be caused in detecting a side effect. However, in the present embodiment, such misidentification can be eliminated by threshold determination. Therefore, the detection accuracy can be further improved.
[0097]
Furthermore, when setting a threshold value based on past motion vectors, it is possible to set a higher threshold value, such as when many offset movements are detected, thereby preventing more erroneous detection. Can do.
[0098]
Further, the optical flow is obtained from at least one reference area defined by a predetermined size and position with respect to the face image. For this reason, the optical flow is obtained from an area smaller than the face image, and detailed detection is performed.
[0099]
Then, based on the detected optical flow, a face motion vector is obtained from an area group defined on the face image including at least one reference area. That is, the face motion vector is obtained based on the optical flow detected in detail.
[0100]
For this reason, compared with the case of obtaining the optical flow directly from the entire area group, it is possible to perform highly accurate detection because the movement of the object or the like in the image is detected in detail.
[0101]
In addition, the reference area is set to the size of the face part based on the size of the face image. For this reason, it is possible to prevent the calculation amount from being increased by setting a reference region that is too large, and to reduce the possibility that a plurality of characteristic parts simultaneously enter one reference region. Furthermore, an area that is too small can be set to prevent the area from having a characteristic part.
[0102]
Next, a second embodiment of the present invention will be described. The look-ahead state detection device 1a according to the second embodiment is substantially the same as that of the first embodiment, but the functions of the look-ahead state detection unit 30 are different. More specifically, the looking-aside state detection unit 30 has the same function as in the first embodiment, but the details of the face direction determination function and the threshold setting function are different.
[0103]
Hereinafter, differences from the first embodiment will be described. First, the look-ahead state detection unit 30 according to the second embodiment compares the accumulated value of the face motion vector with a threshold value using the face direction determination function. Here, the accumulated value of the motion vector is an accumulation from the past to the present in consideration of the magnitude and direction of the motion vector. That is, the accumulated value is accumulated based on the amount of movement and the direction of movement, and thus indicates the current position of the face of the person to be detected.
[0104]
Specifically, in the present embodiment, the cumulative value is calculated by the following formula.
[0105]
[Expression 4]

In addition, since the direction of the face is determined based on the accumulated value, the threshold setting is also slightly different. That is, in the second embodiment, the one-way threshold value is set to detect that the face has moved in one direction. Further, instead of the reverse direction threshold value, a return direction threshold value for detecting that the face of the person to be detected has returned to the original position from one direction is set.
[0106]
For this reason, as a result of determining the movement of the face of the person to be detected in time series, the side-view state detection unit 30 returns to the original position even though a predetermined time has elapsed since the determination that the face has been moved in one direction. If it is not determined that the object has returned, it is detected that the detected person is looking aside.
[0107]
The device 1a can also be used for a person to be detected in a car, a railway vehicle, a ship, a plant operator, etc., as in the first embodiment. Hereinafter, in this embodiment, the vehicle driver The case of detecting a look aside will be described as an example.
[0108]
Next, the operation of the looking-ahead state detection device 1a will be described. FIG. 11 is a flowchart showing the basic operation of the looking-ahead state detection device 1a according to the second embodiment. Note that the processing of steps ST30 to ST35 in FIG. 11 is the same as the processing of ST1 to ST6 shown in FIG.
[0109]
In step ST36, the microcomputer 3 calculates the cumulative value of the face motion vector (ST36). That is, the value of the motion vector (ax, ay) is accumulated at any time.
[0110]
Thereafter, the microcomputer 3 executes a look-ahead state detection process to detect a look-ahead state (ST37). In this case, the following processing is performed when detecting a look-aside. First, the microcomputer 3 sets a threshold value by a threshold value setting function. Specifically, a one-way threshold for detecting that the face of the detected person has moved in one direction (for example, the right direction) and for detecting that the face of the detected person has returned to the original position. Set the return direction threshold value. Note that, in the second embodiment as well, the threshold value may be set in advance based on data obtained through experiments, as in the first embodiment.
[0111]
After setting the threshold value, the microcomputer 3 determines in which direction the face position of the person to be detected exists using the face direction determination function. Specifically, the cumulative value calculated in step ST36 is compared with the one-direction threshold value and the return-direction threshold value to determine whether the face of the person to be detected has moved in one direction and the original position. It is determined whether or not it has returned to.
[0112]
In this process, the cumulative value and only the one-way threshold are compared to determine whether or not the face of the person to be detected has moved in one direction, and it has been determined that the face has moved in one direction. At the stage, the accumulated value and only the return direction threshold value may be compared. That is, when the face of the person to be detected has not moved in one direction, the concept of returning to the original position does not occur.
[0113]
Then, the microcomputer 3 acquires the determination result obtained by the face direction determination function, including the past, and detects the looking-aside state. That is, when it is determined that the face of the person to be detected has moved in one direction in the past, the microcomputer 3 determines whether or not the time from the determination time has reached a predetermined time. When the predetermined time has been reached, it is determined that the detected person is looking aside. On the other hand, if the microcomputer 3 determines that the face has returned to the original position before reaching the predetermined time, the microcomputer 3 does not determine that the driver is looking aside.
[0114]
FIG. 12 is an explanatory diagram showing a cumulative value of the magnitude of motion vectors obtained in time series, a one-direction threshold value, a return direction threshold value, and a state of looking aside. In FIG. 12, the vertical axis indicates the accumulated value of motion vectors, and the horizontal axis indicates time. In FIG. 12, the cumulative value in the horizontal direction of the image is sx, and the cumulative value in the vertical direction of the image is sy.
[0115]
When detecting the looking-aside state, first, the microcomputer 3 sets a threshold value as in the first embodiment. That is, the microcomputer 3 sets the direction threshold (one direction threshold) to a negative value and starts to set the peel end threshold (return direction threshold) when the direction is negative, for example. Set to a value smaller than the threshold. Moreover, you may set by making positive and negative reverse.
[0116]
After the setting, the microcomputer 3 determines that the face of the person to be detected has moved in any direction when the cumulative value starts to turn and shows a value equal to or smaller than the threshold value by the face direction determination function, and is greater than or equal to the direction end threshold value. When the value is indicated, it is determined that the face of the person to be detected has returned to the original position.
[0117]
First, in the period of FIG. 12a, the condition that the orientation is not more than the threshold value is not satisfied. In this case, the driver does not move his face and is viewing the front direction. After that, during the period of FIG. 12b, the accumulated value (only sx) starts to turn and shows a value equal to or less than the threshold (specifically −200). That is, the driver moves his / her face in either direction. Then, timing is performed when the cumulative value of the facial motion vector starts to turn and indicates a value equal to or smaller than the threshold value.
[0118]
Thereafter, the accumulated value maintains a value in the vicinity of “−200” during the period of FIG. That is, it indicates that the return operation is not performed, and the driver's face remains moving.
[0119]
The accumulated value (only sx) during the period of FIG. That is, the driver returns his face to the original position. Thereafter, as in the period of FIG. 12e, the cumulative value does not satisfy the condition that the cumulative value starts and is equal to or less than the threshold value, and thus the driver does not move the face and is viewing the front direction.
[0120]
In the above case, as a result of determining the movement of the face of the person to be detected in time series, it is assumed that the face has returned to the original position until a predetermined time (FIG. 12f) has elapsed since it was determined that the face has moved in either direction Not judged. That is, the driver has been visually recognizing a direction different from the front for a relatively long time. For this reason, the microcomputer 3 detects that the driver is looking aside during a period after the predetermined time has elapsed (FIG. 12g).
[0121]
In this way, according to the aside look detection device 1 according to the present embodiment, the detection accuracy can be improved as in the first embodiment, and the movement of an object or the like in the image is detected in detail. Therefore, highly accurate detection can be performed. In addition, it is possible to prevent the calculation amount from increasing by setting a reference area that is too large, and to reduce the possibility that a plurality of characteristic parts simultaneously enter one reference area. Furthermore, an area that is too small can be set to prevent the area from having a characteristic part.
[0122]
Further, according to the present embodiment, since the accumulated value is obtained, the current position of the face of the detected person can be acquired. For this reason, it is possible to detect in detail from the cumulative value whether the person is looking aside at a shallow angle or is looking aside. Therefore, the detection accuracy can be further improved. (Claims 5 and 6)
Next, a third embodiment of the present invention will be described. The look-ahead state detection device 1b according to the third embodiment is substantially the same as that of the second embodiment, but the functions of the look-ahead state detection unit 30 are different. More specifically, the looking-ahead state detection unit 30 has the same function as that of the second embodiment, but differs in that it does not have a threshold setting function. The details of the face direction determination function are different.
[0123]
Hereinafter, differences from the second embodiment will be described. First, the look-ahead state detection unit 30 according to the third embodiment determines the face direction without performing a comparison between the accumulated value of the face motion vector and the threshold by the face direction determination function. For this reason, the threshold setting function is unnecessary, and the face direction is determined only by the face direction determination function.
[0124]
The face direction determination function determines the face direction based on the idea that the movement of the face of the person to be detected occurs evenly in the vertical and horizontal directions. That is, the face direction determination function assumes that the movement of the face occurs evenly in the vertical and horizontal directions because the detected person finally returns the face to the front.
[0125]
On the other hand, the cumulative value of the face motion vector and the actual current position of the face are in agreement. However, since the actual movement of the face is complicated, an error is accumulated in the accumulated value as the measurement is made longer, and the accumulated value does not indicate the current position of the actual face.
[0126]
Therefore, the face direction determination function in the present embodiment reduces the influence of the error and accurately detects the face direction based on the idea that the movement of the face occurs vertically and horizontally.
[0127]
FIG. 13 is a flowchart showing the basic operation of the looking-ahead state detection device 1b according to the third embodiment. The processes in steps ST40 to ST46 in FIG. 13 are the same as the processes in ST30 to ST36 shown in FIG.
[0128]
In step ST46, the microcomputer 3 executes face orientation calculation processing (ST46). This process is performed by the face direction determination function. In this process, first, a correction value for reducing the influence of the error is calculated based on the idea that the movement of the face occurs vertically and horizontally.
[0129]
FIG. 14 is an explanatory diagram illustrating a face motion vector, a cumulative value, and a correction value in the horizontal direction of the image. In FIG. 14, the vertical axis indicates the magnitude of the motion vector, the accumulated value, and the correction value, and the horizontal axis indicates the time.
[0130]
As shown in the figure, the cumulative value obtained based on the face motion vector is gradually shifted in the negative direction. Here, since the accumulated value indicates the current position of the driver's face, it indicates that the position of the driver's face is gradually shifted to the right when the minus direction is the right side. However, as described above, the movement of the face usually occurs evenly in the vertical and horizontal directions. For this reason, the shift shown in the figure does not occur originally, and it can be said that errors are accumulated.
[0131]
Therefore, in the process of step ST47, a moving average of accumulated values in a predetermined time zone is obtained, and the moving average value is set as a neutral position. The difference between the accumulated value and the neutral position is used as the current face position. That is, by obtaining the moving average of the accumulated values and setting it as the neutral position, the neutral position is changed from the position that was the initial neutral position (the position where the time is “0” in FIG. 14). Even if an error is included in the position, an accurate neutral position is obtained. The current position of the face can be accurately determined from the difference between the accurate neutral position and the accumulated value. In this embodiment, for example, a moving average is obtained from an accumulated value for 3 seconds.
[0132]
FIG. 15 is an explanatory diagram showing the accumulated value after correction by the correction value. In FIG. 15, the vertical axis indicates the magnitude of the accumulated value, and the horizontal axis indicates time.
[0133]
As shown in the figure, the corrected cumulative value is suitable for the theory that the movement of the face occurs evenly in the vertical and horizontal directions. That is, the accumulated value is shown in a state where the shift as shown in FIG. 14 is removed from the place where the neutral position was originally (the place where the time is “0” in FIG. 15). Recognize.
[0134]
When the positive direction of the vertical axis in the figure corresponds to the right direction for the driver, when a positive cumulative value is obtained, it is determined that the driver is facing the right direction, and the negative cumulative value is When it is obtained, it is determined that the driver is facing left. As described above, in the third embodiment, correction is performed by the face direction determination function to accurately acquire the current position, so that the face direction can be accurately determined.
[0135]
Again, a description will be given with reference to FIG. After determining the direction of the face, the microcomputer 3 performs a look-a-side state detection process (ST48). That is, when the microcomputer 3 determines that the driver is facing in a direction other than the front, the microcomputer 3 starts measuring time. When the driver's face direction is determined in chronological order and the driver's face cannot be determined to have returned to the front direction even though a predetermined time has elapsed from the start of timing, the driver looks aside. Detect that you are doing.
[0136]
FIG. 16 is an explanatory diagram showing the relationship between the accumulated value and the correction value. In FIG. 16, the vertical axis indicates the accumulated value and correction value of the motion vector, and the horizontal axis indicates time. As shown at the same time, the accumulated value shifts in the minus direction around time 16. At this time, the correction value also shows a negative value. That is, it is a value for correcting the shifted amount.
[0137]
As described above, according to the present embodiment, the direction of the face is accurately determined to improve the accuracy of looking aside. In this embodiment, the configuration that does not have the threshold setting function has been described. In this case, the face orientation is determined as shown in FIG.
[0138]
FIG. 17 is an explanatory diagram showing the relationship between the cumulative value after correction and the threshold value. In FIG. 17, the vertical axis indicates the accumulated value after correction, and the horizontal axis indicates time. In FIG. 17, the right threshold and the left threshold are set as in the first embodiment.
[0139]
First, in the period shown in FIG. In this case, the driver is viewing the front direction. Thereafter, during the period of FIG. 17B, the accumulated value reaches a value of “−150” or less after showing a value of a rightward threshold or less. That is, the driver moves his / her face in the right direction. Then, timing is performed when the cumulative value of the facial motion vector shows a value equal to or less than the rightward threshold.
[0140]
Thereafter, the accumulated value maintains a value of “−150” or less during the period of FIG. That is, it indicates that the operation in the left direction is not performed, and the face of the driver remains in the right direction.
[0141]
Then, during the period shown in FIG. That is, the driver's face has returned to the front direction. Thereafter, as in the period of FIG. 17e, the condition that the accumulated value is not more than the rightward threshold is not satisfied, and further, the condition that the accumulated value is not less than the leftward threshold is not satisfied. For this reason, the driver does not move his / her face and is viewing the front direction.
[0142]
In the above case, as a result of determining the face direction of the detected person in time series, it is determined that the face has returned to the front direction after a predetermined time (FIG. 17f) has elapsed since it was determined that the face has moved in any direction. It has not been. That is, the driver has been visually recognizing a direction different from the front for a relatively long time. Therefore, the microcomputer 3 detects that the driver is looking aside during a period after the predetermined time has elapsed (FIG. 17g).
[0143]
In FIG. 17, the case where the driver turns to the right has been described as an example, but the direction is not limited to the right and may be in another direction. In addition, as described above, in the present embodiment, aside detection may be performed without setting a threshold value.
[0144]
As described above, according to the aside look detection device 1b according to the present embodiment, the detection accuracy can be improved as in the second embodiment, and the movement of an object or the like in the image can be detected in detail. Therefore, highly accurate detection can be performed. In addition, it is possible to prevent the calculation amount from increasing by setting a reference area that is too large, and to reduce the possibility that a plurality of characteristic parts simultaneously enter one reference area. Furthermore, an area that is too small can be set to prevent the area from having a characteristic part.
[0145]
Further, according to the present embodiment, the face direction is determined from the optical flow, and the looking-aside state is detected based on the face direction and time. For this reason, it is possible to further improve the detection accuracy. (Claim 7) In the present embodiment, in particular, the cumulative value is obtained from the face motion vector, the correction value is obtained from the cumulative value by a predetermined calculation, and the cumulative value is corrected by the correction value to determine the face orientation. I have to. As a result, the orientation of the face is accurately determined, and the detection accuracy can be further improved.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of an armpit state detection apparatus according to a first embodiment of the present invention.
FIG. 2 is an explanatory diagram of a reference area and a search area.
FIG. 3 is an explanatory diagram of reference areas regularly arranged in a face image, where (a) shows an example in which a plurality (for example, seven) reference areas are arranged in the horizontal direction on the face image; ) Shows an example in which a plurality of reference areas (for example, 5 rows and 7 columns) are arranged in a grid pattern on the face image, and (c) shows a plurality of reference areas in a horizontal direction and a grid pattern (for example, in addition to 3 rows and 5 columns) Further, an example in which two in total 17) are arranged in the horizontal direction is shown.
FIG. 4 is a hardware configuration diagram when the armpit state detection device 1 according to the first embodiment of the present invention is mounted on a vehicle.
FIG. 5 is a flowchart showing a basic operation of the looking-ahead state detection device 1 shown in FIG. 1;
6 is a detailed flowchart of the optical flow detection process (step ST5) shown in FIG.
7 is an explanatory diagram of a method of calculating a movement amount (xd, yd) in step ST11 shown in FIG.
FIG. 8 is an explanatory diagram showing an example of an optical flow when the driver changes the face direction to the left, (a) shows an example of an optical flow at time t, and (b) shows time (t + 1). (C) shows an example of an optical flow at time (t + 2), and (d) shows an example of an optical flow at time (t + 3).
FIG. 9 is a detailed flowchart of the motion vector detection process (ST6) shown in FIG.
FIG. 10 is an explanatory diagram showing the magnitude of a motion vector obtained in time series, a one-way threshold value, a reverse-direction threshold value, and a state of looking aside.
FIG. 11 is a flowchart showing a basic operation of an armpit state detection apparatus 1a according to the second embodiment.
FIG. 12 is an explanatory diagram showing a cumulative value of a motion vector obtained in time series, a one-way threshold value, a return direction threshold value, and a state of looking aside.
FIG. 13 is a flowchart showing a basic operation of an armpit state detection apparatus 1b according to the third embodiment.
FIG. 14 is an explanatory diagram showing a face motion vector, a cumulative value, and a correction value in the horizontal direction of the image.
FIG. 15 is an explanatory diagram showing cumulative values after correction by correction values.
FIG. 16 is an explanatory diagram showing a relationship between an accumulated value and a correction value.
FIG. 17 is an explanatory diagram illustrating a relationship between a cumulative value after correction and a threshold value;
[Explanation of symbols]
1, 1a, 1b.
10. Face image capturing unit (face image capturing means)
20: Optical flow detection unit (optical flow detection means)
30 .. A look-a-side state detection unit

Claims (10)

Face image capturing means for capturing the face of the detected person;
Optical flow detection means for detecting optical flow from at least two face images continuous in time series obtained by imaging by the face image imaging means;
A look-a-side state detection means for detecting a look-ahead state of the person to be detected from the optical flow detected by the optical flow detection means;
A device for detecting a state of looking aside. 2. The look-ahead state detection according to claim 1, wherein the face image capturing unit is fixedly installed and has an angle of view sufficient to fit the entire face of the person to be detected in one image. apparatus. The aside look detection means
Motion vector detection means for detecting a face motion vector from the optical flow detected by the optical flow detection means;
Based on experimental data or time-series data of face motion vectors detected in the past, a one-way threshold for detecting that the face of the detected person has moved in one direction, and the face of the detected person Threshold setting means for setting a reverse direction threshold for detecting the movement in the direction opposite to the one direction;
The one-way threshold value and the reverse-direction threshold value set by the threshold value setting means are compared with the magnitude of the face motion vector detected by the motion vector detection means, and the face of the person to be detected is in the one direction And face direction determining means for determining whether or not the face of the person to be detected has moved in the opposite direction,
As a result of determining the face direction of the person to be detected in time series by the face direction determination means, the movement is performed in the opposite direction even though a predetermined time has elapsed since it was determined that the movement was performed in the one direction. If it is not determined that the subject has detected that the person to be detected is looking aside, it is detected that the person is looking aside. The threshold setting means sets the one direction threshold to a positive value and sets the reverse direction threshold to a negative value when the one direction is positive and the opposite direction is negative.
The face direction determining means moves the face of the detected person in the one direction when the size of the face motion vector detected by the motion vector detecting means indicates a value equal to or greater than the one-direction threshold. 4, it is determined that the face of the person to be detected has moved in the reverse direction when the value is equal to or less than the reverse direction threshold value. The aside look detection means
Motion vector detection means for detecting a face motion vector from the optical flow detected by the optical flow detection means;
Based on experimental data or time-series data of face motion vectors detected in the past, a one-way threshold for detecting that the face of the detected person has moved in one direction, and the face of the detected person A threshold value setting means for setting a return direction threshold value for detecting the return to the original position from the one direction;
The one-way threshold value and the return direction threshold value set by the threshold value setting means are compared with the cumulative value of the motion vector of the face detected by the motion vector detection means, and the face of the person to be detected Face direction determination means for determining whether or not the face of the person to be detected has returned to its original position, and
As a result of determining the movement of the face of the person to be detected in time series by the face direction determination means, it is determined that the face has returned to the original position even though a predetermined time has elapsed since it was determined that the face has moved in the one direction. If the determination is not made, it is detected that the person to be detected is looking aside. The aside state detection apparatus according to claim 1, wherein the detection is performed. When the one direction is positive, the threshold setting means sets the one-direction threshold to a positive value, sets the return direction threshold to a value larger than the one-way threshold,
The face direction determination means moves the face of the person to be detected in the one direction when the cumulative value of the face motion vector detected by the motion vector detection means indicates a value equal to or greater than the one-direction threshold. The aside look state detection apparatus according to claim 5, wherein the detection target face is determined to have returned to the original position when the value is equal to or less than the return direction threshold value. The aside look detection means
Motion vector detection means for detecting a face motion vector from the optical flow detected by the optical flow detection means;
Based on the motion vector of the face detected by the motion vector detection means, whether or not the face of the detected person is directed from the front direction to another direction, and the face of the detected person is moved from the other direction to the front direction. And face direction determination means for determining whether or not the vehicle has returned.
As a result of determining the face direction of the person to be detected in time series by the face direction determining means, it is determined that the face has returned to the front direction even though a predetermined time has elapsed since it was determined that the face was directed in the other direction. 3. If not detected, it is detected that the person to be detected is looking aside. The optical flow detection means obtains an optical flow from at least one calculation area defined by a predetermined size and position with respect to the face image,
The motion vector detection means obtains a face motion vector from a group of areas defined on a face image including at least one of the calculation areas, based on the optical flow detected by the optical flow detection means. The looking-ahead state detection device according to any one of claims 3 to 7. The at least one of the calculation areas is set in advance to a size of about a face part based on a size of a face image obtained by imaging by the face image imaging unit. Armpit state detection device. An optical flow is detected from at least two face images obtained by continuously capturing the face of the detected person in time series, and the side-by-side state of the detected person is detected from the optical flow. Armpit state detection device.

Images