JP6785723B2 - Line-of-sight measuring device - Google Patents
Line-of-sight measuring device Download PDFInfo
- Publication number
- JP6785723B2 JP6785723B2 JP2017114762A JP2017114762A JP6785723B2 JP 6785723 B2 JP6785723 B2 JP 6785723B2 JP 2017114762 A JP2017114762 A JP 2017114762A JP 2017114762 A JP2017114762 A JP 2017114762A JP 6785723 B2 JP6785723 B2 JP 6785723B2
- Authority
- JP
- Japan
- Prior art keywords
- image
- center
- line
- corneal
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Eye Examination Apparatus (AREA)
Description
本発明は、視線計測装置に係り、特に、顔を撮像した画像から、視線ベクトルを計測する視線計測装置に関する。 The present invention relates to a line-of-sight measuring device, and more particularly to a line-of-sight measuring device that measures a line-of-sight vector from an image of a face.
従来より、３次元眼球モデルを用いた角膜反射像方式の視線計測装置が知られている（非特許文献１、２）。この角膜反射像方式の視線計測装置では、カメラと光源の位置を離して設置している。 Conventionally, a corneal reflex image type eye-gaze measuring device using a three-dimensional eyeball model has been known (Non-Patent Documents 1 and 2). In this corneal reflection image type eye-gaze measuring device, the camera and the light source are installed at a distance from each other.
非特許文献１では、角膜曲率中心座標を求めるために、カメラと光源との間の距離、角膜曲率中心から角膜までの距離、及びカメラと目との間の距離に応じた近似計算を行っている。 In Non-Patent Document 1, in order to obtain the coordinates of the center of curvature of the cornea, approximate calculation is performed according to the distance between the camera and the light source, the distance from the center of curvature of the cornea to the cornea, and the distance between the camera and the eye. There is.
非特許文献２では、複数の光源を用いて角膜反射像を取得している。 In Non-Patent Document 2, a corneal reflex image is acquired by using a plurality of light sources.
上記非特許文献１に記載の技術では、近似計算を行っているため、近似計算による誤差が生じ、視線計測の精度が低下してしまう。 In the technique described in Non-Patent Document 1, since the approximate calculation is performed, an error occurs due to the approximate calculation, and the accuracy of the line-of-sight measurement is lowered.
また、上記非特許文献２に記載の技術では、角膜反射像を得るために、複数の光源が必要となるため、装置コストがかかってしまう。 In addition, the technique described in Non-Patent Document 2 requires a plurality of light sources in order to obtain a corneal reflex image, which increases the cost of the device.
本発明は、上記の問題点を解決するためになされたもので、簡易な構成で、近似計算を行うことなく、精度よく視線計測を行うことができる視線計測装置を提供することを目的とする。 The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a line-of-sight measurement device capable of performing line-of-sight measurement with high accuracy without performing approximate calculation with a simple configuration. ..
上記の目的を達成するために本発明に係る視線計測装置は、被観察者の顔を撮像する撮像手段と、前記被観察者の目に対して光を照射する光照射手段と、前記撮像手段によって撮像された前記顔を表す顔画像から、前記顔画像上の前記顔の目の角膜反射像と、前記顔画像上の前記顔の目の瞳孔中心位置と、予め定められた３次元眼球モデルとに基づいて、カメラ座標系における３次元の視線ベクトルを計算する視線ベクトル計算手段と、を含み、前記撮像手段と前記光照射手段との位置関係、前記撮像手段と前記目との位置関係、及び前記撮像手段に関するパラメータが、前記撮像手段の撮像方向と前記光照射手段の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たす。 In order to achieve the above object, the line-of-sight measuring device according to the present invention includes an imaging means for photographing the face of the observed person, a light irradiating means for irradiating the eyes of the observed person with light, and the imaging means. From the face image representing the face imaged by, the corneal reflex image of the eye of the face on the face image, the pupil center position of the eye of the face on the face image, and a predetermined three-dimensional eyeball model. A line-of-sight vector calculating means for calculating a three-dimensional line-of-sight vector in a camera coordinate system based on the above, a positional relationship between the imaging means and the light irradiation means, a positional relationship between the imaging means and the eyes, and the like. And the parameters relating to the imaging means satisfy a predetermined constraint condition for considering that the imaging direction of the imaging means and the light irradiation direction of the light irradiation means are coaxial.
本発明によれば、前記撮像手段と前記光照射手段との位置関係、前記撮像手段と前記目との位置関係、及び前記撮像手段に関するパラメータが、前記撮像手段の撮像方向と前記光照射手段の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たす。 According to the present invention, the positional relationship between the imaging means and the light irradiation means, the positional relationship between the imaging means and the eyes, and the parameters related to the imaging means are the imaging direction of the imaging means and the light irradiation means. It satisfies a predetermined constraint condition for considering that the light irradiation direction is coaxial.
また、前記被観察者の目に対して光照射手段から光を照射しているときに、撮像手段によって、前記顔を表す顔画像を撮像する。視線ベクトル計算手段によって、顔画像から、顔画像上の前記顔の目の角膜反射像と、前記顔画像上の前記顔の目の瞳孔中心位置と、予め定められた３次元眼球モデルとに基づいて、カメラ座標系における３次元の視線ベクトルを計算する。 Further, when the eyes of the observed person are irradiated with light from the light irradiating means, the imaging means captures a face image representing the face. Based on the face image, the corneal reflection image of the face eye on the face image, the pupil center position of the face eye on the face image, and a predetermined three-dimensional eyeball model by the line-of-sight vector calculation means. Then, the three-dimensional line-of-sight vector in the camera coordinate system is calculated.
以上説明したように、本発明の視線計測装置によれば、前記撮像手段の撮像方向と前記光照射手段の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たすことにより、簡易な構成で、近似計算を行うことなく、精度よく視線計測を行うことができる、という効果が得られる。 As described above, according to the line-of-sight measuring device of the present invention, by satisfying a predetermined constraint condition for considering that the imaging direction of the imaging means and the light irradiation direction of the light irradiation means are coaxial. With a simple configuration, it is possible to obtain an effect that the line-of-sight measurement can be performed accurately without performing approximate calculation.
以下、図面を参照して本発明の実施の形態を詳細に説明する。なお、本実施の形態では、撮像された顔画像から、視線ベクトルを推定する視線計測装置に本発明を適用した場合を例に説明する。 Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case where the present invention is applied to a line-of-sight measuring device that estimates a line-of-sight vector from an captured face image will be described as an example.
＜本発明の実施の形態の概要＞
カメラと光源が同軸であれば、角膜曲率中心座標を求めるのに近似計算が不要で精度が向上する。
<Outline of Embodiment of the present invention>
If the camera and the light source are coaxial, approximate calculation is not required to obtain the coordinates of the center of curvature of the cornea, and the accuracy is improved.
一方、カメラと光源が同軸でない場合、近似計算を行わずに、厳密計算を行うことができるが、その場合の従来技術では、複数の光源を必要としている。本発明の実施の形態では、光源は１箇所とする。 On the other hand, when the camera and the light source are not coaxial, the exact calculation can be performed without performing the approximate calculation, but the prior art in that case requires a plurality of light sources. In the embodiment of the present invention, there is only one light source.
カメラと光源が同軸でない場合でも、カメラと光源と撮像対象の位置関係、およびカメラのピクセル単位で表わされる焦点距離ｆがある制約条件を満たせば、実用的には同軸として扱える。 Even if the camera and the light source are not coaxial, they can be practically treated as coaxial if the positional relationship between the camera, the light source, and the image pickup target and the focal length f expressed in pixel units of the camera satisfy certain constraints.
そこで、本発明の実施の形態では、光源と撮像部と撮像対象の位置関係が、同軸とみたすための制約条件を満たす光学系を用いる。 Therefore, in the embodiment of the present invention, an optical system is used in which the positional relationship between the light source, the imaging unit, and the imaging target satisfies the constraint condition that the positional relationship is considered to be coaxial.
＜視線計測装置の構成＞
図１に示すように、本発明の実施の形態に係る視線計測装置１０は、対象とする被験者の顔を含む画像を撮像するＣＣＤカメラ等からなる画像撮像部１２と、被験者の目に対して光を照射する照射部１３と、画像処理を行うコンピュータ１４と、ＣＲＴ等で構成された出力部１６とを備えている。
<Configuration of line-of-sight measuring device>
As shown in FIG. 1, the line-of-sight measuring device 10 according to the embodiment of the present invention refers to an image imaging unit 12 including a CCD camera or the like that captures an image including the face of the subject, and the eyes of the subject. It includes an irradiation unit 13 that irradiates light, a computer 14 that performs image processing, and an output unit 16 composed of a CRT or the like.
画像撮像部１２は、１つのカメラであり、照射部１３は、例えば、１つの近赤外ＬＥＤである。本実施の形態では、画像撮像部１２の撮像方向と照射部１３の照射方向とが同軸ではないが、同軸とみなされるように、以下の（１）式に示す制約条件を満たす配置となっている（図２）。 The image capturing unit 12 is one camera, and the irradiation unit 13 is, for example, one near infrared LED. In the present embodiment, the imaging direction of the image capturing unit 12 and the irradiation direction of the irradiation unit 13 are not coaxial, but are arranged so as to satisfy the constraint condition shown in the following equation (1) so as to be regarded as coaxial. (Fig. 2).
・・・（１）
... (1)
ただし、Ｌは、画像撮像部１２から角膜曲率中心へ向かう直線と角膜との交点と、画像撮像部１２との間の距離であり、ｒは、角膜曲率半径であり、ｆは、画像撮像部１２のピクセル単位の焦点距離である。 However, L is the distance between the intersection of the straight line from the image imaging unit 12 toward the center of curvature of the cornea and the cornea and the image imaging unit 12, r is the radius of curvature of the cornea, and f is the image imaging unit. The focal length is in units of 12 pixels.
上記（１）式の導出は、以下の通りである。 The derivation of the above equation (1) is as follows.
図２において、画像撮像部１２の両側に配置した照明Ｓ1とＳ2による２つの角膜反射像中心点Ｐ1とＰ2の距離がカメラ画像で1pixel以下であれば、角膜反射像中心点Ｐ1と角膜反射像中心点Ｐ2は同じ画素上もしくは隣り合う画素上で観測されることになる。これは、角膜表面に映る２つの照明を区別できないということであり、画像撮像部１２からは２つの照明が同じ場所にあるとみなせるということである。従って２つの照明の間にある画像撮像部１２も照明と同じ場所にあるとみなせることになる。まとめると、画像撮像部１２と照明と角膜曲率中心が同軸上にあるとみなせる条件は、角膜反射像中心点Ｐ1とＰ2の画像座標での距離が1pixel以下になることである。 In FIG. 2, if the distance between the two corneal reflex image center points P 1 and P 2 by the illuminations S 1 and S 2 arranged on both sides of the image capturing unit 12 is 1 pixel or less in the camera image, the corneal reflex image center point P 1 and the corneal reflex image center point P 2 are observed on the same pixel or adjacent pixels. This means that the two illuminations reflected on the surface of the cornea cannot be distinguished, and the image capturing unit 12 can consider that the two illuminations are in the same place. Therefore, the image capturing unit 12 between the two lights can be regarded as being in the same place as the lights. In summary, the condition that the image imaging unit 12, the illumination, and the center of curvature of the cornea can be regarded as coaxial is that the distance between the corneal reflex image center points P 1 and P 2 in the image coordinates is 1 pixel or less.
ＣはＳ1Ｓ2の中間にあるものとし、距離ＣＳ1とＣＳ2をそれぞれｘとする。角膜表面を球と仮定したので、Ｐ1における垂線は角膜曲率中心を通るとともに、角ＣＰ1Ｓ1の２等分線になる。従って、以下の式(２)が成立する。 It is assumed that C is in the middle of S 1 and S 2 , and the distances CS 1 and CS 2 are x, respectively. Since the surface of the cornea is assumed to be a sphere, the perpendicular at P 1 passes through the center of curvature of the cornea and becomes a bisector of the angle CP 1 S 1 . Therefore, the following equation (2) holds.
・・・(２)
・ ・ ・ (2)
角ＣＰ2Ｓ2においても同様のことが言えるため、以下の式(３)を得る。 Since the same can be said for the angle CP 2 S 2 , the following equation (3) is obtained.
・・・(３)
・ ・ ・ (3)
三角形Ｍ1ＡＭ2と三角形Ｐ1ＡＰ2の相似条件から、以下の式(５)が得られる。 From the similarity condition of the triangle M 1 AM 2 and the triangle P 1 AP 2 , the following equation (5) is obtained.
・・・(５)
・ ・ ・ (5)
角膜反射像Ｐ1の三次元ベクトルを
、画像座標を
とする。角膜反射像Ｐ2についても同様に定義する。ピクセル単位で表される焦点距離をｆとすると、透視投影における
の関係式、および
の関係式(６)が得られる。
The three-dimensional vector of the corneal reflex image P 1
, Image coordinates
And. The same applies to the corneal reflex image P 2 . Let f be the focal length expressed in pixels, and in perspective projection
Relational expression, and
The relational expression (6) of is obtained.
・・・(６)
・ ・ ・ (6)
なので、角膜反射像中心点Ｐ1とＰ2の画像座標での距離は以下の式(７)で表される。
Therefore, the distance between the center points P 1 and P 2 of the corneal reflex image in the image coordinates is expressed by the following equation (7).
・・・(７)
... (7)
式(５)を式(７)に代入し、以下の式(８)を得る。 Substituting equation (5) into equation (7), the following equation (8) is obtained.
・・・(８)
・ ・ ・ (8)
最後に、角膜反射像中心点Ｐ1とＰ2の画像座標での距離が1pixel以下なので以下の式(１０)を得る。 Finally, since the distance between the corneal reflex image center points P 1 and P 2 at the image coordinates is 1 pixel or less, the following equation (10) is obtained.
・・・(９)
・・・(１０)
... (9)
... (10)
コンピュータ１４は、ＣＰＵ、後述する視線計測処理ルーチンのプログラムを記憶したＲＯＭ、データ等を記憶するＲＡＭ、及びこれらを接続するバスを含んで構成されている。このコンピュータ１４をハードウエアとソフトウエアとに基づいて定まる機能実現手段毎に分割した機能ブロックで説明すると、図１に示すように、コンピュータ１４は、画像撮像部１２から出力される濃淡画像である顔画像を入力する画像入力部２０と、画像入力部２０の出力である顔画像の時系列から、角膜反射法（図３参照）を用いて、カメラ座標系における視線ベクトルを計算する角膜反射法視線検出部２８とを備えている。なお、角膜反射法視線検出部２８が、視線ベクトル計算手段の一例である。 The computer 14 includes a CPU, a ROM that stores a program of a line-of-sight measurement processing routine described later, a RAM that stores data, and a bus that connects them. Explaining the computer 14 with functional blocks divided for each function realization means determined based on hardware and software, as shown in FIG. 1, the computer 14 is a shade image output from the image capturing unit 12. Corneal reflex method that calculates the line-of-sight vector in the camera coordinate system using the corneal reflex method (see FIG. 3) from the time series of the face image that is the output of the image input unit 20 and the image input unit 20 that inputs the face image. It includes a line-of-sight detection unit 28. The corneal reflex line-of-sight detection unit 28 is an example of the line-of-sight vector calculation means.
画像入力部２０は、例えば、Ａ／Ｄコンバータや１画面の画像データを記憶する画像メモリ等で構成される。 The image input unit 20 includes, for example, an A / D converter, an image memory for storing image data of one screen, and the like.
角膜反射法視線検出部２８は、図１に示すように、眼球モデル記憶部４０と、角膜反射像位置推定部４２と、角膜曲率中心計算部４４と、視線ベクトル計算部４６とを備えている。 As shown in FIG. 1, the corneal reflex line-of-sight detection unit 28 includes an eyeball model storage unit 40, a corneal reflex image position estimation unit 42, a corneal curvature center calculation unit 44, and a line-of-sight vector calculation unit 46. ..
眼球モデル記憶部４０には、顔形状モデル座標系における眼球中心座標Ｅと、角膜曲率に応じた球体及び眼球を表す球体の位置関係及びサイズと、瞳孔中心座標と角膜曲率中心座標との距離ｓ、角膜曲率中心座標と眼球中心座標との距離ｔ、角膜反射像中心座標と角膜曲率中心座標との距離ｒとが記憶されている（図４参照）。 In the eyeball model storage unit 40, the eyeball center coordinates E in the face shape model coordinate system, the positional relationship and size of the sphere and the sphere representing the eyeball according to the corneal curvature, and the distance s between the pupil center coordinates and the corneal curvature center coordinates. , The distance t between the center coordinates of the curvature of the corneum and the center coordinates of the eyeball, and the distance r between the center coordinates of the reflection image of the corneum and the center coordinates of the curvature of the corneum are stored (see FIG. 4).
角膜反射像位置推定部４２は、顔画像から、顔画像上の角膜反射像中心の２次元座標を計算し、顔画像上の角膜反射像中心の２次元座標と、顔形状モデル座標系における眼球中心座標Ｅとから、カメラ位置Ｃから角膜反射像中心Ｐへ向かう３次元ベクトルｐを推定する。 The corneal reflex image position estimation unit 42 calculates the two-dimensional coordinates of the center of the corneal reflex image on the face image from the face image, and the two-dimensional coordinates of the center of the corneal reflex image on the face image and the eyeball in the face shape model coordinate system. From the center coordinates E, the three-dimensional vector p from the camera position C toward the center P of the corneal reflex image is estimated.
角膜曲率中心計算部４４は、３次元ベクトルｐと、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒとを用いて、以下の式に従って、カメラ位置Ｃから角膜曲率中心Ａへ向かう３次元ベクトルａを推定する。 The corneal curvature center calculation unit 44 uses the three-dimensional vector p and the distance r between the corneal reflection image center P and the corneal curvature center A, and according to the following equation, the three-dimensional direction from the camera position C to the corneal curvature center A. Estimate the vector a.
視線ベクトル計算部４６は、顔画像から、顔画像上の瞳孔中心（見かけの瞳孔中心）Ｂの２次元座標を計算し、顔画像上の瞳孔中心（見かけの瞳孔中心）Ｂの２次元座標と、３次元ベクトルａと、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒと用いて、カメラ位置Ｃから見かけの瞳孔中心Ｂへ向かう３次元ベクトルｂを求める。ここで、図５に示すように、見かけの瞳孔中心Ｂの候補は２か所存在するため、Zがカメラ側になる方(=小さい方)を、見かけの瞳孔中心Ｂとして選べばよい。 The line-of-sight vector calculation unit 46 calculates the two-dimensional coordinates of the pupil center (apparent pupil center) B on the face image from the face image, and sets the two-dimensional coordinates of the pupil center (apparent pupil center) B on the face image. Using the three-dimensional vector a and the distance r between the corneal reflection image center P and the corneal curvature center A, the three-dimensional vector b from the camera position C toward the apparent pupil center B is obtained. Here, as shown in FIG. 5, since there are two candidates for the apparent pupil center B, the side with Z on the camera side (= smaller one) may be selected as the apparent pupil center B.
そして、視線ベクトル計算部４６は、３次元ベクトルｂと、３次元ベクトルｐとを用いて、角膜反射像中心Ｐと見かけの瞳孔中心Ｂとの距離ｕを計算する。視線ベクトル計算部４６は、３次元ベクトルｐと、３次元ベクトルｂとを用いて、以下の式に従って、３次元ベクトルｐと、３次元ベクトルｂとの角度εを計算する。 Then, the line-of-sight vector calculation unit 46 calculates the distance u between the corneal reflex image center P and the apparent pupil center B by using the three-dimensional vector b and the three-dimensional vector p. The line-of-sight vector calculation unit 46 calculates the angle ε between the three-dimensional vector p and the three-dimensional vector b according to the following equation using the three-dimensional vector p and the three-dimensional vector b.
視線ベクトル計算部４６は、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒと、角膜反射像中心Ｐと見かけの瞳孔中心Ｂとの距離ｕ（＝||ｂ−ｐ||）とを用いて、以下の式に従って、３次元ベクトルｐと、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θを計算する。 The line-of-sight vector calculation unit 46 determines the distance r between the corneal reflex image center P and the corneal curvature center A and the distance u (= || bp ||) between the corneal reflex image center P and the apparent pupil center B. Using the following equation, the angle θ between the three-dimensional vector p and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A is calculated.
視線ベクトル計算部４６は、３次元ベクトルｐと３次元ベクトルｂとの角度εの計算結果と、３次元ベクトルｐと見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θの計算結果とを用いて、以下の式に従って、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルと、真の瞳孔中心Ｂ´から眼球中心Ｅへ向かう３次元ベクトルとの角度ψを計算する。以下の式で、ｒは角膜曲率半径、ｓはＡＢ’間の距離、ｎ1は、空気の屈折率、ｎ2は角膜の内側にある水晶体の屈折率である。 The line-of-sight vector calculation unit 46 determines the angle θ between the calculation result of the angle ε between the three-dimensional vector p and the three-dimensional vector b and the three-dimensional vector p and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A. Using the calculation results, calculate the angle ψ between the 3D vector from the apparent pupil center B to the corneal curvature center A and the 3D vector from the true pupil center B'to the eyeball center E according to the following equation. To do. In the following equation, r is the radius of curvature of the cornea, s is the distance between AB', n 1 is the refractive index of air, and n 2 is the refractive index of the crystalline lens inside the cornea.
視線ベクトル計算部４６は、３次元ベクトルｐと見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θと、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルと真の瞳孔中心Ｂ´から眼球中心Ｅへ向かう３次元ベクトルとの角度ψとの和により、視線ベクトルの角度（＝θ＋ψ）を計算する。 The line-of-sight vector calculation unit 46 includes the angle θ between the three-dimensional vector p and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A, and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A. The angle of the line-of-sight vector (= θ + ψ) is calculated by summing the angle ψ with the three-dimensional vector from the pupil center B'to the eyeball center E.
視線ベクトル計算部４６は、視線ベクトルの角度（＝θ＋ψ）と、カメラ位置Ｃから角膜反射像中心Ｐへ向かう３次元ベクトルｐと、カメラ位置Ｃから見かけの瞳孔中心Ｂへ向かう３次元ベクトルｂとに基づいて、視線ベクトルｄを求めて、出力部１６により出力する。ただし、-(θ＋ψ)でも式が成立するため、視線ベクトルｄのｙ成分が３次元ベクトルｐのｙ成分よりも大きくなる方を選ぶか、ｒｄがＢに近い方を選べばよい。 The line-of-sight vector calculation unit 46 includes the angle of the line-of-sight vector (= θ + ψ), the three-dimensional vector p from the camera position C toward the corneal reflection image center P, and the three-dimensional vector b from the camera position C toward the apparent pupil center B. The line-of-sight vector d is obtained based on the above, and is output by the output unit 16. However, since the equation holds even with-(θ + ψ), it is sufficient to select the one in which the y component of the line-of-sight vector d is larger than the y component of the three-dimensional vector p, or the one in which rd is close to B.
＜視線計測装置の動作＞
次に、視線計測装置１０の動作について説明する。まず、照射部１３により近赤外の光を被験者の目に照射しているときに、画像撮像部１２で被験者の顔画像を連続的に撮像する。
<Operation of line-of-sight measuring device>
Next, the operation of the line-of-sight measuring device 10 will be described. First, while the irradiation unit 13 is irradiating the subject's eyes with near-infrared light, the image imaging unit 12 continuously images the subject's face image.
そして、コンピュータ１４において、撮像された顔画像毎に、図６に示す視線計測処理ルーチンを実行する。まず、ステップＳ１１８において、画像撮像部１２で撮像された顔画像を取得する。 Then, the computer 14 executes the line-of-sight measurement processing routine shown in FIG. 6 for each captured face image. First, in step S118, the face image captured by the image capturing unit 12 is acquired.
そして、ステップＳ１２０において、顔画像から、顔画像上の角膜反射像中心の２次元座標を計算し、顔画像上の角膜反射像中心の２次元座標と、顔形状モデル座標系における仮の眼球中心座標Ｅとから、カメラ位置Ｃから角膜反射像中心Ｐへ向かう３次元ベクトルｐを推定する。 Then, in step S120, the two-dimensional coordinates of the center of the corneal reflex image on the face image are calculated from the face image, and the two-dimensional coordinates of the center of the corneal reflex image on the face image and the temporary eyeball center in the face shape model coordinate system are calculated. From the coordinates E, the three-dimensional vector p from the camera position C toward the center P of the corneal reflex image is estimated.
ステップＳ１２２では、上記ステップＳ１２０で推定された３次元ベクトルｐと、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒとを用いて、カメラ位置Ｃから角膜曲率中心Ａへ向かう３次元ベクトルａを推定する。 In step S122, the three-dimensional vector a from the camera position C toward the corneal curvature center A is used by using the three-dimensional vector p estimated in step S120 and the distance r between the corneal reflection image center P and the corneal curvature center A. To estimate.
ステップＳ１２４では、顔画像から、顔画像上の瞳孔中心（見かけの瞳孔中心）Ｂの２次元座標を計算する。そして、ステップＳ１２６では、顔画像上の瞳孔中心（見かけの瞳孔中心）Ｂの２次元座標と、上記ステップＳ１２２で推定された３次元ベクトルａと、角膜反射像中心Ｐと眼球中心Ｅとの距離ｒと用いて、カメラ位置Ｃから見かけの瞳孔中心Ｂへ向かう３次元ベクトルｂを求める。 In step S124, the two-dimensional coordinates of the pupil center (apparent pupil center) B on the face image are calculated from the face image. Then, in step S126, the two-dimensional coordinates of the pupil center (apparent pupil center) B on the face image, the three-dimensional vector a estimated in step S122, and the distance between the corneal reflection image center P and the eyeball center E. Using r, the three-dimensional vector b from the camera position C toward the apparent pupil center B is obtained.
そして、ステップＳ１２８では、上記ステップＳ１２４で求めた３次元ベクトルｂと、上記ステップＳ１２０で推定された３次元ベクトルｐとを用いて、角膜反射像中心Ｐと見かけの瞳孔中心Ｂとの距離ｕを計算する。また、３次元ベクトルｐと、３次元ベクトルｂとを用いて、３次元ベクトルｐと、３次元ベクトルｂとの角度εを計算する。また、角膜反射像中心Ｐと角膜曲率中心Ａとの距離ｒと、角膜反射像中心Ｐと見かけの瞳孔中心Ｂとの距離ｕ（＝||ｂ−ｐ||）とを用いて、３次元ベクトルｐと、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θを計算する。また、３次元ベクトルｐと３次元ベクトルｂとの角度εの計算結果と、３次元ベクトルｐと見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θの計算結果とを用いて、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルと、真の瞳孔中心Ｂ´から眼球中心Ｅへ向かう３次元ベクトルとの角度ψを計算する。 Then, in step S128, the distance u between the corneal reflex image center P and the apparent pupil center B is determined by using the three-dimensional vector b obtained in step S124 and the three-dimensional vector p estimated in step S120. calculate. Further, the angle ε between the three-dimensional vector p and the three-dimensional vector b is calculated using the three-dimensional vector p and the three-dimensional vector b. Further, using the distance r between the corneal reflex image center P and the corneal curvature center A and the distance u (= || bp ||) between the corneal reflex image center P and the apparent pupil center B, three dimensions are used. The angle θ between the vector p and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A is calculated. Further, the calculation result of the angle ε between the three-dimensional vector p and the three-dimensional vector b and the calculation result of the angle θ between the three-dimensional vector p and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A are used. Then, the angle ψ between the three-dimensional vector from the apparent pupil center B to the corneal curvature center A and the three-dimensional vector from the true pupil center B'to the eyeball center E is calculated.
そして、上記で計算された、３次元ベクトルｐと見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルとの角度θと、見かけの瞳孔中心Ｂから角膜曲率中心Ａへ向かう３次元ベクトルと真の瞳孔中心Ｂ´から眼球中心Ｅへ向かう３次元ベクトルとの角度ψとの和により、視線ベクトルの角度を計算する。 Then, the angle θ between the three-dimensional vector p calculated above and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A, and the three-dimensional vector from the apparent pupil center B toward the corneal curvature center A. The angle of the line-of-sight vector is calculated by summing the angle ψ with the three-dimensional vector from the true pupil center B'to the eyeball center E.
そして、ステップＳ１３０では、上記ステップＳ１２８で計算された、視線ベクトルの角度と、カメラ位置Ｃから角膜反射像中心Ｐへ向かう３次元ベクトルｐと、カメラ位置Ｃから見かけの瞳孔中心Ｂへ向かう３次元ベクトルｂとに基づいて、視線ベクトルｄを求め、出力部１６により出力して、視線計測処理ルーチンを終了する。 Then, in step S130, the angle of the line-of-sight vector calculated in step S128, the three-dimensional vector p from the camera position C toward the corneal reflex image center P, and the three-dimensional from the camera position C toward the apparent pupil center B. The line-of-sight vector d is obtained based on the vector b and is output by the output unit 16, and the line-of-sight measurement processing routine is terminated.
以上説明したように、本発明の実施の形態に係る視線計測装置によれば、画像撮像部の撮像方向と照射部の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たすように配置し、角膜反射像を用いて、視線ベクトルを計算することにより、画像撮像部の撮像方向と照射部の光照射方向とが同軸でない場合であっても、簡易な構成で、近似計算を行うことなく、精度よく視線計測を行うことができる。 As described above, according to the line-of-sight measuring device according to the embodiment of the present invention, a predetermined constraint condition for considering that the imaging direction of the image capturing unit and the light irradiation direction of the irradiation unit are coaxial is set. By arranging so as to satisfy and calculating the line-of-sight vector using the corneal reflex image, even if the imaging direction of the image imaging unit and the light irradiation direction of the irradiation unit are not coaxial, it can be approximated with a simple configuration. The line-of-sight measurement can be performed accurately without performing a calculation.
また、画像撮像部と照射部とが同軸として扱えるため、角膜曲率中心推定計算に近似計算が含まれず、角膜曲率中心の推定精度が向上すると共に、角膜曲率中心推定の計算量を削減することができる。また、１つの光源で視線計算が可能になり装置コストが下げられる。 In addition, since the image imaging unit and the irradiation unit can be treated as coaxial, the calculation for estimating the center of curvature of the cornea does not include approximate calculation, which improves the estimation accuracy of the center of curvature of the cornea and reduces the amount of calculation for estimating the center of curvature of the cornea. it can. In addition, the line-of-sight calculation can be performed with one light source, and the equipment cost can be reduced.
１０ 視線計測装置
１２ 画像撮像部
１３ 照射部
１４ コンピュータ
１６ 出力部
２０ 画像入力部
２８ 角膜反射法視線検出部
４０ 眼球モデル記憶部
４２ 角膜反射像位置推定部
４４ 角膜曲率中心計算部
４６ 視線ベクトル計算部
10 Eye-gaze measurement device 12 Image imaging unit 13 Irradiation unit 14 Computer 16 Output unit 20 Image input unit 28 Corneal reflex line-of-sight detection unit 40 Eyeball model storage unit 42 Corneal reflex image position estimation unit 44 Corneal curvature center calculation unit 46 Line-of-sight vector calculation unit
Claims (2)
前記被観察者の目に対して光を照射する光照射手段と、
前記撮像手段によって撮像された前記顔を表す顔画像から、前記顔画像上の前記顔の目の角膜反射像と、前記顔画像上の前記顔の目の瞳孔中心位置と、予め定められた３次元眼球モデルとに基づいて、カメラ座標系における３次元の視線ベクトルを計算する視線ベクトル計算手段と、
を含み、
前記撮像手段と前記光照射手段との位置関係、前記撮像手段と前記目との位置関係、及び前記撮像手段に関するパラメータが、前記撮像手段の撮像方向と前記光照射手段の光照射方向とが同軸であるとみなすための予め定められた制約条件を満たし、
前記制約条件は、以下の式で表わされる視線計測装置。
ただし、ｘは、前記撮像手段と前記光照射手段との間の距離、Ｌは、前記撮像手段から角膜曲率中心へ向かう直線と角膜との交点と、前記撮像手段との間の距離、ｒは、角膜曲率半径、ｆは、前記撮像手段のピクセル単位の焦点距離である。 An imaging means that captures the face of the observer,
A light irradiation means for irradiating the eyes of the observed person with light,
From the face image representing the face captured by the imaging means, the corneal reflection image of the eye of the face on the face image and the pupil center position of the eye of the face on the face image are predetermined 3D. A line-of-sight vector calculation means for calculating a three-dimensional line-of-sight vector in the camera coordinate system based on a three-dimensional eyeball model,
Including
The positional relationship between the image pickup means and the light irradiation means, the positional relationship between the image pickup means and the eye, and the parameters related to the image pickup means are coaxial with the image pickup direction of the image pickup means and the light irradiation direction of the light irradiation means. It meets a predetermined constraint condition for regarding that is,
The constraint condition is a line-of-sight measuring device represented by the following equation .
However, x is the distance between the imaging means and the light irradiation means, L is the distance between the intersection of the straight line and the cornea from the imaging means toward the center of curvature of the cornea, and r is the distance between the imaging means. , Corneal radius of curvature, f is the focal length in pixels of the imaging means.
前記顔の目の角膜反射像と、前記３次元眼球モデルとに基づいて、前記角膜反射像の３次元位置を推定する角膜反射像位置推定手段と、
前記角膜反射像の３次元位置に基づいて、角膜曲率中心の３次元位置を計算する角膜曲率中心計算手段と、を含み、
前記角膜反射像の３次元位置と、前記角膜曲率中心の３次元位置とに基づいて、前記カメラ座標系における３次元の視線ベクトルを計算する請求項１記載の視線計測装置。 The line-of-sight vector calculation means
A corneal reflex image position estimating means for estimating a three-dimensional position of the corneal reflex image based on the corneal reflex image of the facial eye and the three-dimensional eyeball model.
Includes a corneal curvature center calculation means for calculating the three-dimensional position of the corneal curvature center based on the three-dimensional position of the corneal reflection image.
A three-dimensional position of the corneal reflection image, the based on the three-dimensional position of the center of corneal curvature, gaze measuring device according to claim 1 for calculating the three-dimensional line-of-sight vector in the camera coordinate system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2017114762A JP6785723B2 (en) | 2017-06-09 | 2017-06-09 | Line-of-sight measuring device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2017114762A JP6785723B2 (en) | 2017-06-09 | 2017-06-09 | Line-of-sight measuring device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| JP2019000135A JP2019000135A (en) | 2019-01-10 |
| JP6785723B2 true JP6785723B2 (en) | 2020-11-18 |
Family
ID=65005490
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| JP2017114762A Active JP6785723B2 (en) | 2017-06-09 | 2017-06-09 | Line-of-sight measuring device |
Country Status (1)
| Country | Link |
|---|---|
| JP (1) | JP6785723B2 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP7283841B2 (en) | 2019-03-28 | 2023-05-30 | 株式会社豊田中央研究所 | Gaze measurement device |
| JP7255436B2 (en) * | 2019-09-25 | 2023-04-11 | 株式会社豊田中央研究所 | Eyeball structure estimation device |
| CN113662506B (en) * | 2021-09-26 | 2023-08-04 | 温州医科大学 | Cornea surface morphology measurement method and device, medium and electronic equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5485502A (en) * | 1994-07-26 | 1996-01-16 | Lunar Corporation | Radiographic gantry with software collision avoidance |
| JP4059266B2 (en) * | 2005-10-31 | 2008-03-12 | 株式会社日立メディコ | X-ray fluoroscopic equipment |
| US7682026B2 (en) * | 2006-08-22 | 2010-03-23 | Southwest Research Institute | Eye location and gaze detection system and method |
| JP2010259605A (en) * | 2009-05-01 | 2010-11-18 | Nippon Hoso Kyokai <Nhk> | Visual line measuring device and visual line measuring program |
| JP5404508B2 (en) * | 2010-04-08 | 2014-02-05 | 株式会社東芝 | X-ray diagnostic equipment |
| JP6360752B2 (en) * | 2014-08-29 | 2018-07-18 | アルプス電気株式会社 | Illumination imaging device |
-
2017
- 2017-06-09 JP JP2017114762A patent/JP6785723B2/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| JP2019000135A (en) | 2019-01-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6800091B2 (en) | Line-of-sight measuring device and program | |
| US10878237B2 (en) | Systems and methods for performing eye gaze tracking | |
| JP5158842B2 (en) | Eye movement measuring method and eye movement measuring apparatus | |
| US10140726B2 (en) | Apparatus and method for estimating gazed position of person | |
| WO2018030515A1 (en) | Line-of-sight detection device | |
| JP2016173313A (en) | Visual line direction estimation system, visual line direction estimation method and visual line direction estimation program | |
| US20150029322A1 (en) | Method and computations for calculating an optical axis vector of an imaged eye | |
| JP7030317B2 (en) | Pupil detection device and pupil detection method | |
| JP6785723B2 (en) | Line-of-sight measuring device | |
| JP6452235B2 (en) | Face detection method, face detection device, and face detection program | |
| JP7283841B2 (en) | Gaze measurement device | |
| JP2021531601A (en) | Neural network training, line-of-sight detection methods and devices, and electronic devices | |
| JP7255436B2 (en) | Eyeball structure estimation device | |
| JP2016532217A (en) | Method and apparatus for detecting eyes with glint | |
| JP6288770B2 (en) | Face detection method, face detection system, and face detection program | |
| US11054659B2 (en) | Head mounted display apparatus and distance measurement device thereof | |
| WO2016159255A1 (en) | Mouth region detection device and mouth region detection method | |
| CN112528714B (en) | Single-light-source-based gaze point estimation method, system, processor and equipment | |
| JP2011118767A (en) | Facial expression monitoring method and facial expression monitoring apparatus | |
| KR102074977B1 (en) | Electronic devices and methods thereof | |
| JP2013019721A (en) | Position detector, position detection method, and computer program | |
| KR101348903B1 (en) | Cornea radius estimation apparatus using cornea radius estimation algorithm based on geometrical optics for eye tracking and method | |
| Fujiyoshi et al. | Inside-out camera for acquiring 3D gaze points | |
| JP2016045707A (en) | Feature point detection system, feature point detection method, and feature point detection program | |
| JP2015206764A (en) | Object person existence range estimation device and object person existence range estimation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| A621 | Written request for application examination |
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20190403 |
|
| A977 | Report on retrieval |
Free format text: JAPANESE INTERMEDIATE CODE: A971007 Effective date: 20200226 |
|
| A131 | Notification of reasons for refusal |
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20200310 |
|
| A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20200501 |
|
| TRDD | Decision of grant or rejection written | ||
| A01 | Written decision to grant a patent or to grant a registration (utility model) |
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20200929 |
|
| A61 | First payment of annual fees (during grant procedure) |
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20201027 |
|
| R150 | Certificate of patent or registration of utility model |
Ref document number: 6785723 Country of ref document: JP Free format text: JAPANESE INTERMEDIATE CODE: R150 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |