JP2015169959A - Rotation angle calculation method, gaze point detection method, information input method, rotation angle calculation apparatus, gaze point detection apparatus, information input apparatus, rotation angle calculation program, gaze point detection program, and information input program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a gaze point with high accuracy, regardless of a posture of a subject.SOLUTION: A gaze point detection apparatus 1 includes two stereo cameras 2a, 2b for acquiring a face image of a subject A, light sources 3a, 3b, control circuits 4-6, and an image processing apparatus 7. The image processing apparatus 7 calculates coordinates of a corneal reflection point of the subject and a pupil center, on the basis of the face image acquired by the stereo cameras 2a, 2b, and calculates a direction of an optical axis on the basis of the coordinates. An origin correction vector ris calculated, as an error between the optical axis and a visual axis, to be formed when the subject is in one posture. The image processing apparatus 7 corrects the origin correction vector raccording to the posture of the subject, on the basis of constraints that the origin correction vector ris rotated around the visual axis at the same rotation angle in the left eye and the right eye of the subject. On the basis of the corrected origin correction vector r, the optical axis direction is corrected, to calculate a visual axis direction of the subject A. On the basis of the visual axis direction, a gaze point is detected.

Description

  The present invention relates to a rotation angle calculation method for detecting a gaze point of a subject on a predetermined plane based on the image of the subject, a gaze point detection method, an information input method using the gaze point detection method, a rotation angle calculation device, The present invention relates to a gaze point detection device, an information input device, a rotation angle calculation program, a gaze point detection program, and an information input program.

  2. Description of the Related Art Conventionally, an apparatus that detects a subject's gaze or gaze point in a non-contact manner has been regarded as important in the field of human interaction. If high-precision line-of-sight detection technology is put into practical use, it can be applied to various uses such as monitoring drivers, investigating the degree of interest in products, and inputting data into personal computers of severely disabled people.

  In the gaze point detection methods described in Non-Patent Documents 1 and 2, by making the subject gaze at a total of two points, a camera with a known position and one point on the display screen with a known position, A function for calculating the gaze direction of the subject is corrected from the distance between the center and the corneal reflection point, and the gaze direction and the gaze point are detected using the corrected function. According to such a gaze point detection method, it is possible to detect a line of sight with high accuracy even if the subject's head moves.

  In particular, in the gaze point detection method described in Non-Patent Document 2, an optical axis that is a central axis when considering the symmetry of the eyeball by letting the subject gaze at one point on the display screen whose position is known, By obtaining a correction vector for correcting a deviation between the visual axis that is an axis passing through the fovea on the retina of the eyeball and the gaze point of the subject, the gaze point detection accuracy and the convenience of the apparatus are improved.

Ebisawa Y. and Fukumoto K .: "Head-free, Remote Eye-gaze Detection System Based on Pupil-corneal Reflection Method with Easy Calibration Using Two Stereo-calibrated Video Cameras" IEEE Trans Biomed Eng., 60 (10), pp. 2952-2960, 2013. Masato Ogasawara, Toshiki Anbo, Kotaro Nishida, Kiyotake Fukumoto, Yoshinobu Ebizawa “Remote and high-precision gaze detection device using simple calibration method”, ViEW2011, I1-31, 7pages, 2010.

  However, for example, when the subject turns his / her head sideways while facing the front or the display screen, the subject's head may rotate. In this case, in the gaze point detection method described in Non-Patent Document 2, the rotation of the subject's head is not reflected in the correction vector, and the accuracy of gaze point detection may be reduced.

  Therefore, the present invention provides a rotation angle calculation method, a rotation angle calculation device, a rotation angle calculation program, and a gazing point detection method using these, which can calculate the rotation angle of the correction vector with high accuracy regardless of the posture of the subject. An object is to provide a gaze point detection device, a gaze point detection program, an information input method, an information input device, and an information input program.

  In order to solve the above problem, a rotation angle calculation method of the present invention includes a face image generation step of generating a face image of a subject using two or more cameras and a light source that irradiates the subject with light. Based on the respective face images from two or more cameras, the coordinates of the corneal reflection point, which is the reflection point of the light from the light source on the subject's cornea, and the coordinates of the pupil center, which is the center of the subject's pupil, A coordinate calculation step that calculates the direction of the optical axis that is the central axis of the eyeball of the subject based on the coordinates of the corneal reflection point and the coordinates of the pupil center, Using the straight line connecting the fovea and the subject's gazing point as the visual axis, the deviation between the optical axis and visual axis when the subject takes one posture is calculated based on the coordinates of the corneal reflection point and the coordinates of the pupil center Deviation calculation step and rotation around the visual axis of the deviation between the optical axis and the visual axis. Angle based on the constraint that is identical with the left and right eyes of a subject, comprising a rotation angle calculation step of calculating a rotation angle of the visual axis around a.

  The rotation angle calculation device according to the present invention is a gaze point detection device that detects a gaze point of the subject based on the face image of the subject, and two or more cameras that acquire the face image of the subject And a light source that irradiates the subject with light and an image processing unit that processes image signals output from two or more cameras, and the image processing unit applies each of the face images from the two or more cameras to each face image. Based on the coordinates of the corneal reflection point, which is the reflection point of the light from the light source on the subject's cornea, and the coordinates of the pupil center, which is the center of the subject's pupil, the coordinates of the corneal reflection point and the pupil are calculated. Based on the coordinates of the center, the direction of the optical axis, which is the central axis of the subject's eyeball, is calculated, and the subject is identified by using the straight line connecting the fovea of the subject's eyeball and the subject's point of interest as the visual axis. The difference between the optical axis and the visual axis when the posture is taken is changed to the coordinates of the corneal reflection point and the coordinates of the pupil center. The rotation angle around the visual axis is calculated based on the constraint that the rotation angle around the visual axis of the deviation between the optical axis and the visual axis is the same for the left eye and the right eye of the subject. .

  In addition, the rotation angle calculation program of the present invention includes two or more face image generating means for generating a face image of a subject using two or more cameras and a light source that irradiates the subject with light. The coordinates of the corneal reflection point, which is the reflection point of the light from the light source on the subject's cornea, and the coordinates of the pupil center, which is the center of the subject's pupil, are calculated based on each face image from the camera Coordinate calculation means, optical axis direction calculation means for calculating the direction of the optical axis that is the central axis of the eyeball of the subject based on the coordinates of the cornea reflection point and the coordinates of the pupil center, the fovea of the eyeball of the subject and the subject A deviation calculating means for calculating a deviation between the optical axis and the visual axis when the subject takes one posture based on the coordinates of the corneal reflection point and the coordinates of the pupil center, with the straight line connecting the gazing point of And the rotation angle around the visual axis of the deviation between the optical axis and the visual axis is the subject's left And based on the constraint that is identical with the right eye, the rotation angle calculation means for calculating a rotation angle of the visual axis around to function as.

  According to such a rotation angle calculation method, rotation angle calculation device, and rotation angle calculation program, first, the direction of the optical axis of the eyeball of the subject is calculated based on the coordinates of the corneal reflection point and the coordinates of the pupil center. The Also, the deviation between the optical axis and the visual axis when the subject takes one posture is calculated. Next, based on the constraint that the rotation angle around the visual axis of the deviation between the optical axis and the visual axis is the same for the left eye and the right eye of the subject, the visual axis of the deviation between the optical axis and the visual axis A rotation angle around is calculated. In this way, since the rotation angle around the visual axis of the deviation between the optical axis and the visual axis is calculated according to the posture of the subject, the deviation between the optical axis and the visual axis is highly accurate regardless of the orientation of the subject. The rotation angle around the visual axis can be calculated.

  In the rotation angle calculation step, the direction parallel to the straight line connecting the center of the pupil of the left eye of the subject and the center of the pupil of the right eye is defined as the first direction, and is perpendicular to both the first direction and the optical axis of the camera. It is preferable to calculate the rotation angle around the visual axis by using the component in the second direction, with the correct direction as the second direction.

  As a human eye property, the direction of the line of sight of the left eye and the right eye may be close to or away from the direction parallel to the straight line connecting the left eye and the right eye, that is, along the first direction, There is no approaching or leaving along the second direction. For this reason, when the shift is calculated using the component in the second direction, for example, even if the direction of the line of sight of the left eye and the right eye of the subject is separated, the direction of the line of sight of the left eye and the right eye Less susceptible to displacement. Therefore, it is possible to improve the accuracy of calculation of the rotation angle when the direction of the line of sight of the left eye and the right eye of the subject is separated.

  In the gaze point detection method of the present invention, the direction of the optical axis calculated in the optical axis direction calculation step in the rotation angle calculation method is based on the rotation angle around the visual axis calculated by the rotation angle calculation method. The direction of the visual axis is calculated by correcting the deviation, and the gaze point of the subject is detected based on the direction of the visual axis.

  As a result, since the deviation between the optical axis and the visual axis is corrected based on the rotation angle calculated according to the posture of the subject, the gaze point of the subject can be detected with high accuracy regardless of the posture of the subject. be able to.

  Furthermore, the information input method of the present invention includes both a binocular gazing point detection step for detecting the gazing point of both eyes of the subject by the gazing point detection method described above, and both of the subject detected in the binocular gazing point detection step. Determining that information has been input by the subject based on the distance between the gazing points of the eyes and the time within which the distance is within a predetermined range.

  According to this information input method, information on the subject's eyes is detected based on the distance between the eyes of the detected eyes and the time during which the distance is within a predetermined range. Is determined to have been entered. Here, in general, when a person wants to input some information, the shift of the gazing point of both eyes tends to be small, that is, the distance between the gazing points of both eyes tends to be narrow. On the other hand, when a person is wondering what to input, looking over the entire screen, or distracted, the distance between the gazing points of both eyes tends to increase. For this reason, according to said information input method, since it can detect appropriately that the subject is going to input some information, a subject can perform input operation efficiently.

  According to the gaze point detection method and the gaze point detection apparatus of the present invention, it is possible to detect a gaze point with high accuracy regardless of the posture of the subject.

It is a perspective view which shows the gazing point detection apparatus concerning suitable one Embodiment of this invention. It is a top view of the light source attached to the opening of the camera of FIG. It is a figure which shows the positional relationship of the coordinate system set with the gazing point detection apparatus of FIG. It is a figure for demonstrating the gaze point detection procedure by the gaze point detection apparatus of FIG. It is a figure for demonstrating the gaze point detection procedure by the gaze point detection apparatus of FIG. It is a perspective view which shows the shift | offset | difference of an optical axis and a visual axis. It is the figure which projected the gaze point and the camera on the virtual viewpoint plane. It is a figure which shows a cornea reflection position and a pupil position on a cornea reflection-pupil vector plane. It is the figure which translated each vector of FIG. 8 for description. It is a figure which shows the case where a subject's head rotates in FIG. It is a figure which shows rotation of the left and right pupil by rotation of a subject's head. In FIG. 10, it is a figure corresponding to the case where the gaze point of both eyes does not correspond. It is a figure which shows the fixation point detection error before and behind correction | amendment. It is a figure which shows the gaze point distribution before and behind correction | amendment.

  Hereinafter, preferred embodiments of a gaze point detection method and a gaze point detection apparatus according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted.

(Configuration of gaze point detection device)
First, the configuration of a gazing point detection apparatus for implementing the gazing point detection apparatus according to the present invention will be described with reference to the drawings. The gaze point detection device of the present invention is a device that detects a gaze point on a monitor screen of an information processing terminal such as a personal computer based on a face image of a subject. Note that this gaze point detection device also functions as a rotation angle calculation device.

  FIG. 1 is a perspective view showing a gazing point detection apparatus 1 which is a preferred embodiment of the present invention. As shown in the figure, the gazing point detection device 1 is provided outside the imaging lens of the two stereo cameras 2a and 2b that capture the face image of the subject A and the openings of the respective stereo cameras 2a and 2b. Light sources 3a and 3b, a light emitting circuit (control circuit) 4 for supplying power to the light sources 3a and 3b, a synchronization signal generator (control circuit) 5 for generating a synchronization signal to be input to the stereo cameras 2a and 2b, A delay circuit (control circuit) 6 for delaying the signal, an image processing device (image processing unit) 7 such as a personal computer for processing an image signal generated by the stereo cameras 2a and 2b, and the stereo cameras 2a and 2b The display device 8 is disposed so as to face the target person A and connected to the image processing device 7. The light emitting circuit 4, the synchronization signal generator 5, and the delay circuit 6 constitute a control circuit for controlling the operations of the stereo cameras 2a and 2b and the light sources 3a and 3b.

  The stereo cameras 2a and 2b generate image data by imaging the face of the subject A. As the stereo cameras 2a and 2b, NTSC system cameras which are one of interlace scan systems are used. In the NTSC system, one frame of image data obtained 30 frames per second is composed of an odd field composed of odd-numbered horizontal pixel lines and an even field composed of even-numbered horizontal pixel lines excluding odd fields. The odd field image and the even field image are alternately captured at 1/60 second intervals. Specifically, in one frame, the pixel lines in the odd field and the pixel lines in the even field are generated alternately.

  One stereo camera 2a receives a vertical synchronizing signal (VD signal) from the synchronizing signal generator 5, and the other stereo camera 2b is delayed from the synchronizing signal generator 5 via the delay circuit 6. When the vertical synchronization signal (VD signal) is input, the shooting timings of the two stereo cameras 2a and 2b are shifted from each other.

  Further, light sources 3a and 3b are fixed to the outside of the circular openings 9a and 9b in which the objective lenses of the stereo cameras 2a and 2b are accommodated, respectively. FIG. 2 shows a plan view of the light sources 3a and 3b. The light sources 3a and 3b are for irradiating illumination light toward the face of the subject A, and have a structure in which a plurality of two types of light emitting elements 11 and 12 are embedded in a ring-shaped pedestal 10. Yes. The light emitting element 11 is a semiconductor light emitting element (LED) having a center wavelength of output light of 850 nm, and is arranged on the pedestal 10 in a ring shape at equal intervals along the edges of the openings 9a and 9b. The light emitting element 12 is a semiconductor light emitting element having a center wavelength of output light of 950 nm, and is arranged on the pedestal 10 adjacent to the outside of the light emitting element 11 in a ring shape at equal intervals. That is, the distance of the light emitting element 12 from the optical axis of the stereo cameras 2 a and 2 b is set to be larger than the distance from the optical axis of the light emitting element 11. At this time, each light emitting element 11 and 12 is provided on the base part 10 so that illumination light may be emitted along the optical axis of the stereo cameras 2a and 2b.

  The light emitting elements 11 and 12 can be independently controlled in light emission timing by the light emitting circuit 4. Specifically, the light emission timing is controlled so that the light emitting elements 11 and 12 emit light alternately in accordance with the shutter timing of the stereo cameras 2a and 2b synchronized with the VD signal output from the synchronization signal generator 5. .

  Instead of the stereo cameras 2a and 2b, three or more cameras may be arranged and two of them may be used. That is, the gazing point detection device only needs to include two or more cameras.

  By such operation of the control circuit, when illumination light is irradiated from the light emitting element 11 of the light source 3a to the left and right eyeballs B of the subject A, a bright pupil image is generated in the eyeball B, and the light emitting element 12 moves to the eyeball B. When the illumination light is irradiated, a dark pupil image is generated in the eyeball B. This is because when the illumination light having a wavelength shorter than 900 nm is received, the pupil appears brighter than when the illumination light having a wavelength longer than 900 nm is received, and the illumination light to the eyeball B is further away from the optical axis of the camera. This is due to the nature of the pupil appearing darker when the light enters from. As a result, the bright and dark pupil images of the eyeball B are reflected in the odd and even fields generated by the stereo cameras 2a and 2b, respectively.

  The image processing device 7 processes the image data output from the two stereo cameras 2a and 2b. Specifically, the image processing device 7 separates one frame of image data output from the stereo cameras 2a and 2b into an odd field and an even field. For example, this odd field image data (odd image data) is a bright pupil image, and even field image data (even image data) is a dark pupil image. Since these image data have effective pixels only in the odd field or even field, the image processing device 7 embeds the luminance average of the pixel lines of the adjacent effective pixels in the pixel values between the lines. Bright pupil image data and dark pupil image data are generated.

  The image processing device 7 also detects the positions of the left and right corneal reflection points of the subject A with respect to the bright pupil image data and the dark pupil image data. That is, a window centered on the detected pupil is set, image data in which only the window range is increased in resolution is created, and corneal reflection is detected from the image data. Specifically, a threshold value for binarization is determined by the P tile method, a binarized image is created from the image, labeling is performed, and a portion whose area is not more than a certain value is selected. Here, the image processing device 7 gives a separability filter to the center coordinates of the selected portion, obtains a feature amount obtained by multiplying the separability and the luminance, and if the value is equal to or less than a certain value, it is not corneal reflection. Judge. Further, the image processing device 7 calculates the movement amount of the corneal reflection in the bright pupil image data and the dark pupil image data, and sets the movement amount as the difference position correction amount. The image processing device 7 shifts the difference position correction amount so that the corneal reflection positions of the bright pupil image data and the dark pupil image data coincide with each other, adds the luminance of the image data, and sets the luminance centroid coordinates to the corneal reflection coordinates. And decide.

  Further, the image processing device 7 calculates the three-dimensional positions of the left and right pupils of the subject A from the pupil center coordinates detected based on the image data output from the two stereo cameras 2a and 2b. At this time, the image processing device 7 measures the three-dimensional coordinates of the left and right pupils by a stereo method. The stereo method measures the internal parameters such as the focal length of the camera lens, image center, and pixel size, and external parameters such as the camera position and orientation in advance, and shoots the object using multiple stereo cameras. In some cases, the position of a point in space is determined using the internal and external parameters based on the coordinates of the point in the image.

When the image processing device 7 calculates the three-dimensional coordinates of the pupil using the stereo method, a coordinate system as shown in FIG. 3 is used. World coordinate system shown in FIG. _{(X W, Y W, Z} W) is two stereo cameras 2a, the coordinate system origin _{O W} is positioned for example at the center of the screen of the display unit 8 to be shared by 2b, a camera The coordinate system (X, Y, Z) is a coordinate system in which the origin C is the optical center of the stereo cameras 2a, 2b, and the Z axis is parallel to the optical axis drawn perpendicularly to the image plane from the optical center. The image coordinate system (X _G , Y _G ) is a coordinate system that is parallel to the XY plane along the image plane on which the image sensor is placed, and has an intersection Ci (image center) between the optical axis and the image plane as the origin Ci. . When the point P is the coordinates of the target point, the projection point (X _d , Y _d ) on the image coordinate system when using the stereo cameras 2a, 2b is an ideal projection point (X _u , Y) due to image distortion. _u ). Therefore, in order to accurately perform the three-dimensional position measurement using the stereo method, it is necessary to previously acquire calibration data in which the correspondence between the world coordinates of the target point P and the image coordinates is recorded. For example, as such calibration data, the translation vector of the camera coordinate system with respect to the world coordinates as external parameters and the rotation matrix of the camera coordinate system with respect to the world coordinate system, the focal length as the internal parameters, the image center coordinates, the scale A coefficient, a lens distortion coefficient, an image sensor interval, and the like are acquired in advance and stored in the image processing device 7.

  Then, the image processing device 7 calculates a relational expression between the pupil center coordinates in the image coordinate system detected based on the output data from the two stereo cameras 2a and 2b and the pupil center coordinates in the world coordinate system as calibration data. Get while referring to. Next, the image processing device 7 obtains the three-dimensional position coordinates in the world coordinate system of the pupil of the subject A from the two relational expressions. Similarly, the image processing apparatus 7 can obtain the three-dimensional positions of the left and right pupils of the subject A.

  Further, the image processing device 7 detects the gaze point of the subject on the display device 8 using the detected positions of the left and right corneal reflection points of the subject A and the positions of the left and right pupil centers. Hereinafter, a gaze point detection procedure by the image processing apparatus 7 will be described with reference to FIGS. 4 and 5. This gaze point detection procedure is a premise for performing the gaze point detection method of the present embodiment.

(Gaze point detection procedure)
Here, as shown in FIG. 4, based on the detected three-dimensional position P of the pupil, the center of the openings 9a and 9b of the stereo cameras 2a and 2b is set as the origin O, and the reference connecting the origin O and the pupil center P is used. A virtual viewpoint plane X′-Y ′ having the line OP as a normal is set. Here, the X ′ axis corresponds to a line of intersection between the X _W -Y _W plane of the world coordinate system and the virtual viewpoint plane X′-Y ′.

First, the image processing device 7 calculates a vector r _G from the corneal reflection point G to the pupil center P on the image plane S _G. Then, the vector r _G is converted into a vector r ′ converted to the actual size by using the magnification of the camera obtained from the distance OP. At this time, each stereo camera 2a, 2b is considered as a pinhole model, and it is assumed that the corneal reflection point G and the pupil center P are on a plane parallel to the virtual viewpoint plane X′-Y ′. That is, the image processing device 7 calculates the relative coordinate between the pupil center P and the corneal reflection point G as a vector r ′ on a plane parallel to the virtual viewpoint plane and including the three-dimensional coordinates of the pupil center P. This vector r ′ represents the actual distance from the corneal reflection point G to the pupil center P.

によって計算する。ここで、ベクトルｒ’は、角膜反射点Ｇから瞳孔中心Ｐに向かう瞳孔ベクトルの実測値を表す。 Thereafter, the image processing apparatus 7 regards the angle φ with respect to the horizontal axis X ′ of the straight line OT with respect to the gazing point T on the virtual viewpoint plane of the subject A as the angle φ ′ with respect to the horizontal axis X _G on the image plane of the vector r. Find as equal. Further, the image processing apparatus 7 uses the gain value k and the origin correction vector (offset vector) as the angle θ formed between the reference line OP and the line of sight vector of the subject A, that is, the vector PT connecting the pupil center P and the gazing point T. ) Using the function f ₁ using parameters including r ₀ , the following formula (1);

Calculate by Here, the vector r ′ represents an actual measurement value of the pupil vector from the corneal reflection point G toward the pupil center P.

Such calculation of the angles φ and θ is such that an enlarged vector r (= r′−r ₀ ) on the plane where the pupil center P exists corresponds to the gaze point of the subject A as it is. It is done by considering. More specifically, it is assumed that the angle θ of the visual line PT of the subject A with respect to the reference line OP has a linear relationship between the pupil center and the correction value | r′−r ₀ | of the corneal reflection distance. Note that the origin correction vector r ₀ included in the function f ₁ includes the cornea reflection-pupil center vector when the subject A looks at the camera (θ = 0) because the vector is not zero. A vector r ₀ is set as a vector between the reflection and the pupil center. Here, the gain value k and the origin correction vector r ₀ need to be calibrated because they differ depending on each subject A and the left and right eyeballs. Therefore, as the gain value k and the origin correction vector r _0, values corrected by parameter correction processing described later with respect to preset initial values are used.

Further, the image processing device 7 refers to the φ ₁ , φ ₂ , θ ₁ , and θ ₂ that are the angles φ and θ calculated corresponding to the camera images of the two stereo cameras 2 a and 2 b, and A gazing point on the screen of the display device 8 of A is detected. Here, in order to explain the mechanism of gaze point detection, a coordinate system as shown in FIG. 5 is defined. Two virtual viewpoint planes H ₁ and H ₂ having origins O ₁ ′ and O ₂ ′ corresponding to the positions of the two stereo cameras 2b and 2a, and a virtual viewpoint spherical surface with an arbitrary radius around the pupil center P Define S. The two virtual viewpoint planes H ₁ and H ₂ are planes perpendicular to the straight lines PO ₁ ′ and PO ₂ ′, respectively. The intersection of the intersection of the virtual viewpoint sphere S and a straight line (line of sight) passing through the fixation point Q on the pupil center P and the display screen and a straight line passing through the G _S, pupil center P and the origin O _{1 'and} the virtual viewpoint sphere S O ₁ , and the intersection of the virtual viewpoint spherical surface S and the straight line passing through the pupil center P and the origin O ₂ ′ is O ₂ . If the intersection of the line of sight PQ and the virtual viewpoint plane H ₁ is G ₁ , the angle formed by the horizontal axis of the straight line O ₁ ′ G ₁ and the virtual viewpoint plane H ₁ is φ ₁ . Similarly, when the intersection of the virtual viewpoint plane between H ₂ sight PQ and G _2, angle between the horizontal axis and the straight line O ₂ 'G ₂ and the virtual viewpoint plane H ₂ is phi _2. Further, on the virtual viewpoint sphere S, the angle formed line of intersection between the horizontal plane and the sphere S passing through the point O ₁ at the point O ₁ (curve) of the curve O ₁ G _S is equal to the angle phi _1. Similarly, the angle formed on the virtual viewpoint sphere S, the line of intersection of the horizontal plane passing through the point O ₂ at the point O ₂ and the virtual viewpoint sphere S (curve) of the curve O ₂ G _S is equal to the angle phi ₂ . As described above, the point _{P, O 1,} O 1 'is present on the same straight line _{L 1,} the point _{P, O 2,} O 2' so are present on the same straight line _{L 2,} the straight line _{L 1} The angle formed by the line of sight is θ ₁ , and the angle formed by the straight line L ₂ and the line of sight is θ ₂ .

The image processing apparatus 7 uses the relationship as described above to refer to the position coordinates of the origins O ₁ ′ and O ₂ ′ known in advance and the position and orientation data of the display device 8 on the screen. A gaze point can be calculated. That is, the relative positions of the points G _S , O ₁ and O ₂ on the virtual viewpoint spherical surface S from the angles φ ₁ , φ ₂ , θ ₁ and θ ₂ calculated from the camera images of the two stereo cameras 2a and 2b. Relationships can be acquired. Accordingly, the image processing apparatus 7 is a known origin O _{1 ',} O 2' and coordinates, from an already calculated pupil center P of coordinates, uniquely able to determine the line of sight PG _S, and the line of sight PG _S The gazing point Q can be detected by calculating the intersection with the screen of the display device 8. The angle phi _1, a sight PG _S obtained from theta _1, the angle phi _2, when the line of sight is determined from theta ₂ PG _S is misaligned calculating an average of the them as the final gaze vector You can also.

Here, the function f ₁ used by the image processing device 7 for calculating the line-of-sight direction includes a gain value k and an origin correction vector r ₀ as parameters. As can be seen from the above equation (1), the gain value k is linear in the magnitude of the vector r = r′−r ₀ after adjusting the cornea reflection-pupil center vector r ′ and the angle θ indicating the line-of-sight direction. It is a magnification used when an angle θ is obtained from the vector r assuming that there is a relationship. Ideally, if the angle θ and the vector | r ′ | have a linear relationship, the angle θ should be able to be calculated if only the gain value k is obtained. In other words, when the angle θ = 0, that is, when the subject A gazes at the camera, the vector | r ′ | = 0. However, in reality, the visual axis (line of sight) of the eyeball does not coincide with the optical axis, and the vector | r ′ | ≠ 0 when the angle θ = 0. Further, if the subject A changes, the vector | r | at the angle θ = 0 is different. The visual axis of the eyeball is a straight line that connects the fovea of the subject's eyeball and the subject's gaze point.

Hereinafter, a method for obtaining k and the vector r ₀ will be described. From the equation (1), the vectors θ ₁ and θ ₂ are respectively expressed by the following equations.

In equations (2) and (3), the origin correction vector r ₀ is uniquely determined for each eyeball, and thus is set to r ₀ regardless of the camera. The interval between the two cameras is represented by an angle and can be defined by the following equation (4).

From the equations (2) to (4), the following equation (5) is obtained, and the coefficient k (calibration value) of the equation (3) is calculated.

Here, since the positions of the two cameras are known, ∠O ₁ PO ₂ in equation (5) is always known. That is, the coefficient k can be calculated from the vector r ′ that is actually measured in each camera without the subject looking at a specific position.

Further, the image processing device 7 obtains an origin correction vector r ₀ by a one-point calibration method. The method for obtaining the origin correction vector r ₀ by the one-point calibration method is specifically as follows. First, the image processing apparatus 7 displays one target (specified point) at an arbitrary position on the display screen of the display device 8 and causes the subject A to gaze at the target. In this state, the image processing device 7 detects a gazing point on the projection virtual viewpoint plane. Next, the image processing apparatus 7 calculates the difference between the detected gazing point and the point at which the coordinates of the target are projected on the projection virtual viewpoint plane as a correction amount. Then, the image processing device 7 determines an origin correction vector r ₀ based on this correction amount. Thereby, the image processing apparatus 7 can perform high-precision gazing point detection according to Expression (1).

(Gaze point detection error correction method by head side bending)
The origin correction vector r ₀ used in the above-described gazing point detection procedure is a vector that is determined when the subject takes one posture, and when the subject's posture changes, the subject particularly bends the head. in the case where is, the direction of the origin correction vector r ₀ is changed. Therefore, in the gazing point detection method of the present embodiment, the image processing device 7 corrects the origin correction vector r ₀ according to the subject's head side bending.

Hereinafter, with reference to FIG. 6, a method for correcting the error of the gaze point due to the subject's head side bending will be described. 6, the one camera O ', the pupil center P _L and P _R of the left and right eyes of a subject A is captured. Both pupils are gazing at the gazing point Q on the viewing plane, and the intersection of the visual axis of both eyes and the viewing plane is coincident with the gazing point Q. In general, since there is a deviation between the visual axis and the optical axis, the intersection of the optical axis and the plane to be viewed is detected at a position different from the gazing point Q. In FIG. 6, the optical axes of the left and right eyeballs of the subject A are indicated by broken arrows. The angles formed by the camera-pupil vectors P _L O ′ and P _R O ′ and the left and right line-of-sight vectors P _L Q and P _R Q are represented by angle vectors θ _L and θ _R , respectively.

FIG. 7 shows the vector O′Q for the left eye and the right eye on one plane. When there is no deviation between the visual axis of the eyeball and the optical axis, θ _L (= k _L r _L ) and θ _R (= k _R r _R ) are in the gazing point direction from the camera O ′ in the world coordinate system. It is expressed as an angle vector toward. In other words, the pupil center P _L, when viewed from the P _R, the angle between the camera O 'and the fixation point Q is the angle theta _L, is detected as theta _R. However, in practice, since the deviation between the visual axis and the optical axis is present, it is actually detected, the pupil center P _L, when viewed from the P _R, and the optical axis direction of the camera O ' Is the angle formed by This angle is detected as vectors k _L r ′ _L and k _R r ′ _R. Therefore, _gaze point detection is performed by subtracting a vector obtained by multiplying the origin correction vectors r _0L and r _0R by the coefficients (k _L , k _R ) from the detected vectors k _L r ′ _L and k _R r ′ _R. .

Here, from Equation (1), k · r = k (r′−r ₀ ). By dividing both sides by a common gain value k, r = r′−r ₀ is obtained. Therefore, the relationship in the world coordinate system shown in FIG. 7 can be replaced with the relationship on the corneal reflection-pupil vector r plane. Further, assuming that the coefficients k _L and k _R are substantially the same value, and dividing each vector shown in FIG. 7 by k _L or k _R , as shown in FIG. 8, each vector is converted into a corneal reflection-pupil vector. It can be expressed on a plane. The representation of FIG. 8 is the simplest representation of each vector. In FIG. 8, the corneal reflection is detected at a position shifted from the original position due to the shift between the visual axis and the optical axis. The corneal reflection detected at such a shifted position is corrected by the origin correction vectors r _0L and r _0R .

Next, for convenience of explanation, FIG. 9 shows the translation of the starting points of the origin correction vectors r _0L and r _{0R in} FIG. 8 to the end points of the vectors r _L and r _R. The angle vector theta _L and theta _R in FIG. 6, in order to go to the same fixation point Q, the end point of the vector coincide. Therefore, it is considered that the end points of r _L and r _R , which are vectors obtained by dividing θ _L and θ _R by the coefficients k _L and k _R , are the same. Therefore, in FIG. 9, as the end point of the vector r _L and r _R are identical, they are moved parallel to each vector of both eyes.

  In general, when the subject's head tilts left and right, rotation about the visual axis of the eyeball that compensates for the tilt occurs. This rotation is called a rotational motion. The rotational movement is often measured by a goggle-type device, and in general, the rotational movement is measured as a rotation angle around the visual axis with respect to the head. However, in this specification, the angle of the rotational movement means the angle of rotation in the world coordinate system, not relative to the head.

Now, when the subject's head is tilted to the left by an angle γ _H , assuming that both eyeballs rotate about the visual axis by an angle γ, the origin correction vectors r _0L and r _0R are Rotate by γ. This is shown in FIG.

In FIG. 10, a point G represents a point on the r plane corresponding to the subject's gaze point. Hereinafter, this is simply referred to as a gaze point. The points a _L and a _R are the end points of the vectors r ′ _L and r ′ _R when the origin correction vectors r _0L and r _0R are not considered to have been rotated by the angle γ. Points a ′ _L and a ′ _R are the end points of the vectors r ′ _L and r ′ _R that are actually detected. The points g _L and g _R do not take into account that the origin correction vectors r _0L and r _0R have been rotated by an angle γ, and the points a ′ _L and a ′ _R are in the opposite direction to the origin correction vectors r _0L and r _0R. It is a point that has moved. That is, the points g _L and g _R are points calculated as gazing points when it is not considered that the origin correction vectors r _0L and r _0R are rotated by the angle γ.

Here, since it can be assumed that the direction of the origin correction vector r ₀ reflects the direction of deviation between the visual axis and the optical axis, r ₀ changes its magnitude with rotation around the visual axis. Should rotate without. In the case of taking an image of the eyeball while enlarging it, there is a method of directly measuring the eyeball rotation angle around the visual axis by using the brightness of an elliptical iris image with the pupil center as the origin. However, depending on the quality of the image can be acquired by the camera, it is difficult to perform such a method, it is impossible to determine the r ₀ for each eye.

Therefore, in this embodiment, when the subject is not buckled side the head, r ₀ of both eyes at the same angle gamma, providing constraint that rotates in the visual axis direction. Further, it is assumed that the visual axes of both eyes always pass through the camera (light source) (point O ′ in FIG. 6) and the pupils of both eyes are on the same plane only when obtaining the angle γ.

Hereinafter, a gaze point detection method using such a constraint condition will be described. First, before starting the gazing point detection process, the deviation between the optical axis and the visual axis when the subject takes one posture is calculated based on the coordinates of the corneal reflection point and the coordinates of the pupil center. Calculation step). Specifically, in order to calibrate the gazing point, the image processing device 7 presents a visual target whose coordinates at the center of the screen are known to the display device 8, and causes the subject to gaze at the visual target. As a result, the origin correction vector r ₀ and the coefficient k as the deviation are determined.

  Thereafter, a face image of the subject is generated using the stereo cameras 2a and 2b and the light sources 3a and 3b (face image generation step). Subsequently, based on the respective face images (for example, bright pupil image and dark pupil image) by the stereo cameras 2a and 2b, the corneal reflection point, which is the reflection point on the subject's cornea, of the light from the light sources 3a and 3b. The coordinates and the coordinates of the pupil center, which is the center of the pupil of the subject, are calculated (coordinate calculation step).

Next, the direction of the optical axis that is the central axis of the eyeball of the subject is calculated based on the coordinates of the corneal reflection point and the coordinates of the pupil center (optical axis direction calculation step). By this optical axis direction calculation step, the coordinates of the points a ′ _L and a ′ _{R in} FIG. 6 are obtained.

Next, based on the constraint that the rotation angle around the visual axis of the deviation between the optical axis and the visual axis is the same for the left eye and the right eye of the subject, the visual axis of the deviation between the optical axis and the visual axis A rotation angle around is calculated (rotation angle calculation step). Here, the deviation corresponds to the origin correction vector r ₀ described above. Next, the shift correction step will be described in detail.

Now, assume that the head is bent to the left and both eyes are rotated counterclockwise by the same unknown angle γ around the visual axis. Accordingly, the origin correction vectors r _0L and r _0R are both rotated by an angle γ, whereby the vectors r ′ _L and r ′ _R are changed (FIG. 10). That is, when correction of rotation due to head side bending is not performed, the gaze point (gaze direction) of the left and right eyes originally matched at the G position is shifted to the positions of the vectors g _L and g _R , respectively. Calculated.

Here, a matrix {γ} that is a rotation matrix of the angle γ is expressed by the following equation (6).

In this case, the gaze point of the right eye is expressed by the following expressions (7) and (8).

Similarly, since the gaze point of the left eye can be calculated, the vector g _R g _L is obtained by the following equation (9).

When the equation (9) is transformed, the following equation (10) is obtained.

Here, vectors A and B are defined as in the following equation (11).

In this case, the equation (10) can be written as the following equation (12).

Equation (12) means that the vector A is equal to the vector B when rotated by the angle γ. Therefore, the angle γ can be calculated by the following equation (13).

  In addition, from Formula (13), the sign of γ is unknown. However, the sign of γ can be determined based on the sign of the outer product of the vector A and the vector B, for example.

The origin correction vector {γ} r ₀ after correction is obtained by multiplying the origin correction vector r ₀ by the rotation matrix {γ} corresponding to the angle γ obtained in this way and correcting the rotation.

Next, the direction of the optical axis is calculated by correcting the direction of the optical axis based on the rotationally corrected deviation, and the point of gaze is detected based on the direction of the visual axis (gaze point detection step). Specifically, a correct gazing point is obtained using the origin correction vector {γ} r ₀ corrected by γ calculated by Expression (13). The rotated optical axis vector a ′ _R can be expressed by the following formula (14) obtained by transforming the formula (8).

The following equation (15) is finally derived from the equations (7) and (14). In equation (15), G _newR represents a correct gazing point after the rotation correction of the origin correction vector r _0R in the right eye.

  Note that the equation (12) may be displayed as components to obtain two equations, and the angle γ may be obtained based on at least one of these two equations. When displaying the component of Expression (12), for example, the first direction is a direction parallel to a straight line connecting the center of the pupil of the left eye of the subject and the center of the pupil of the right eye as the coordinate axis. The direction perpendicular to both the direction and the optical axis of the camera is taken as the second direction, and component display in these two directions of the first direction and the second direction can be made. In particular, the angle γ may be obtained by calculating using the component in the second direction.

(If the eyes of both eyes do not match)
At the stage of gazing point calibration before starting gazing point detection, the subject is staring at a very small point, so the gazing point of both eyes of the subject almost matches this very small point. . At this stage, the left eye and right eye origin correction vectors r _0L and r _0R are determined. However, after gazing point calibration, the convergence angle, which is the angle formed by the visual axes of both eyes of the subject, decreases and the gazing point of both eyes of the subject separates. The visual axes of the left and right eyes may intersect and the gazing points of the left and right eyeballs may be switched. In this case, the angle γ cannot be accurately calculated by the above-described method, and as a result, the gazing point of the right and left eyeballs cannot be accurately calculated.

Hereinafter, a method for calculating the angle γ and the gaze points of the right and left eyeballs with high accuracy even in such a case will be described. As shown in FIG. 11, the rotation angle of the subject's head when the subject's head rotates is obtained as an angle γ _H that is a rotation angle of a straight line connecting the left and right pupils as viewed from each camera.

FIG. 12 is a diagram corresponding to FIG. 10 when the gaze point of the subject's left and right eyes is separated. Note that the vectors r _L , r _R , r ′ _L , and r ′ _R are not shown in the figure because they are not necessary for the following calculation. Further, in the following description in the present specification, characters with the symbol “^” added to the upper part of the characters x, y in FIG. 12 are represented as “x ^” and “y ^”, respectively. In FIG. 12, x ^ axis and y ^ axis is an axis obtained by rotating the x-axis and y-axis by an angle gamma _H, respectively. The first direction and the second direction described above correspond to the x ^ -axis direction and the y ^ -axis direction, respectively.

In FIG. 12, the origin correction vectors r _0R and r _0L at the reference head position at the time of gazing point calibration are indicated by broken lines as in FIG. Further, similarly to the case shown in FIG. 10, it is assumed that the origin correction vectors r _0R and r _0L are rotated by the angle γ as the subject's head is rotated by the angle γ _H. At this time, it is assumed that the gazing points of the left and right eyes coincide at the time of gazing point calibration, and are separated by a vector G ′ after gazing point calibration. Here, it is assumed that the left and right pupils and the gazing points of the left and right eyes exist on the same plane in the three-dimensional space. It is also assumed that this assumption holds even when the subject tilts his head. In the following calculation, this assumption is used as a constraint condition. Under this assumption, the vector G ′ is parallel to the x ^ axis.

Now, in the xy coordinate system, the expressions (7) and (9) are changed to the following expressions (16) and (17), respectively.

  In Expression (17), the vector G ′ is an unknown two-dimensional vector, and the number of unknowns is increased by two compared to the case where the gaze points of the left and right eyes coincide with each other. Even under such conditions, the following method is used to obtain the angle γ and to estimate the gaze points of the left and right eyes.

First, as previously described, in FIG. 11, when setting up a new x ^ -y ^ coordinate system obtained by rotating the x-y coordinate system by an angle gamma _H, G 'is a vector of x ^ component only. As a result, the undetermined constant can be reduced by one.

Thereby, Formula (17) is changed into following Formula (18).

Note that the vector given the ^ is a vector when each is rotated by an angle gamma _H coordinate system counterclockwise.

Here, variables p, q, s, and t are defined as in the following equations (19) and (20).

In this case, the unknown value X is used as a vector G ′ ^ = (X, 0), and each term is displayed as a component by using the matrix {γ} of Equation (6), so that Equation (21) is obtained.

From Expression (21), two expressions of Expression (22) as the expression in the first direction and Expression (23) as the expression in the second direction are obtained.

Of these, since the unknown is only γ, the γ can be determined from the equation (23), and the gaze point of the left eye and the right eye is calculated using this γ and the equation (24). In addition, the distance between the two gazing points can be obtained.

  Further, by substituting γ determined from Expression (23) into Expression (22), an unknown number X corresponding to the distance between two gazing points can also be obtained.

Once γ is obtained, the right-eye gazing point G _newR can be calculated using the above-described equation (15). Similarly, it is possible to calculate the gaze point of the left eye.

(Experimental example)
The gaze point detection error due to the left and right side bending of the head was compared before and after the rotation correction by the correction method described above. The subjects were six university students, and the experiment was conducted with the subject's head on the chin rest at a position approximately 80 cm from the front of the display device. After gazing point calibration was performed in advance, 9 points of the target arranged in 3 rows and 3 columns on the screen were sequentially observed by the subject for 1 second. In addition, the angle of the side bending of the head is in a range of about 30 ° to 45 °, and is different for each subject.

  FIG. 13 shows gaze point detection errors in each subject before and after correction. The vertical axis represents a gazing point detection error (unit: °). In particular, in subjects A and B, the error was improved by performing the correction method of the present embodiment. In subjects C and D, the effect of the correction method of the present embodiment is not so much seen, and in subjects E and F, the error is large.

  FIG. 14 shows the distribution of gazing points before and after correction in the subject A. When the head is bent sideways, before correction, the left and right gazing points do not coincide as shown in FIG. 14A, and the gazing point is detected at a position shifted from the target. On the other hand, after correction, the left and right gaze points coincide as shown in FIG. With the correction method described above, in subject A, the gaze point detection error could be reduced by 28.2% with the left eye and 61.2% with the right eye. Note that the error could be reduced in the subject B as well.

  For subjects C and D, there was no significant difference in gazing point detection error before and after rotation correction. In subject C, although the head was bent sideways to the left and right, the intersection between the optical axis and the plane to be viewed did not change much depending on the presence or absence of side bending, and did not rotate significantly. As a cause of this, it is considered that when a human side-bends the head, a rotational motion that is an eyeball rotation around the visual axis occurs in the direction opposite to the rotation direction of the head. In such a subject C, it is considered that the rotation correction of the present embodiment is unnecessary.

  In subjects E and F, the error was larger after correction than before rotation correction. In the correction method of the present embodiment, it is assumed that the visual axes of both eyes coincide with each other on the screen. However, both subjects could not accurately calculate the rotation angle γ because the visual axis intersection of both eyes did not coincide on the point of sight even when the head was not rotating. This is considered to be one of the factors that increased the error.

  From the above results, in the subject considered to satisfy the constraint conditions such as the optical axis rotating in the same direction around the visual axis in both eyes, the visual axis intersection point of both eyes on the screen match, It can be said that the correction method of this embodiment is effective.

According to the rotation angle calculation method and the rotation angle calculation apparatus of the present embodiment, first, the direction of the optical axis of the eyeball of the subject is calculated based on the coordinates of the corneal reflection point and the coordinates of the pupil center. Also, the origin correction vector r ₀ is calculated as the deviation between the optical axis and the visual axis when the subject takes one posture. Then, based on the constraint that the rotation angle of the visual axis of the origin correction vector r ₀ is the same between the left and right eyes of the subject, the rotation angle of the visual axis of the origin correction vector r ₀ is calculated Is done. Thus, since the rotation angle around the visual axis of the origin correction vector r ₀ is calculated according to the posture of the subject person, the rotation around the visual axis of the origin correction vector r ₀ is highly accurate regardless of the orientation of the subject person. The angle can be calculated.

In the rotation angle calculation step, a direction parallel to a straight line connecting the center of the pupil of the left eye of the subject and the center of the pupil of the right eye is defined as the first direction (x ^ -axis direction), and the first direction and the camera The origin correction vector r ₀ may be calculated using the component in the second direction with the direction perpendicular to both of the optical axes as the second direction (y ^ -axis direction). As the nature of the human eye, the direction of the line of sight of the left eye and the right eye is difficult to leave along the second direction. Therefore, it is possible to improve the accuracy of calculation of the rotation angle when the direction of the line of sight of the left eye and the right eye of the subject is separated.

  The rotation angle calculation method and the gaze point detection method described above may be executed by a computer downloading a rotation angle calculation program or a gaze point detection program onto a memory and executing the program.

(Information input method)
According to the gazing point detection method described above, it is possible to detect the gazing point without depending on the rotation of the subject's head. Further, by calculating the unknown X from the equation (21), it is possible to calculate how far the gazing points of the left and right eyes are apart. Therefore, the following information input method can be implemented based on the above-described gaze point detection method.

  An information input device for implementing this information input method has a configuration substantially similar to that of the gazing point detection device shown in FIG. The image processing device 7 causes the display device 8 to display a plurality of characters arranged in a matrix, for example. The subject gazes at one character among the plurality of characters displayed on the display device 8 and inputs information corresponding to the watched character to an information processing terminal such as a personal computer. This information processing terminal may be the same information processing terminal as the image processing apparatus 7 or may be an information processing terminal different from the image processing apparatus 7.

  Next, the procedure of this information input method will be described. First, the gaze point of both eyes of the subject is detected by the above-described procedure (a binocular gaze point detection step). Thereby, the distance between the gazing points of both eyes of the subject can be detected.

  Subsequently, it is determined that an input has been made by the subject based on the distance between the gazing points of both eyes of the subject and the time during which this distance is within a predetermined range. As an example, when the distance between the gazing points of both eyes of the subject continues for a predetermined time (for example, 0.5 seconds) or longer for a predetermined time (for example, 8 mm) or less, the information processing terminal It is determined that input has been made. And the information corresponding to the character currently displayed on the screen of the display apparatus 8 at the subject's gaze point is input into an information processing terminal.

  Alternatively, the weighted points may be added for each frame in accordance with the magnitude of the absolute value of the distance between the gazing points of both eyes of the subject. For example, when the weighting value is w and the predetermined value is Dis, and Dis is 8 mm or less, w = 8−Dis is given. Then, it is added every time the gazing point is detected (for example, 60 times / second). When the added value exceeds a predetermined value (for example, 100), information is input to the information processing terminal. Here, when Dis exceeds 8 mm, the added value is returned to zero.

  Note that the shorter the distance between the gazing points of both eyes of the subject person, the shorter the time until the information processing terminal determines that the input by the subject person has been made.

  According to this information input method, information on the subject's eyes is detected based on the distance between the eyes of the detected eyes and the time during which the distance is within a predetermined range. Is determined to have been entered. Here, in general, when a human wants to input some information, the distance between the gazing points of both eyes tends to be narrow. On the other hand, when a person is wondering what to input, looking over the entire screen, or distracted, the distance between the gazing points of both eyes tends to increase. For this reason, according to said information input method, it can detect appropriately that the subject is going to input some information. For example, when the subject happens to be facing the display device 8, the subject's gaze point (for example, a binocular gaze point) even though the subject has no particular intention to input some characters. The character displayed at the point detected as the middle point) is prevented from being input against the intention of the subject. For this reason, a subject can perform input operation efficiently.

  The information input method described above may be executed by a computer downloading an information input program onto a memory and executing the program.

DESCRIPTION OF SYMBOLS 1 ... Gaze point detection apparatus, 2a, 2b ... Stereo camera (camera), 3a, 3b ... Light source, 7 ... Image processing apparatus (image processing part), A ... Subject, G ... Cornea reflection point, P ... Pupil center, Q: gazing point, r ₀ : origin correction vector (deviation).

Claims (12)

A face image generation step of generating a face image of the subject using two or more cameras and a light source that irradiates the subject with light;
Based on the respective face images from the two or more cameras, the coordinates of the corneal reflection point, which is the reflection point on the cornea of the subject, of the light from the light source, and the pupil center that is the center of the pupil of the subject A coordinate calculation step for calculating the coordinates of
An optical axis direction calculating step for calculating the direction of the optical axis that is the central axis of the eyeball of the subject based on the coordinates of the corneal reflection point and the coordinates of the pupil center;
Using the straight line connecting the fovea of the eyeball of the subject and the gazing point of the subject as a visual axis, the difference between the optical axis and the visual axis when the subject takes one posture is the corneal reflection point. A deviation calculation step for calculating based on the coordinates of and the pupil center coordinates;
The rotation angle around the visual axis is calculated based on the constraint that the rotation angle around the visual axis of the deviation between the optical axis and the visual axis is the same for the left eye and the right eye of the subject. A rotation angle calculating step;
A rotation angle calculation method comprising:   In the rotation angle calculating step, a direction parallel to a straight line connecting the center of the left eye pupil and the center of the right eye pupil of the subject is defined as a first direction, and both the first direction and the optical axis of the camera The rotation angle calculation method according to claim 1, wherein a rotation angle around the visual axis is calculated using a component in the second direction, with a direction perpendicular to the second direction as a second direction. The direction of the optical axis calculated in the optical axis direction calculation step in the rotation angle calculation method according to claim 1 or 2, wherein the direction of the optical axis around the visual axis calculated by the rotation angle calculation method according to claim 1 or 2 is used. Calculating the direction of the visual axis by correcting the shift based on a rotation angle, and detecting the gaze point of the subject based on the direction of the visual axis;
Gaze point detection method. A binocular gazing point detection step of detecting a gazing point of both eyes of the subject by the gazing point detection method according to claim 3;
Determining that an input has been made by the subject based on the distance between the gazing points of both eyes of the subject detected in the binocular gazing point detection step;
An information input method comprising: A gazing point detection device that detects a gazing point of the subject based on the face image of the subject,
Two or more cameras for acquiring the subject's face image;
A light source for irradiating the subject with light;
An image processing unit that processes image signals output from the two or more cameras;
With
The image processing unit
Based on the respective face images from the two or more cameras, the coordinates of the corneal reflection point, which is the reflection point on the cornea of the subject, of the light from the light source, and the pupil center that is the center of the pupil of the subject And the coordinates of
Based on the coordinates of the corneal reflection point and the coordinates of the pupil center, calculate the direction of the optical axis that is the central axis of the eyeball of the subject,
Using the straight line connecting the fovea of the eyeball of the subject and the gazing point of the subject as a visual axis, the difference between the optical axis and the visual axis when the subject takes one posture is the corneal reflection point. Based on the coordinates of and the pupil center coordinates,
The rotation angle around the visual axis is calculated based on the constraint that the rotation angle around the visual axis of the deviation between the optical axis and the visual axis is the same for the left eye and the right eye of the subject. Rotation angle calculation device.   When calculating the rotation angle around the visual axis, a direction parallel to a straight line connecting the center of the pupil of the left eye of the subject and the center of the pupil of the right eye is defined as the first direction, and the first direction and The rotation angle calculation apparatus according to claim 5, wherein a rotation angle around the visual axis is calculated using a component in the second direction with a direction perpendicular to both of the optical axes of the camera as a second direction.   The direction of the optical axis calculated by the rotation angle calculation device according to claim 5 or 6 is changed based on the rotation angle around the visual axis calculated by the rotation angle calculation device according to claim 5 or 6. A gaze point detection device that calculates the direction of the visual axis by correcting the visual axis and detects the gaze point of the subject based on the direction of the visual axis. Binocular gazing point detection means for detecting a gazing point of both eyes of the subject by the gazing point detection device according to claim 7;
An input determination unit that determines that an input is made by the subject based on a distance between the gazing points of both eyes of the subject detected by the binocular gazing point detection unit;
An information input device comprising: Computer
Face image generation means for generating a face image of the subject using two or more cameras and a light source that irradiates the subject with light;
Based on the respective face images from the two or more cameras, the coordinates of the corneal reflection point, which is the reflection point on the cornea of the subject, of the light from the light source, and the pupil center that is the center of the pupil of the subject And coordinate calculation means for calculating
An optical axis direction calculating means for calculating the direction of the optical axis that is the central axis of the eyeball of the subject based on the coordinates of the corneal reflection point and the coordinates of the pupil center;
Using the straight line connecting the fovea of the eyeball of the subject and the gazing point of the subject as a visual axis, the difference between the optical axis and the visual axis when the subject takes one posture is the corneal reflection point. And a rotational angle around the visual axis of the shift between the optical axis and the visual axis is the same for the left eye and the right eye of the subject. A rotation angle calculation means for calculating a rotation angle around the visual axis based on the constraint condition,
Rotation angle calculation program to function as.   The rotation angle calculation means sets a direction parallel to a straight line connecting the center of the pupil of the left eye of the subject and the center of the pupil of the right eye as a first direction, and both the first direction and the optical axis of the camera The rotation angle calculation program according to claim 9, wherein a rotation angle around the visual axis is calculated using a component in the second direction with a direction perpendicular to the second direction as a second direction. The direction of the optical axis calculated by the optical axis direction calculation means in the rotation angle calculation program according to claim 9 or 10 is calculated around the visual axis calculated by the rotation angle calculation program according to claim 1 or 2. Calculating the direction of the visual axis by correcting the shift based on a rotation angle, and detecting the gaze point of the subject based on the direction of the visual axis;
Gaze point detection program. A binocular gazing point detection means for detecting a gazing point of both eyes of the subject by the gazing point detection program according to claim 11;
Determining that an input has been made by the subject based on the distance between the gazing points of both eyes of the subject detected by the binocular gazing point detection means;
Information input program to function as.

Images