JP2019098024A - Image processing device and method - Google Patents

Abstract

To accurately and stably detect a sight line direction of a subject.SOLUTION: A sight line detection device 1 comprises: a camera 10 for imaging left and right pupils P, Pof an object person A and surrounding parts thereof, for acquiring eye images; and a calculation part 23 for calculating any optic axis of left and right pupils P, Pbased on the eye images, and correcting the optic axis to a preset deviation angle Δθand deviation direction Δφfor calculating a visual axis of the object person A, in which the calculation part 23 calculates an inclination of a line connecting three-dimensional coordinates of the left and right pupils P, Pin view along a reference line which is any of the calculated optic axis and calculated sight line, then corrects the deviation direction Δφbased on the inclination and calculates a visual axis using the corrected deviation direction Δφ.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing apparatus and an image processing method for processing an image of a person.

  In recent years, a gaze detection apparatus using a light source such as a near-infrared light source and a video camera is becoming popular (for example, see Patent Document 1 below). In such a gaze detection apparatus, a detection method called "pupil-corneal reflection method" is used. The “pupil-corneal reflex method” refers to the change in the positional relationship between the pupil and the corneal reflection in the image obtained by the video camera due to the change in the direction of the optical axis of the subject's eye with respect to the video camera. Is a method of detecting

  Generally, for the optical axis of the eye detected by the image acquired by the camera (for example, the target axis of the eye) and the visual axis passing through the gaze point and the central point of the eye (for example, pupil center or eye center) There is a gap. This deviation is different between individuals or between the left and right eyeballs. In the above apparatus, the origin correction vector is calculated in advance as a deviation between the subject's optical axis and the visual axis passing through the actual fixation point, and when the subject's head is inclined, the rotation angle of the origin correction vector around the visual axis is Based on the constraint that the left eye and right eye of the subject are identical, the origin correction vector is corrected by finding the rotation angle so that the fixation points of both eyes are close on a predetermined plane, and the origin correction vector is Use to calculate the subject's gaze direction.

  On the other hand, there is also known a gaze detection apparatus capable of correcting the origin correction vector by a different method (see Patent Document 2 below). In this device, the pupil contour is calculated using a pupil image obtained by imaging the subject's eye, and the rotation angle of the subject's eye around the visual axis is calculated based on the contour, and this rotation angle Use to correct the origin correction vector.

JP, 2015-169959, A JP, 2016-24616, A

  However, in the device described in Patent Document 1 described above, when the fixation point on the predetermined plane of both eyes of the subject is separated due to convergence open eye movement or the like, the state of the rotational angle of the eye can not be correctly recognized, and The solution of the correction value is not stable, and the line of sight to be detected tends to be unstable. Further, in the device described in Patent Document 2 described above, when the shape of the contour of the pupil can not be accurately detected due to the influence of the resolution of the camera, or when the eye of the subject is obliquely directed to the camera In this case, the calculation accuracy of the rotation angle of the eyeball tends to decrease.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus and an image processing method capable of stably and accurately detecting the rotation angle of the eyeball of a subject.

  In order to solve the above problems, an image processing apparatus according to an aspect of the present invention includes at least one camera for acquiring an eye image by imaging the left and right eyes of a subject and the peripheral part of the eye, and an eye image Calculating the optical axis or the line of sight of either of the left and right eyes based on the left and right of the right eye as viewed along the reference line which is either the calculated optical axis or the line of sight The inclination of a line connecting the three-dimensional coordinates of the feature points of the eye or the three-dimensional coordinates of the feature points at the peripheral portions of the left and right eyes is calculated, and the rotation angle of the inclination of the head of the subject is calculated based on the inclination.

  Alternatively, in the image processing method according to another aspect of the present invention, an image acquisition step of acquiring an eye image by imaging the left and right eyes of the subject and the periphery of the eye, and the left and right eyes based on the eye image Calculating at least one of the optical axis or the line of sight, and in the calculating step, three of the feature points of the left and right eyes when viewed along the reference line which is either the calculated optical axis or the line of sight The inclination of the line connecting the three-dimensional coordinates of the feature points of the peripheral portions of the left and right eyes is calculated, and the rotation angle of the inclination of the subject's head is calculated based on the inclination.

  According to the image processing apparatus or the image processing method of the above aspect, an optical axis or a line of sight of either of the left and right eyes is calculated based on the eye images of the left and right eyes of the object person. At that time, the inclination between the feature points of the left and right eyes of the object person or the feature points around the left and right eyes as viewed from the direction of the line of sight or the optical axis of the object person is calculated. The rotation angle of the head tilt is determined. This makes it possible to stably detect the rotational angle of the eye of the subject regardless of the positional relationship of the fixation points of the subject's eyes, and also to the performance of the camera or the camera of the eye of the subject. Regardless of the orientation, it is possible to accurately detect the rotation angle of the subject's eyeball.

  Here, the calculation unit may calculate the rotation angle of the inclination of the head when viewed along the reference line. According to this configuration, it is possible to accurately detect the rotation angle of the eyeball of the subject when viewed along the reference line.

  Further, the calculation unit calculates the line of sight by correcting the optical axis to the preset deviation angle and direction, corrects the deviation direction based on the inclination, and calculates the line of sight using the corrected deviation direction. , May be. In this way, the optical axis of either of the left and right eyes is calculated based on the eye images of the left and right eyes of the object person, and the line of sight is calculated by correcting the optical axis to the preset shift angle and shift direction. Ru. At that time, the inclination between the feature points of the left and right eyes of the subject or the feature points of the peripheral portions of the left and right eyes as calculated from the line of sight or the optical axis of the subject is calculated, and the line of sight is calculated. The deviation direction is corrected by the inclination. As a result, the line of sight can be stably detected regardless of the positional relationship of the fixation points of the subject's eyes and the line of sight regardless of the camera performance or the direction of the subject's eyes with respect to the camera. Can be accurately detected.

  In addition, the calculation unit includes a reference plane and a reference plane along a predetermined axis, and a face including the reference line and the three-dimensional coordinates of the left and right eye feature points or the three-dimensional coordinates of the left and right eye peripheral points. Set the orientation plane and calculate the inclination of the face orientation plane with respect to the reference plane as the inclination of the line connecting the three-dimensional coordinates of the left and right eye feature points or the three-dimensional coordinates of the left and right eye peripheral points. You may do it. In this case, the inclinations of the feature points of the left and right eyes of the subject or the feature points around the left and right eyes viewed from the direction of the line of sight or the optical axis of the subject can be easily calculated. The amount can be reduced.

  In addition, the calculation unit may set the inclination between the normal to the reference plane and the normal to the face direction plane, the three-dimensional coordinates of the feature points of the left and right eyes, or the three-dimensional coordinates of the feature points of the peripheral portions of the left and right eyes. It may be calculated as the slope of the connecting line. Also in this case, the inclinations of the feature points of the left and right eyes of the subject or the feature points of the peripheral portions of the left and right eyes viewed from the direction of the line of sight or the optical axis of the subject can be calculated more simply. Can further reduce the amount of computation for

  In addition, the calculation unit may be configured to calculate the three-dimensional coordinates of the feature points of the left and right eyes or the three-dimensional coordinates and the optical axis of the feature points of the peripheral portions of the left and right eyes When the line of sight is detected by calculating the initial value of the shift direction and the initial value of the inclination based on the three-dimensional coordinates of the feature points of the left and right eyes or the three-dimensional coordinates of the feature points of the peripheral portions of the left and right eyes The actual measured value of the inclination is calculated based on the optical axis, and the initial value of the deviation direction is corrected based on the value obtained by subtracting the initial value of the inclination from the measured value of inclination, and the sight line is used using the initial value of the deviation direction after correction. May be calculated. In this case, the initial values of the shift direction and the inclination can be accurately calculated using the visual target, and the line of sight can be accurately calculated based on the actual measurement values of the inclination thereafter.

  In addition, the camera includes two cameras for acquiring eye images, and the calculation unit calculates three-dimensional coordinates of feature points of the left and right eyes or peripheral parts of the left and right eyes based on the eye images acquired by the two cameras. The three-dimensional coordinates of the feature points may be calculated. In this case, the three-dimensional coordinates of the feature points of the left and right eyes or the three-dimensional coordinates of the feature points of the peripheral portions of the left and right eyes can be calculated with a simple configuration, and detection of the sight line can be realized with a simple configuration.

  In addition, the calculation unit may use any one of the pupil, the inside of the eye, the corner of the eye, an iris, a cornea ball, and a corneal reflex as feature points of the left and right eyes, and a binocular midpoint and an intercostal space as feature points of the peripheral portions of the left and right eyes The eyebrow, the nostril, the tip of the nose, the circumference of the nose, the corner of the mouth, or the center of the mouth may be detected.

  According to the present invention, it is possible to stably and accurately detect the rotation angle of the eyeball of the subject.

It is a perspective view showing a gaze detection device concerning an embodiment. It is a top view which shows the lens part of a camera. It is a figure showing hardware constitutions of an image processing device concerning an embodiment. It is a block diagram showing functional composition of a look detection apparatus concerning an embodiment. It is a flowchart which shows the basic operation | movement of the gaze detection apparatus which concerns on embodiment. It is a figure for demonstrating detection of a look. It is a figure which shows the detection state of the gaze point on the display apparatus in a conventional apparatus. FIG. 2 is a diagram showing a detection state of a pupil of a subject person A in a two-dimensional image acquired by a camera 10. FIG. 2 is a diagram showing a detection state of a pupil of a subject person A in a two-dimensional image acquired by a camera 10. FIG. 2 is a diagram showing a detection state of a pupil of a subject person A in a two-dimensional image acquired by a camera 10. It is a figure which shows the image which correct | amends angle vector V (theta) between the optical axis by the calculation part 23, and a visual axis. The initial value of the angle vector Vθ by the calculating section 23 and the left and right pupils P _L, is a diagram showing a calculation image of the slope of a line connecting the three-dimensional coordinates of P _R. 5 is a graph showing experimental results using the gaze detection apparatus 1;

  Hereinafter, preferred embodiments of an image processing apparatus and an image processing method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts will be denoted by the same reference symbols, without redundant description.

[Configuration of gaze detection device]
First, the configuration of the visual axis detection device 1 which is an image processing device according to the embodiment will be described with reference to FIGS. The gaze detection apparatus 1 is a computer system that detects the gaze direction of the target person by imaging the eye of the target person, and the gaze detection method according to the present embodiment is implemented by this device. The subject is a person who is a target for detecting the gaze direction, and can also be referred to as a subject. The purpose of using the gaze detection apparatus 1 and the gaze detection method is not limited at all. For example, the detection of the driver looking away, the confirmation of the safety confirmation operation of the driver's side mirror and the rearview mirror, the driver's drowsiness detection, and the product interest The line-of-sight detection device 1 can be used for examination of the degree, data input to a computer used for an amusement device or the like, a diagnostic device for autism diagnosis of an infant or the like, or the like.

As schematically shown in FIG. 1, the visual axis detection device 1 includes a pair of cameras 10 functioning as a stereo camera and an image processing device 20. Hereinafter, if necessary, it distinguishes a pair of camera 10, and the left camera 10 _L on the left side of the subject A, in the right camera 10 _R on the right side of the subject A. In the present embodiment, the visual axis detection device 1 further includes the display device 30 which the target person A views, but the purpose of use of the visual axis detection device 1 is not limited as described above. The object is not limited to the display device 30, but may be, for example, a windshield of a car. Therefore, the display device 30 is not an essential element in the gaze detection device 1. Each camera 10 is connected to the image processing apparatus 20 wirelessly or by wire, and various data or commands are transmitted and received between the camera 10 and the image processing apparatus 20. Camera calibration is performed on each camera 10 in advance.

  The camera 10 is used to capture a part including the left and right eyes of the subject person A and their peripheral parts. The pair of cameras 10 are arranged at predetermined intervals along the horizontal direction, and the face of the subject A for the purpose of preventing reflection of reflected light in the face image when the subject A is wearing glasses It is provided at a lower position. The elevation angle of the camera 10 with respect to the horizontal direction is set, for example, in the range of 20 to 35 degrees in consideration of both the reliable detection of the pupil and the avoidance of the obstruction of the visual field range of the subject A. Camera calibration is performed on each camera 10 in advance.

  In the present embodiment, the camera 10 is an NTSC camera which is one of interlace scan methods. In the NTSC system, one frame of image data obtained for 30 sheets per second is composed of an odd field composed of odd-numbered horizontal pixel lines and an even field composed of even-numbered horizontal pixel lines. And an even field image are alternately captured at intervals of 1/60 seconds. Thus, one frame corresponds to a pair of odd and even fields. The camera 10 captures an image of the subject A in response to an instruction from the image processing device 20 and outputs image data to the image processing device 20.

  The lens portion of the camera 10 is schematically shown in FIG. As shown in this figure, in the camera 10, the objective lens 11 is accommodated in the circular opening 12, and the light source 13 is attached to the outside of the opening 12. The light source 13 is a device for emitting illumination light toward the eye of the target person A, and includes a plurality of light emitting elements 13a and a plurality of light emitting elements 13b. The light emitting elements 13 a are semiconductor light emitting elements (LEDs) having a central wavelength of output light of 850 nm, and are arranged in a ring along the edge of the opening 12 at equal intervals. The light emitting element 13b is a semiconductor light emitting element whose center wavelength of output light is 940 nm, and is arranged in a ring shape at equal intervals outside the light emitting element 13a. Therefore, the distance from the optical axis of the camera 10 to the light emitting element 13b is larger than the distance from the optical axis to the light emitting element 13a. The respective light emitting elements 13 a and 13 b are provided to emit illumination light along the optical axis of the camera 10. The arrangement of the light sources 13 is not limited to the configuration shown in FIG. 2 and may be another arrangement as long as the camera can be regarded as a pinhole model. The light source 13 emits illumination light at a timing according to an instruction from the image processing device 20.

  The image processing device 20 is a computer that executes control of the camera 10 and the light source 13 and detection of the gaze direction of the subject A. The image processing apparatus 20 may be configured by a stationary or portable personal computer (PC), may be configured by a workstation, or may be configured by another type of computer. Alternatively, the image processing apparatus 20 may be constructed by combining a plurality of arbitrary types of computers. When using a plurality of computers, these computers are connected via a communication network such as the Internet or an intranet.

  A general hardware configuration of the image processing apparatus 20 is shown in FIG. The image processing apparatus 20 includes a CPU (processor) 101 that executes an operating system and application programs, a main storage unit 102 configured by a ROM and a RAM, and an auxiliary storage unit 103 configured by a hard disk and a flash memory. And a communication control unit 104 configured by a network card or a wireless communication module, an input device 105 such as a keyboard or a mouse, and an output device 106 such as a display or a printer.

  Each functional element of the image processing apparatus 20 described later loads predetermined software on the CPU 101 or the main storage unit 102, and operates the communication control unit 104, the input device 105, the output device 106, etc. under the control of the CPU 101. This is realized by reading and writing data in the main storage unit 102 or the auxiliary storage unit 103. Data and databases necessary for processing are stored in the main storage unit 102 or the auxiliary storage unit 103.

  As shown in FIG. 4, the image processing apparatus 20 includes a lighting control unit 21, an image acquisition unit 22, and a calculation unit 23 as functional components. The lighting control unit 21 controls the lighting timing of the light source 13. The image acquisition unit 22 is a functional element that acquires image data (data of an eye image) from the camera 10 by controlling the imaging timing of the camera 10 in synchronization with the lighting timing of the light source 13. The calculation unit 23 is a functional element that detects the direction of a visual axis (also referred to as a line of sight) based on a line-of-sight vector obtained from image data. The visual axis (line of sight) is a line connecting the pupil center of the subject and the gaze point of the subject (the point at which the subject is looking). The term "visual axis" includes the meaning (concept) of the starting point, the end point, and the direction. Further, the “line-of-sight vector” is a vector representing the direction of the visual axis of the subject, and is a form representing the “direction of the visual axis”. The output destination of the direction of the visual axis of the detection result of the image processing apparatus 20 is not limited at all. For example, the image processing apparatus 20 may display the determination result in the form of an image, a figure, or a text on a monitor, or may store the determination result in a storage device such as a memory or a database. It may be sent.

[Basic operation of gaze detection method]
Next, the basic operation of the gaze detection method (image processing method) by the gaze detection apparatus 1 will be described using FIGS. 5 and 6. FIG. 5 is a flowchart showing the basic operation of the gaze detection method, and FIG. 6 is a diagram for explaining the detection of the gaze by the gaze detection apparatus 1.

  First, the lighting control unit 21 controls the lighting timing of the light source 13, and the image acquiring unit 22 acquires a bright pupil image (eye image) and a dark pupil image (eye image) from each camera 10 according to the timing Step S11: Image acquisition step). Subsequently, the calculation unit 23 targets the bright pupil image and the dark pupil image from each camera 10, and the position of the pupil center (feature point of the eye) in the eye image of each camera 10 and the position of the corneal reflection Is detected (step S12). Then, the calculation unit 23 calculates three-dimensional coordinates of the pupil centers on the left and right of the subject person A (step S13). Furthermore, the calculation unit 23 calculates the optical axes of the left and right pupils based on the calculated position of the pupil center and the position of the corneal reflection in the eye image of any one of the cameras 10 and the three-dimensional coordinates of the left and right pupils Step S14). Here, the “optical axis” is an optical axis of an eyeball detected by an image acquired by the camera 10, and coincides with, for example, a target axis of the eyeball. Thereafter, the calculation unit 23 calculates a shift angle vector indicating a shift angle and a shift direction between the optical axis and the visual axis in the left and right pupils (step S15). Furthermore, the calculator 23 calculates the visual axis by correcting the optical axis using the calculated displacement angle vector, and then calculates the gaze point on the predetermined visual symmetry plane with reference to the calculated visual axis Step S16, calculation step above). The above process is repeatedly executed until an instruction to end the sight line detection process is received (step S17).

  The process of each step will be described in detail below.

(Acquisition of eye image)
The light entering the eye is diffusely reflected by the retina, and of the reflected light, the light passing through the pupil has the property of returning to the light source with strong directivity. When the camera is exposed when the light source near the camera aperture emits light, a part of the light reflected by the retina enters the aperture, so it is possible to obtain an image in which the pupil is brighter than the periphery of the pupil . This image is a bright pupil image. On the other hand, if the camera is exposed when the light source located at a distance from the camera aperture emits light, the light returned from the eye hardly returns to the camera aperture, so an image with a dark pupil appears It can be acquired. This image is a dark pupil image. In addition, when light with a wavelength with high transmittance is irradiated to the eye, reflection of light at the retina increases, so the pupil appears bright, and when light with a wavelength with low transmittance is irradiated to the eye, light is reflected in the retina The pupils appear dark because they are reduced.

  In the present embodiment, the light emitting element 13a emitting light of a wavelength with high transmittance (the center wavelength is 850 nm) is provided at a position adjacent to the opening 12, and light of a wavelength with low transmittance of the eye (the center wavelength is 940 nm) The light emitting element 13 b emitting light is provided at a position away from the opening 12. The lighting control unit 21 and the image acquisition unit 22 light the light emitting element 13a in accordance with the odd field of the camera 10 to capture a bright pupil image, and light the light emitting element 13b in accordance with the even field of the camera 10 to perform the dark pupil Take an image. Furthermore, the image acquisition unit 22 slightly shifts the operation timing between the two cameras 10, and the exposure time of each camera 10 is set to less than the shift time. The lighting control unit 21 causes the corresponding light emitting element 13 a and the light emitting element 13 b to emit light alternately during the exposure time of each camera 10, so that the light from the light source 13 of one camera 10 becomes the image of the other camera 10. Do not affect (do not create crosstalk).

  The image acquisition unit 22 acquires a bright pupil image and a dark pupil image obtained by the series of control. Since the obtained image data has effective pixels only in the odd field or even field, the image acquiring unit 22 embeds the luminance average of the pixel lines of the adjacent effective pixels in the pixel values between the lines. Generate a bright pupil image or a dark pupil image. The image acquisition unit 22 outputs the bright pupil image and the dark pupil image obtained by the two cameras 10 to the calculation unit 23.

(Detection of position of corneal reflex and position of pupil center)
The calculation unit 23 detects corneal reflection in the left and right eyes from each of the bright pupil image and the dark pupil image input from the image acquisition unit 22. Specifically, the calculating unit 23 performs binarization and labeling by a P tile method on one image, and selects corneal reflections in the left and right eyes from the image based on information such as shape and luminance average. Do. By such processing, the calculation unit 23 obtains corneal reflection in the left and right eyes from each of the bright pupil image and the dark pupil image.

  Furthermore, the calculation unit 23 calculates the movement amount of the corneal reflection between the bright pupil image and the dark pupil image as the position correction amount based on the calculated position of the corneal reflection. Subsequently, the calculation unit 23 shifts the image of the previous field (i-th field) to the image of the next field ((i + 1) -th field) by the position correction amount so that the positions of the two corneal reflections coincide. Then, a difference image is generated from these two images. Then, the calculation unit 23 acquires the coordinates (position) on the image of the corneal reflection that has been made to coincide.

  Subsequently, the calculation unit 23 identifies left and right pupil positions from the difference image. Specifically, calculation unit 23 makes use of the fact that the luminance does not change significantly with the previous frame, and uses the luminance average of the pupils detected in the previous frame to set a difference of half the average luminance as a threshold The image is binarized and labeled. Subsequently, the calculation unit 23 selects a pupil from connected components of labeled pixels based on shape parameters such as pupil-like area, size, area ratio, squareness, and pupil feature amount, and Calculate the coordinates (position) of the pupil center.

(Calculation of 3D coordinates of left and right pupils)
Subsequently, the calculation unit 23 obtains three-dimensional coordinates of the left and right pupil centers. Specifically, the calculation unit 23 calculates the three-dimensional positions of the left and right pupil centers from the coordinates of the left and right pupil centers calculated using the difference images acquired from the two cameras 10 using the stereo method. With stereo method, internal parameters such as camera lens focal length, image center, and pixel size, and external parameters such as camera position and attitude are measured in advance, and objects are photographed with multiple stereo cameras. Then, based on the coordinates of the point in the image, the internal parameter and the external parameter are used to determine the spatial position of the point. Specifically, the calculating unit 23 calculates the relationship between the coordinates of the pupil center in the image coordinate system detected based on the image data from the two cameras 10 and the coordinates of the pupil center in the world coordinate system in the three-dimensional space. An equation is obtained with reference to calibration data. Next, the calculation unit 23 obtains three-dimensional coordinates of the pupil center of the subject A in the world coordinate system from the relational expression.

(Calculation of optical axis)
The calculation unit 23 calculates the optical axis based on the three-dimensional coordinates of the pupil centers on the left and right, the position of the pupil center on the difference image, and the position of the corneal reflection on the difference image. The calculating unit 23 may detect the optical axis of the eye based on the position of the pupil center of either of the left and right eyes and the position of corneal reflection of either of the left and right eyes. The optical axes of both eyes may be detected based on the positions of both pupil centers of the left and right eyes and the positions of corneal reflexes of both the left and right eyes. In the following, detection of the optical axis of one of the eyes will be described as an example.

As shown in FIG. 6, based on the three-dimensional position P of the pupil center, a virtual viewpoint with the center of the opening 12 of the camera 10 as an origin O and a reference line OP connecting the origin O and the pupil center P as a normal Consider a plane X'-Y '. The virtual viewpoint plane X′-Y ′ corresponds to a projection plane (image plane) of an image captured by the camera 10. Here, X 'axis corresponds to the line of intersection between X _W -Z _W plane and the virtual viewpoint plane of the world coordinate system.

Calculator 23 calculates a vector r _G from corneal reflection G in the image plane S _G to the pupil center P, and the vector r _G, was converted to the actual size with the magnification of the camera obtained from the distance OP vector Convert to r At this time, each camera 10 is considered to be a pinhole model, and it is assumed that the corneal reflection G and the pupil center P are on a plane parallel to the virtual viewpoint plane X'-Y '. That is, on the plane parallel to the virtual viewpoint plane and including the three-dimensional coordinates of the pupil center P, the calculation unit 23 calculates the relative coordinates of the pupil center P and the corneal reflection point G as a vector r. This represents the actual distance from the corneal reflection point G to the pupil center P.

Subsequently, the calculating unit 23 calculates the inclination φ of the straight line OT ′ with respect to the horizontal axis X ′ with respect to the intersection T ′ with the optical axis on the virtual viewpoint plane of the subject A as the horizontal axis X _G on the image plane of the vector r. It is assumed that it is equal to the slope φ 'for. Furthermore, the calculation unit 23 calculates the parameter including the gain value k, the angle θ formed by the vector of the optical axis of the subject A, that is, the vector PT ′ connecting the pupil center P and the intersection T ′, and the reference line OP. Calculated by the following equation (1) used.
θ = f ₁ (r) = k × | r | (1)

  Such calculation of the angles φ and θ is performed by assuming that the vector r on the plane on which the pupil center P exists is enlarged on the virtual viewpoint plane as it corresponds to the intersection T ′ with the optical axis of the object person A as it is. To be done. More specifically, it is assumed that the angle θ of the vector PT 'of the optical axis of the subject A with respect to the reference line OP has a linear relationship between the pupil center and the distance | r | of the corneal reflection.

  By using the assumption that the angle θ and the distance | r | can be linearly approximated and the assumption that the two inclinations φ and φ 'are equal, (θ, φ) and (| r |, φ') can be 1 It can be made to correspond to a pair. Then, the calculation unit 23 can obtain the vector PT ′ of the optical axis of the subject A based on the calculated (θ, φ).

(Calculation of deviation angle vector between the optical axis and the visual axis)
Before describing the calculation procedure of the shift angle vector by the calculation unit 23, the necessity of calculation of the shift angle vector will be described.

  In the case where the light source 13 and the camera 10 are at substantially the same position, for example, when the subject A looks at the camera 10, if there is symmetry of the eye, the pupil in the image acquired by the camera 10 The corneal reflex should be visible in the center of. However, in actuality, it is common that the optical axis and the visual axis deviate in the eye of the subject person A. Such deviation between the optical axis and the visual axis differs between individuals or between the left and right eyeballs. That is, when the gaze point on the virtual viewpoint plane of the subject A is T, the vector PT of the visual axis deviates from the vector PT 'of the optical axis (FIG. 6).

The conventional gaze detection apparatus calculates a correction vector r ₀ for correcting the vector r so that the optical axis calculated when the target person A sees the specified point coincides with the visual axis, and detects it on the image The following equation r;
r = r '-r ₀
, And the vector PT of the visual axis is detected using the corrected vector r.

However, even when the correction vector r ₀ is determined once for the subject A, the head of the subject A is inclined in the left-right direction while the sight line is directed in the same direction with respect to the device (hereinafter referred to as “side In the case of “bending”), since the direction of displacement between the visual axis and the optical axis changes, the visual axis can not be accurately detected from the optical axis. Specifically, as shown in FIG. 7A, when the inclination of the lateral bending of the subject A at the time of calculating the correction vector r ₀ and the inclination of the lateral bending at the time of visual axis detection are the same, The visual axes detected in both eyes coincide at the gaze point Q on the display device 30 in order to correct the deviation of the optical axis and the visual axis in each of the eyes correctly. On the other hand, as shown in FIG. 7B, when the inclination of lateral bending at the time of visual axis detection changes from the inclination of lateral bending of the subject A when the correction vector r ₀ is calculated, direction deviation of the visual axis also visual axis (optical axis) in the case of using as a correction vector r ₀ to changes around the detected visual axis in both eyes do not match the gaze point Q.

  In this embodiment, the detection accuracy of the visual axis is changed by changing the direction in which the visual axis deviates from the optical axis based on the inclination of the lateral bending of the subject person A viewed from the optical axis (or visual axis) direction. Try to raise However, there is a limit to simply obtaining the inclination of the lateral bending of the subject A from the positions of the eyes of the subject A on the image obtained by the camera 10. That is, in the case where the camera 10 acquires a two-dimensional image of the subject's eye from a direction oblique to the gaze direction of the subject A (for example, from a direction shifted up and down), the two-dimensional image It becomes difficult to distinguish between the state of lateral bending of A and the state in which the subject person A rotates about a direction perpendicular to the viewing direction (hereinafter simply referred to as “rotation”).

  The detection state of the pupil of the subject person A in the two-dimensional image acquired by the camera 10 will be described with reference to FIGS. 8 to 10.

As shown in FIG. 8, the camera 10 R of the two in front of the subject _A, 10 _L is Yes installed, the subject A are gazing at the left camera 10 _L, viewed from the left camera 10 _L target who inclination of lateral bending of the head a and the pupil P _L and the right pupil P _R of the left rather than to assume when positioned in a horizontal line. When this time paying attention to the pupil P _L of the left of the subject A, so that the visual axis V passes through the center and left camera 10 _L of the pupil P _L. Be detected by the calculating unit 23 of the image processing apparatus 20 is an optical axis E of the pupil P _L, the deviation of the visual axis V relative to the optical axis E, the virtual as a plane perpendicular to the left camera 10 _L as the visual axis V setting plane VP _1, the angle (deviation angle) [Delta] [theta] ₀ of the optical axis E and the visual axis V, the straight line LN ₁ connecting the two intersection points on the virtual plane VP ₁ of the optical axis E and the visual axis V virtual It can be expressed as inclination (displacement direction) Δφ ₀ on the plane VP ₁ . In other words, the inclination (displacement direction) [Delta] [phi ₀ is the plane PLA ₁ passing through the two cameras ₁₀ L, 10 _R and the pupil _{P L,} the plane PLA ₂ containing both the optical axis E and the visual axis V It is also an angle. At this time, the right pupil P _R is also located on the plane PLA ₁ . Although the reference line of this inclination Δφ ₀ is not limited to a specific one, it may be, for example, a horizontal axis LN ₀ connecting two cameras 10 _L and 10 _R. In the present embodiment, the shift of the visual axis V with respect to the optical axis E is expressed by an angle vector (shift angle vector) Vθ = (Δθ ₀ , Δφ ₀ ). In the present embodiment, when representing a visual axis V virtual illustrates a planar VP ₁ at the angle vector V [theta] _V as a reference, the optical axis E of the virtual plane VP ₁ at the angle vector V [theta] _E with a reference, V [theta] _{V =} V [theta] Expressed as _E −Vθ, vθ is defined as an angle vector (deviation angle vector) Vθ = (Δθ ₀ , Δφ ₀ ). Then, in the present embodiment, as will be described later, the value Δφ ₀ of the angle vector Vθ is to be corrected according to the state of the side bend of the subject person A.

Here, considering that the head of the subject A rotates and the right pupil P _R of the subject A moves on the plane PLA ₁ , lateral bending of the subject A does not occur. The shift of the visual axis V does not change. At the same time, the height from the horizontal axis LN ₀ of the right pupil P _{R with} respect to the left pupil P _L in the two-dimensional image viewed from the camera 10 _L does not change.

On the other hand, as shown in FIG. 9, the inclination angle of the lateral bending is changed to be inclined in the left position of the left pupil P _L is the right pupil P _R as it around the visual axis V of the subject A alpha Assume a state of rotating by _two . In this case, the optical axis E rotates around the visual axis V by the change angle α ₂ of the inclination of the side bending. As a result, the angle vector Vθ detected by the calculation unit 23 becomes (Δθ ₀ , Δφ ₀ + α ₂ ). That is, when the state of FIG. 8 is used as a reference, a plane PLA ₁ including the camera 10 _L and the left and right pupils P _L and P _R in the head posture of the target person A serving as the reference The angle made with the plane PLA ₃ after the change of is the rotation angle of the optical axis E around the visual axis V. At this time, in the two-dimensional image viewed from the camera 10 _L, the height from the horizontal axis LN ₀ of the right pupil P _{R with} respect to the left pupil P _L changes, so the left pupil P _L on the two-dimensional image it is possible to detect the angle alpha ₂ by calculating the rotational angle of the right pupil P _R, as a result, angle vector _{Vθ = (Δθ 0, Δφ 0} + α 2) can be correctly calculated.

Furthermore, it can be considered similarly when the subject A gazes at the right camera 10 _R , and when the two pupils P _L and P _R and the two cameras 10 _L and 10 _R are on the same plane, There is no rotation about the visual axis V of the optical axis E. Thus, for the left pupil P _L, the right camera 10 _R or with either of the two-dimensional image of the left camera 10 _L, the pupil P _R of right around the viewing axis V of the left pupil P _L is rotated The angle can be measured and an angle vector Vθ can be calculated based thereon. Regard right pupil P _R, can be calculated angle vector Vθ similarly using the two-dimensional image.

Next, as shown in FIG. 10, downwardly while keeping parallel to the line of the camera ₁₀ L, 10 _R connecting the camera ₁₀ L, 10 _R with respect to the visual axis V of the subject A from the state shown in FIG. 8 Assume the case of moving. In FIG. 10, the position of the camera before moving is indicated by reference numerals 10 _L ′ and 10 _R ′. In this case, it can be considered to be the same as changing the visual axis V upward with respect to the height of the cameras 10 _L and 10 _R.

First, position 10 _L of the original camera ', 10 _R' is located right of the pupil P _R on the plane PLA ₁ containing and left pupil P _L, then the head of the subject A is rotated results, given that the right pupil P _R becomes moved backward as viewed on the plane PLA ₁ from the left camera 10 _L and the position of the pupil P _{R '.} At this time, the right pupil P _R ′ is no longer present on the plane PLA ₄ including the two cameras 10 _L and 10 _R and the left pupil P _L. This means that, from the viewpoint of the left camera 10 _L, it is impossible to distinguish between the state of rotation of the head of the subject A and the inclination of lateral bending of the head of the subject A as shown in FIG. means. As described above, it is difficult to detect the rotation angle of the optical axis E around the visual axis V of the target person A only with the two-dimensional images captured by the cameras 10 _L and 10 _R.

  Therefore, the calculation unit 23 according to the present embodiment corrects and calculates the angle vector Vθ between the optical axis and the visual axis as follows.

That is, as illustrated in FIG. 11, the calculation unit 23 causes the subject A to gaze at the left and right pupils (feature points of the eye) in a state in which the target person A gazes at the calibration specified point (target) PO displayed on the display device 30. The three-dimensional position of P _L and P _R and the optical axis E ₀ of the left pupil P _L are calculated. Subsequently, the computing unit 23 sets the virtual plane VP ₀ perpendicular to the optical axis _{E 0,} and the optical axis _{E 0,} based on the visual axis _{V 0} which is determined by the position of the specified point PO and left pupil _{P L} , Calibrate the initial value Vθ of the angle vector (Δθ ₀ , Δφ ₀ ). The horizontal axis of the virtual plane VP _0, the virtual plane VP ₀ and corresponds to _X W -Z _W plane intersection line in the world coordinate system (hereinafter, the same in another virtual plane). Furthermore, the calculation unit 23 determines the slope (initial value of the slope) α ₀ of the line connecting the three-dimensional coordinates of the left and right pupils P _L and P _R when viewed from the direction along the optical axis E ₀ which is a reference line. calculate. The inclination alpha ₀ is the angle of rotation of the right pupil P _R around the optical axis E _0. Specifically, the calculation unit 23 projects projection points PJ _L , which are coordinate-transformed on the virtual plane VP ₀ by projecting the three-dimensional coordinates of the left and right pupils P _L and P _R onto the virtual plane VP ₀ , respectively. Regarding PJ _R , the inclination α of the line connecting the three-dimensional coordinates of the left and right pupils P _L and P _R by finding the inclination of a line connecting two projection points PJ _L and PJ _R from the horizontal line on the virtual plane VP ₀ Calculate ₀ . The calculating unit 23 may calculate an inclination of a line connecting the three-dimensional coordinates of the left and right pupils P _L and P _R when viewed from the direction of the reference line along the calculated visual axis V ₀ .

Furthermore, the calculation unit 23 calculates the three-dimensional positions of the left and right pupils P _L and P _R and the optical axis E ₁ of the left pupil P _{L at} the timing at which the visual axis of the object person A is actually calculated. Subsequently, the computing unit 23 again sets the virtual plane perpendicular VP ₂ to the optical axis _{E 1.} Furthermore, the calculation unit 23 determines the inclination (measured value of inclination) α ₁ of the line connecting the three-dimensional coordinates of the left and right pupils P _L and P _R when viewed from the direction along the optical axis E ₁ which is a reference line. calculate. Specifically, the calculation unit 23, left and right pupils _P L, _P projection points are coordinate transformation on the virtual plane VP ₂ by the respective three-dimensional coordinates projected onto the virtual plane VP ₂ of _{R PJ} L, For PJ _R , the inclination α of the line connecting the three-dimensional coordinates of the left and right pupils P _L and P _R by finding the inclination of the line connecting the two projected points PJ _L and PJ _R from the horizontal line on the virtual plane VP ₂ Calculate ₁ The calculation unit 23 is viewed from the reference line with the direction along the visual axis once calculated based on the initial value Vθ = (Δθ ₀ , Δφ ₀ ) of the angle vector and the optical axis E ₁ as a reference line. The inclination of a line connecting the three-dimensional coordinates of the left and right pupils P _L and P _R may be calculated.

The calculation unit 23 calculates the left and right pupils P _L, based on the gradient alpha ₁ of a line connecting the three-dimensional coordinates of P _R, correcting the initial value of the angle vector Vθ the angle vector Vθ for visual axis calculation Do. Specifically, the calculation unit 23 subtracts the inclination alpha ₀ from the inclination alpha _1, the initial value [Delta] [phi ₀ of inclination of the angle vector Vθ the subtracted value (α ₁ -α ₀₎ based on the correction by the following formula Do.
Δφ = Δφ ₀ + (α ₁ −α ₀ )
Thus, the calculation unit 23 determines the angle vector Vθ = (Δθ ₀ , Δφ) used when detecting the visual axis. The calculation of the inclination α ₀ and the initial values Δθ ₀ and Δφ ₀ of the angle vector Vθ in a state in which the target person A gazes at the specified point PO for calibration may be performed once at the time of the first visual axis detection. These values may be repeatedly used in subsequent detection of a plurality of visual axes.

Details of a method of calculating the inclination of a line connecting the initial value of the angle vector Vθ and the three-dimensional coordinates of the left and right pupils P _L and P _R by the calculation unit 23 will be described with reference to FIG.

The calculation unit 23 can detect the initial value Δθ ₀ of the formed angle by calculating the angle formed by the visual axis V and the optical axis E obtained when the target person gazes at the specified point PO. Further, the calculation unit 23, to the outer product vector between the normal vector _{N S1} and the optical axis E of the plane PLA ₄ passing through the camera ₁₀ L, 10 _R detects the subject left pupil _{P L} and two of the visual axis A straight line LN ₂ passing through the left pupil is set, and a plane obtained by rotating the plane PLA ₄ around the straight line LN ₂ by an angle η is set as a plane PLA ₁ . This angle η is determined as the angle between the plane PLA ₄ and the optical axis E. The plane PLA ₁ set in this manner includes the optical axis E which is a reference line, and is a reference plane along the straight line LN ₂ . The calculation unit 23, by calculating the angle between the normal vector _{N P} of the plane PLA ₂ containing the normal vector _{N S2} plane PLA ₁ viewing axis V and the optical axis E, the inclination initial value of Δφ ₀ can be detected.

Further, the calculation unit 23, left and right pupils _P L, _P inclination alpha ₀ of a line connecting the three-dimensional coordinates of _R, the alpha _1, with respect to the plane PLA _1, the left and right pupil _P L, and _{P R} and optical axis E It can be obtained by calculating the angle (inclination) formed by the included plane (face direction plane). Specifically, the calculation unit 23 calculates the angle between the normal vector of N _S2 and the face direction plane of plane PLA _1.

  (Calculation of visual axis and gaze point)

The calculation unit 23 obtains the visual axis V as a vector PT by correcting the optical axis E obtained by the calculation using the angle vector Vθ corrected by the above method. Furthermore, the calculation unit 23 obtains a gaze point Q which is an intersection point of the viewing axis V and the viewing target plane (display device 30) by the following equation.
Q = nPT + P

  According to the line-of-sight detection device 1 described above and the line-of-sight detection method using the same, based on the images of the left and right pupils of the subject A, the three-dimensional positions of the left and right pupils E is calculated, and the visual axis V is calculated by correcting the optical axis to the predetermined deviation angle and deviation direction. At that time, the inclination between the right and left pupils of the subject A as viewed from the direction along the visual axis or the optical axis of the subject A is calculated, and the shift direction for calculating the visual axis is corrected by the inclination . This makes it possible to stably detect the line of sight regardless of the positional relationship of the fixation points of both eyes of the subject A, and relates to the performance of the camera 10 or the direction of the eye of the subject A relative to the camera. The line of sight can be detected accurately.

  Here, the calculating unit 23 of the image processing apparatus 20 calculates the inclination of two planes by calculating the inclination of a line connecting the three-dimensional coordinates of the left and right pupils. In this way, the inclinations of the right and left pupils of the subject A as viewed from the line of sight of the subject A or the direction along the optical axis can be easily calculated, so that the amount of calculation for sight line detection can be reduced. it can. Furthermore, the calculation unit 23 calculates the inclination of the normals of the two planes as the inclination of a line connecting the three-dimensional coordinates of the left and right pupils. By doing this, the inclinations of the right and left pupils of the subject A as viewed from the line of sight of the subject A or the direction along the optical axis can be calculated more simply, thereby further reducing the amount of calculation for sight line detection. be able to.

  In addition, the calculation unit 23 sets the initial value of the direction of displacement of the visual axis with respect to the optical axis based on the three-dimensional coordinates of the left and right pupils and the optical axis when the target person A looks at the target at the preset position. And the initial value of the inclination of the line connecting the three-dimensional coordinates of the left and right pupils is calculated. Thereafter, when detecting the line of sight, the calculation unit 23 calculates the actual measurement value of the inclination based on the three-dimensional coordinates of the pupil of the subject A and the optical axis, and subtracts the initial value of the inclination from the actual measurement value of the inclination. Based on the above, the initial value of the shift direction is corrected, and the line of sight is calculated using the corrected initial value of the shift direction. In this case, the initial values of the shift direction and the inclination can be accurately calculated using the visual target, and the line of sight can be accurately calculated based on the measured values of the inclination thereafter.

In the present embodiment, two cameras 10 _L and 10 _R are used. With such a configuration, the three-dimensional coordinates of the left and right pupils can be calculated with a simple configuration, and detection of the sight line can be realized with a simple configuration.

  Here, the effect of the gaze detection by the gaze detection apparatus 1 according to the present embodiment will be specifically described.

  FIG. 13 shows an experimental result using the sight line detection device 1. In the experiment, a digital camera provided with a 16 mm lens and a visible light cut filter as the camera 10, a near infrared LED light source with a central wavelength of 850 nm and a near infrared LED light source with a central wavelength of 940 nm as the light source 13 A 19 inch display was used. Place five subjects A to E at a position approximately 80 cm from the front of the screen of the display device 30 as a subject of the experiment, and place 25 marks in order at positions evenly arranged in 5 rows and 5 columns on the screen I displayed it and watched one point at a time. Then, the visual axis detection device 1 is operated to detect the fixation point, and the average visual angle error [deg] is measured from the detection result. FIG. 13 (a) shows the average viewing angle error for each subject A to E and the average left eye of the whole subject, and FIG. 13 (b) relates to the average right eye for each subject A to E and the whole subject The average viewing angle error is shown. In each of the measurement results, there is no inclination of the head of the test subjects A to E (tilt of the lateral bending), and there is an inclination of the head of the test subjects A to E, and the correction processing of the angle vector Vθ is operated. The results in the case where there is no inclination of the head of the test subjects A to E and the correction processing of the angle vector Vθ is operated are shown separately.

  From this result, the average visual angle error of the left eye is 0.90 ± 0.33 deg, the average visual angle error of the right eye is 1.10 ± 0.41 deg when the head inclination does not occur, and the head inclination is The average viewing angle error of the left eye is 1.58 ± 0.61 deg and the average viewing angle error of the right eye is 1.73 ± 0.59 deg when the correction processing is not operated, and the head tilt is When the correction process was operated, the average visual angle error of the left eye was 0.95 ± 0.57 deg, and the average visual angle error of the right eye was 1.22 ± 0.62 deg. Thus, it was found that the error was reduced by about 30.9% with the left eye and about 27.6% with the right eye when corrected with respect to the case without correction. The results with correction were almost the same as the results without head tilt. Although the effect of the correction differs depending on the subject, since the method of the present embodiment corrects the deviation between the visual axis and the optical axis, the subject with a large deviation between the visual axis and the optical axis has an error. It seems that the effect of mitigation is large.

The present invention is not limited to the embodiments described above. For example, in the above embodiment, a stereo camera including two cameras 10 _L and 10 _R is used as the camera 10, but one camera such as a TOF (Time-of-Flight) camera is used for the target person A A camera capable of detecting three-dimensional coordinates of the pupil may be used. In addition, even if the TOF camera is not used, one feature camera is included in the two pupils P _L and P _R of the subject A to image other feature points of the subject A such as a nostril, and the result is obtained. The three-dimensional position of the two pupils P _L and P _R may be detected using the positions on the three-point image.

In the above embodiment, the visual axes are calculated using the left and right pupils P _L and P _R as feature points of the left and right eyes of the subject person A, but the present invention is not limited to this. In addition, as the feature points of the left and right eyes, any one of the inside of the eye, the back of the eye, the iris, the cornea, the corneal reflection, or a combination thereof may be detected.

  For example, the calculation unit 23 may detect the pupil center, the iris center, or the corneal sphere center, or may detect their ends. The corneal sphere center can be detected by giving the corneal radius in advance. The position of reflection on the surface of the corneal sphere may be detected.

  The calculation unit 23 may obtain three-dimensional coordinates of the feature points such as the inner and outer corners of the eye and the like, or may obtain three-dimensional coordinates of both middle points. Furthermore, the slope of the straight line determined by the least squares method may be calculated based on the positions of the four eyes of the eyes and the corners of the eyes. Furthermore, a black part in an eye image may be detected as an eye part, and the luminance gravity centers may be detected as the positions of feature points.

  Further, in the above embodiment, instead of the feature points of the left and right eyes of the subject person A, the three-dimensional coordinates of the feature points of the peripheral portions of the left and right eyes of the subject person A are detected. The inclination angle of the side bend of the subject person A may be calculated. As such a peripheral part of the eye, any one of a binocular middle point, intercostal space, eyebrow, nostril, nose tip, circumference of nose, mouth corner, mouth center, or a combination thereof is exemplified. When detecting the tip of the nose, intercostal space, center of the mouth, etc. as the feature points of the peripheral part of the subject A's eyes, the calculating unit 23 sets the horizon to the horizon when considering the straight line connecting the left and right pupils. A line that is perpendicular to and passes through the feature points can instead be determined and the slope can be calculated for the face direction plane containing this line.

A: target person, 1: eye gaze detection device, 10, 10 _L , 10 _R : camera, 13: light source, 20: image processing device, 21: lighting control unit, 22: image acquisition unit, 23: calculation unit, E, E ₀ , E ₁ ... optical axis, P _L , P _R ... pupil, PO ... prescribed point (target), Q ... fixation point, V, V ₀ ... visual axis, Vθ ... angle vector, Δθ ₀ ... shift angle, Δφ ₀ ... shift direction.

Claims (9)

At least one camera for acquiring an eye image by imaging the left and right eyes of the subject and the periphery of the eye;
A calculation unit that calculates an optical axis or a line of sight of one of the left and right eyes based on the eye image;
The calculation unit may select three-dimensional coordinates of feature points of the left and right eyes or feature points of peripheral portions of the left and right eyes when viewed along a reference line which is either the calculated optical axis or the line of sight. The inclination of a line connecting three-dimensional coordinates is calculated, and the rotation angle of the inclination of the head of the subject is calculated based on the inclination.
Image processing device. The calculation unit calculates a rotation angle of the inclination of the head when viewed along the reference line.
An image processing apparatus according to claim 1. The calculation unit calculates the sight line by correcting the optical axis to a preset shift angle and shift direction, corrects the shift direction based on the inclination, and uses the corrected shift direction. Calculate the line of sight,
The image processing apparatus according to claim 1. The calculation unit includes: a reference plane including the reference line and extending along a predetermined axis; three-dimensional coordinates of the reference line and the feature points of the left and right eyes or three-dimensional coordinates of feature points of peripheral portions of the left and right eyes Setting the inclination of the face direction plane with respect to the reference plane, and connecting the three-dimensional coordinates of the feature points of the left and right eyes or the three-dimensional coordinates of the feature points of the peripheral portions of the left and right eyes Calculated as the slope of the line,
The image processing apparatus according to any one of claims 1 to 3. The calculation unit may determine an inclination between the normal line of the reference plane and the normal line of the face direction plane, the three-dimensional coordinates of the feature points of the left and right eyes, or the feature points of peripheral parts of the left and right eyes. Calculated as the slope of a line connecting dimensional coordinates,
The image processing apparatus according to claim 4. The calculation unit is configured to: three-dimensional coordinates of feature points of the left and right eyes or three-dimensional coordinates of feature points of peripheral portions of the left and right eyes when the target person is caused to look at a target at a preset position. When calculating the initial value of the shift direction and the initial value of the inclination based on the optical axis and detecting the sight line, three-dimensional coordinates of the feature points of the left and right eyes or peripheral parts of the left and right eyes The actual measurement value of the inclination is calculated based on the three-dimensional coordinates of the feature points of the feature point and the optical axis, and the initial value in the deviation direction is corrected based on a value obtained by subtracting the initial value of the inclination from the actual measurement value of the inclination. Calculating the sight line using an initial value of the deviation direction after correction;
The image processing apparatus according to claim 3. It has two cameras for acquiring the eye image,
The calculation unit calculates the three-dimensional coordinates of the feature points of the left and right eyes or the three-dimensional coordinates of the feature points of the peripheral part of the left and right eyes based on the eye images acquired by the two cameras. ,
The image processing apparatus according to any one of claims 1 to 6. The calculation unit is any one of the pupil, the inner eye, the corner of the eye, an iris, a corneal ball, and a corneal reflex as feature points of the left and right eyes, and a binocular midpoint as a feature point of peripheral portions of the left and right eyes. Detect any of the intercostal region, eyebrows, nostrils, nose tip, nose circumference, mouth corners, mouth center,
The image processing apparatus according to any one of claims 1 to 7. An image acquisition step of acquiring an eye image by imaging the left and right eyes of the subject and the periphery of the eye;
Calculating an optical axis or a line of sight of one of the left and right eyes based on the eye image;
In the calculating step, the three-dimensional coordinates of the feature points of the left and right eyes or the feature points of the peripheral part of the left and right eyes when viewed along the reference line which is either the calculated optical axis or the sight line The inclination of a line connecting three-dimensional coordinates is calculated, and the rotation angle of the inclination of the head of the subject is calculated based on the inclination.
Image processing method.

Images