JP2018099174A - Pupil detector and pupil detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a pupil detector that improves detection accuracy of a position of a pupil while simplifying a configuration of an optical system.SOLUTION: A pupil detector includes: an optical system including a camera 2 for photographing a target person's face and light sources 3a, 3b for illuminating the face; and a detection unit 13 for calculating three-dimensional positions of two pupils of the target person on the basis of a face image output by the camera 2. The detection unit 13 includes: a feature point detection unit 17 that on the basis of a position of each of the two pupils detected on the face image and a direction of an optical axis of the target person's eyeball, calculates a position on the image of a rotation center of each of the target person's two eyeballs; and a face posture detection unit 18 and line-of-sight detection unit 19 that calculate three-dimensional positions of the two pupils on the basis of the positions on the image of the rotation centers of the two eyeballs and positions on the image of the target person's nostrils.SELECTED DRAWING: Figure 3

Description

  One embodiment of the present invention relates to a pupil detection device and a pupil detection method.

  Conventionally, a target person's face image is acquired using one camera and a light source provided in the vicinity of the imaging lens of the camera, and the two-dimensional coordinates of the left and right pupil centers and There is a known method of detecting the face posture based on the three-dimensional coordinates of the left and right pupil centers and the three-dimensional coordinates of the left and right nostrils centers, which are detected based on the two-dimensional coordinates of the nostrils. (See Patent Document 1 below). According to this method, the three-dimensional position of each of the left and right pupils and the three-dimensional coordinates of the midpoint between the left and right nostrils can be calculated, and the normal of the plane passing through those three points is detected as the head direction. The center of gravity of these three points can be detected as the head position. In addition, this method has been developed as a method for predicting the movement of feature points such as pupils or nostrils in a time-series image frame with high accuracy so that the subject's head moves. A method for improving the robustness of feature point tracking is known (see Patent Document 2 below).

  On the other hand, in a device that detects the line of sight of the subject, the line of sight and gaze point are calculated using the two-dimensional position of the pupil and the two-dimensional position of the corneal reflection detected based on the face image acquired by one camera. In order to achieve this, a configuration is known in which two cameras are further provided to detect the three-dimensional position of the pupil (see Patent Document 3 below).

Japanese Patent No. 4343149 Japanese Patent No. 5429885 Japanese Patent No. 4517049

  However, in the methods described in Patent Document 1 and Patent Document 1 described above, when the subject's line of sight changes, the detection accuracy of the three-dimensional position of the pupil tends to decrease. That is, when the line of sight changes, the positional relationship between the two points of the two pupils to be detected and the nostrils as viewed from the camera changes, so that the three-dimensional position of the pupil detected using the face image acquired by the camera Errors are likely to occur. According to the configuration described in Patent Document 3 described above, the detection accuracy of the three-dimensional coordinates of the pupil can be maintained by using two cameras, but the configuration of the optical system tends to be complicated.

  The present invention has been made in view of the above problems, and provides a pupil detection device and a pupil detection method capable of further improving the detection accuracy of the position of the pupil while simplifying the configuration of the optical system. Objective.

  In order to solve the above-described problem, a pupil detection device according to an aspect of the present invention includes an optical system including an imaging device that captures an image of a subject's face and an illumination device that illuminates the face, and an image output by the imaging device. A processing unit that calculates a three-dimensional position of the two pupils of the subject based on the position of each of the two pupils detected on the image and the direction of the optical axis of the eyeball of the subject Based on the rotation center calculation unit for calculating the position on the image of the rotation center of each of the two eyeballs of the subject, the position on the image of the rotation center of the two eyeballs, and the image on the nostril of the subject And a pupil position calculation unit that calculates a three-dimensional position of the two pupils based on the position.

  Alternatively, in the pupil detection method according to another aspect of the present invention, the face of the subject is illuminated by the illumination device while the face of the subject is illuminated by the illumination device, and the subject's 2 is detected based on the image output by the imaging device. Calculating a three-dimensional position of two pupils, and the step of calculating the three-dimensional position is based on the positions of the two pupils detected on the image and the direction of the optical axis of the eyeball of the subject. And calculating the position of the rotation center of each of the two eyeballs of the subject on the image, the position of the rotation center of the two eyeballs on the image, and the position of the subject's nostril on the image. Calculating a three-dimensional position of the two pupils.

  According to the pupil detection device or the pupil detection method of the above aspect, the respective rotation centers of the two eyeballs based on the respective positions of the two pupils on the image acquired by the optical system and the direction of the optical axis of the eyeball The positions of the two pupils are calculated, and the three-dimensional positions of the two pupils are calculated based on the calculated positions on the images of the rotation centers of the two eyeballs and the positions of the nostrils on the image. As a result, even if the subject's line-of-sight direction changes, the positional relationship between the three rotation centers of the two eyeballs and the nostril is maintained, so that highly accurate pupil position detection is realized. In addition, the number of imaging devices in the optical system can be reduced, and the configuration of the optical system is simplified.

  Here, in the pupil detection device of the above-described form, the rotation center calculation unit calculates the light of the eyeball based on the position of each of the two pupils and the position on the image of the corneal reflection that occurs corresponding to each of the two pupils. It is preferable to obtain the direction of the axis and calculate the position of the rotation center on the image based on the direction of the optical axis of the eyeball. In this case, the position on the image of the center of rotation of the eyeball is calculated from the position on the image of the pupil and the position on the image of corneal reflection, and the position of the pupil can be detected with high accuracy by simple calculation.

  The rotation center calculation unit obtains the direction of the optical axis of the eyeball based on the ellipticity related to each of the two pupils detected on the image, and the position of the rotation center on the image based on the direction of the optical axis of the eyeball. It is also preferable to calculate. Also in this case, the position of the center of rotation of the eyeball on the image is calculated based on the direction of the optical axis obtained using the ellipticity related to the pupil on the image, and the position of the pupil can be detected with high accuracy. Become.

  In addition, the processing unit obtains the direction of the optical axis of the eyeball based on the position of each of the two pupils and the position on the image of the corneal reflection corresponding to each of the two pupils as the rotation center calculation unit, The first processing unit that calculates the position of the center of rotation on the image based on the direction of the optical axis of the eyeball, and the direction of the optical axis of the eyeball based on the ellipticity of each of the two pupils detected on the image And a second processing unit that calculates a position on the image of the center of rotation based on the direction of the optical axis of the eyeball. The processing result by the first processing unit and the processing result by the second processing unit are obtained. In consideration of this, it is also preferable to calculate the three-dimensional positions of the two pupils. In this case, by adding the calculation result using the position of the cornea reflection on the image and the calculation result using the ellipticity of the pupil detected on the image, more accurate detection of the position of the pupil is realized.

  In addition, the processing unit obtains the direction of the optical axis of the eyeball based on the position of each of the two pupils and the position on the image of the corneal reflection corresponding to each of the two pupils as the rotation center calculation unit, The first processing unit that calculates the position of the center of rotation on the image based on the direction of the optical axis of the eyeball, and the direction of the optical axis of the eyeball based on the ellipticity of each of the two pupils detected on the image And a second processing unit that calculates the position of the center of rotation on the image based on the direction of the optical axis of the eyeball, and the processing result of either the first processing unit or the second processing unit is obtained. It is also preferable to calculate the three-dimensional positions of the two pupils by selecting and using them. In such a configuration, more accurate pupil position detection is realized by selecting and using the calculation result using the position of the corneal reflection on the image and the calculation result using the ellipticity of the pupil detected on the image. Is done.

  Further, when the corneal reflection corresponding to one pupil of the subject is detected and the corneal reflection corresponding to the other pupil of the subject is not detected, the processing result of the first processing unit and the second processing unit Of these, it is preferable to preferentially use the processing result of the first processing unit related to one pupil. As a result, more accurate pupil position detection is realized.

  Further, the pupil position calculation unit is configured to determine the three-dimensional position of the rotation center of the two eyeballs and the three-dimensional of the nostril based on the position on the image of the rotation center of the two eyeballs and the position on the image of the nostril of the subject. It is also preferable to detect the face direction of the subject person by calculating the position. With this configuration, the face direction of the subject can be detected with high accuracy.

  Further, the image processing apparatus further includes a line-of-sight detection unit that obtains the direction of the subject's line of sight based on the direction of the optical axis of the eyeball, and the pupil position calculation unit includes the position of the rotation center of the two eyeballs on the image, The face direction of the subject is detected by calculating the three-dimensional position of the center of rotation of the two eyeballs and the three-dimensional position of the nostril based on the position on the image. It is also preferable to determine which of the two directions the subject's line-of-sight direction is based on the direction. Thereby, it is possible to detect the line of sight with high accuracy.

  Furthermore, at least two systems of optical systems are provided, and the processing unit has a first distance between the two pupils and a first distance between the pupils and the nostrils based on images output from the at least two systems of optical systems. It is preferable to measure the two distances and calculate the three-dimensional positions of the two pupils using the first distance and the second distance. Thereby, it is possible to detect a line of sight with high accuracy for a target person located in a wide range.

  According to the present invention, it is possible to further improve the detection accuracy of the pupil position while simplifying the configuration of the optical system.

It is a top view which shows the pupil detection apparatus which concerns on 1st Embodiment. It is a block diagram which shows the hardware constitutions of the image processing apparatus 1 of FIG. It is a block diagram which shows the function structure of the image processing apparatus 1 of FIG. It is a top view which shows the eyeball model used as the premise of the calculation by the feature point detection part 17 of FIG. It is a figure which shows the image image imaged with the camera 2 aiming at the eye part of the subject person A modeled with the eyeball model of FIG. It is a block diagram which shows the function structure of 1 A of image processing apparatuses which comprise the pupil detection apparatus of 2nd Embodiment. Parts (a) and (b) are diagrams for explaining the principle of the pupil shape method executed by the line-of-sight detection unit 19A of FIG. It is a figure for demonstrating the principle of the pupil shape method which 19 A of gaze detection parts of FIG. 6 perform. It is a block diagram which shows the function structure of the image processing apparatus 1B which comprises the pupil detection apparatus of 3rd Embodiment. 4 is a plan view showing a state of a subject A with respect to the optical system 4. FIG. 4 is a plan view showing a state of a subject A with respect to the optical system 4. FIG.

Hereinafter, a preferred embodiment of a pupil detection device according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted. Each drawing is made for the purpose of explanation, and is drawn so as to particularly emphasize the target portion of the explanation. Therefore, the dimensional ratio of each part in the drawings does not necessarily match the actual one.
[First Embodiment]

  First, the configuration of the pupil detection device 10 according to the first embodiment will be described with reference to FIGS. FIG. 1 is a plan view of the pupil detection device 10, FIG. 2 is a block diagram showing a hardware configuration of the image processing device 1 in FIG. 1, and FIG. 3 shows a functional configuration of the image processing device 1 in FIG. It is a block diagram. The pupil detection device of the present embodiment is a pointing device that moves the cursor on the monitor screen of a personal computer by detecting the movement of the subject's line of sight, the sleepiness of detecting the sleepiness of the driver by monitoring the movement of the pupil or the line of sight It is used as a detection system, a diagnosis system for diagnosing autism or the like by detecting the movement of the subject's pupil or line of sight.

  As shown in FIG. 1, the pupil detection device 10 includes a single camera (imaging device) 2 that captures a face image of the subject A and the face of the subject A provided in the vicinity of the imaging lens on the front surface 2 a of the camera 2. An optical system (optical system) 4 including a light source (illumination device) 3a that illuminates the subject 2 and a light source (illumination device) 3b that is provided at a position away from the front surface 2a of the camera 2 and illuminates the face of the subject A And an image processing apparatus 1 connected to the light sources 3a and 3b. The image processing apparatus 1 calculates the three-dimensional positions of the left and right pupils of the subject A based on the face image of the subject A by the camera 2 and uses the gaze direction or gazing point of the subject A as a basis. It is a calculating part (processor) which detects the subject's A face posture (face direction).

  The camera 2 is not limited to a specific type as long as it is an imaging device that can generate a face image of the subject A, but an image sensor such as a CCD or CMOS is used in that the image data can be processed with high real-time properties. Use the built-in digital camera. The camera 2 is arranged so that the subject A faces the front surface 2 a of the camera 2.

  The light source 3a is configured to be able to irradiate illumination light having a near-infrared light component toward a range covering the subject A located on the optical axis L1 along the optical axis L1 of the camera 2. The light source 3b is fixed at a position where the distance from the optical axis L1 is further away from the light source 3a, and irradiates illumination light having a near-infrared light component toward a range covering the subject A along the optical axis L1. It is configured to be possible. Here, the illumination light irradiated from the two light sources 3a and 3b is set so as to have different wavelength components (for example, the center wavelengths are 850 nm and 950 nm) that cause a luminance difference in the pupil part, and the light source 3b may be fixed at a position where the distance from the optical axis L1 is equal to the light source 3a. In addition, the light sources 3a and 3b are not limited to being configured by a single light emitting element, and a plurality of light emitting elements may be arranged in a predetermined shape (circular shape, ring shape, or the like). In this case, the configuration of the light source can be simplified and reduced in size while causing a luminance difference in the pupil portion. However, the light sources 3a and 3b do not necessarily have to emit illumination light having different wavelength components, and the light sources 3a and 3b do not have to be ring-shaped.

  The camera 2 and the light sources 3a and 3b are provided for the purpose of preventing the reflected light from being reflected in the face image when the subject A is wearing glasses and making it easier to detect the nostril of the subject A. It is preferable to be provided at a position lower than the face height (for example, a position where the inclination angle of the optical axis L1 with respect to the horizontal plane is 20 to 35 degrees and the optical axis L1 faces the direction of the subject A).

  The image processing apparatus 1 controls imaging by the camera 2 and irradiation of illumination light by the light sources 3a and 3b, and based on the head image of the subject A acquired by the camera 2, the tertiary of the pupil of the subject A A process of detecting the original position, the line-of-sight direction and gazing point of the subject A, and the face posture of the subject A is executed.

  Hereinafter, the configuration of the above-described image processing apparatus 1 will be described in detail with reference to FIGS. 2 and 3. The image processing apparatus 1 may be a computer that executes control of the camera 2 and the light sources 3 a and 3 b and processing of a face image acquired by the camera 2. The image processing apparatus 1 may be constructed by a stationary or portable personal computer (PC), may be constructed by a workstation, or may be constructed by another type of computer. Alternatively, the image processing apparatus 1 may be constructed by combining a plurality of arbitrary types of computers. When a plurality of computers are used, these computers are connected via a communication network such as the Internet or an intranet.

  As illustrated in FIG. 2, the image processing apparatus 1 includes a CPU (processor) 101, a main storage unit 102, an auxiliary storage unit 103, a communication control unit 104, an input device 105, and an output device 106. . The CPU 101 executes an operating system, application programs, and the like. The main storage unit 102 includes a ROM and a RAM. The auxiliary storage unit 103 is configured by a hard disk, a flash memory, or the like. The communication control unit 104 includes a network card or a wireless communication module. The input device 105 includes a keyboard and a mouse. The output device 106 includes a display, a printer, and the like. The input device 105 and the output device 106 may be touch panel displays.

  Each functional element of the image processing apparatus 1 to be described later reads predetermined software on the CPU 101 or the main storage unit 102, and operates the communication control unit 104, the input device 105, the output device 106, and the like under the control of the CPU 101. This is realized by reading and writing data in the main storage unit 102 or the auxiliary storage unit 103. Data and a database necessary for processing are stored in the main storage unit 102 or the auxiliary storage unit 103.

  As illustrated in FIG. 3, the image processing apparatus 1 includes a camera drive unit 11, a lighting control unit 12, and a detection unit (processing unit) 13 as functional components. The camera drive unit 11 is a functional element that controls the shooting timing of the camera 2. Specifically, control is performed so that the camera 2 repeatedly captures images at a predetermined frame rate and a predetermined exposure time, and alternately acquires a bright pupil image and a dark pupil image as a face image. The bright pupil image is a face image in a state where the pupil of the subject A is relatively bright, and the dark pupil image is a face image in a state where the pupil of the subject A is relatively dark. The lighting control unit 12 is a functional element that controls the lighting timing of the light sources 3 a and 3 b in synchronization with the photographing timing of the camera 2. Specifically, the lighting control unit 12 performs control so that the light source 3a is turned on when a bright pupil image is captured, and the light source 3b is turned on when a dark pupil image is captured. The detection unit 13 is a functional element that processes face image data output from the camera 2. The output destination of the processing result is not limited at all. For example, the image processing apparatus 1 may display the processing result as an image, graphic, or text on a monitor, store it in a storage device such as a memory or database, or other computer system via a communication network. May be sent to.

  The detection unit 13 includes, as functional components, an image acquisition unit 14, a difference image generation unit 16, a feature point detection unit (rotation center calculation unit) 17, a face posture detection unit (pupil position calculation unit) 18, A line-of-sight detection unit (pupil position calculation unit) 19. The image acquisition unit 14 acquires, as face image data, bright pupil images and dark pupil images that are alternately captured from the camera 2 at a predetermined frame rate. The difference image generation unit 16 generates a difference image between the bright pupil image and the dark pupil image. Specifically, the difference image generation unit 16 generates a difference image by calculating a luminance difference between corresponding pixels of the bright pupil image and the dark pupil image.

  The feature point detection unit 17 calculates the on-image position (two-dimensional position) of the centers of the two pupil images (pupil center) using the difference image, and uses the bright pupil image and the dark pupil image to calculate each pupil. This is a functional element that calculates the two-dimensional position of the corneal reflection image corresponding to the image and the two-dimensional position of the center of the two nostril images (the nostril center). First, the feature point detection unit 17 binarizes the difference image, and performs isolated point removal, noise removal by morphological processing, and labeling. Then, the feature point detection unit 17 detects a pixel group having a shape most likely to be a pupil as a pupil. At this time, even if the pupil is hidden by the eyelids or eyelashes, the boundary between the eyelids or eyelashes and the pupil is excluded as a false pupil contour, and only the true pupil contour is elliptically fitted. The two-dimensional position of is calculated. Also, the feature point detection unit 17 binarizes from the vicinity of the pupil of the bright pupil image with a threshold value higher than the pupil luminance, and uses the two-dimensional position of the center of the cornea reflection image (corneal reflection center) as the center of gravity considering luminance. Ask. The pupil luminance is given not by the area of the ellipse obtained as a result of the ellipse fitting, but by the luminance average of the pixels constituting the pupil obtained by binarization. The feature point detection unit 17 calculates the two-dimensional position of the corneal reflection center even for the dark pupil image. Furthermore, the feature point detection unit 17 calculates the two-dimensional positions of the two nostril centers using the bright pupil image and the dark pupil image in the same manner as the method described in Japanese Patent No. 4431799 by the present inventor.

  In addition to the above functions, the feature point detection unit 17 also has a function of calculating the position (two-dimensional position) on the image of the rotation center of the two eyeballs of the subject A. That is, the feature point detection unit 17 calculates the two-dimensional position of the rotation center of two eyeballs on the assumption of an eyeball model as shown in FIG. As shown in FIG. 4, the eye part of the subject A has a structure in which a substantially spherical corneal sphere CB is fixed toward the front inside the substantially spherical eyeball EB, and the eyeball changes when the line of sight of the subject A changes. The EB and the corneal sphere CB are modeled by an eyeball model in which the EB and the corneal sphere CB integrally rotate around the rotation center E that is the center of the eyeball EB. In this eyeball model, the component BM in the direction passing through the corneal sphere center C in the illumination light irradiated by the light sources 3a and 3b on the front surface of the camera 2 is reflected by the point Cr on the surface of the corneal sphere CB and is reflected as corneal reflection. Will appear on the image. Further, it can be considered that the optical axis L2 of the eyeball EB corresponding to the line of sight of the subject A approximates a straight line passing through the rotation center E of the eyeball, the corneal ball center C, and the pupil center P. FIG. 5 shows an image image captured by the camera 2 targeting the eye part of the subject A modeled by the eyeball model of FIG. 4, and part (a) shows an image image of the entire eye part. , (B) shows the positional relationship between the main points in the image. As shown in part (a) of FIG. 5, on the image of the camera 2, the rotation center E of the eyeball, the corneal sphere center C, and the pupil center P are located substantially in a straight line. A cornea reflection image appears in agreement. At this time, according to the eyeball model described above, as shown in FIG. 5B, the distance PCr between the pupil center P and the corneal reflection center Cr on the image, and the pupil center P and the eyeball on the image. The ratio of the distance PE to the center of rotation E can be approximated by a predetermined ratio (1: t), and the direction of the optical axis L2 (FIG. 4) of the subject A changes due to a change in the line of sight. Is also constant. For example, the paper “Akira Banno,“ Design method of pupil imaging optical system for gaze detection ”, IEICE Transactions D-II Vol.74-DII No.6 pp.736-747, June 1991” According to the eyeball model described in the above, it is assumed that PC = 4.3 mm, PE = 9.8 mm, and t = PE / PC = 2.3. That is, the ratio between the distance PC and the distance PE in the three-dimensional space is the same as the ratio between the distance PCr and the distance PE on the image.

Assuming such an eyeball model, the feature point detector 17 is estimated based on the two-dimensional position of each of the two pupil centers detected on the face image and the cornea reflection center detected correspondingly. Based on the direction on the image of the optical axis L2 of the eyeball of the subject A who has the function of processing as follows. Specifically, the feature point detection unit 17, a position vector V _C indicating the two-dimensional position of the position vector V _P and the corneal reflection center corresponding thereto indicating the two-dimensional position of the pupil center calculated at each imaging timing On the basis of the following formula (1);
V _E = V _P + t (V _C −V _P ) = (1−t) V _P + tV _C (1)
By calculating the position vector V _E indicating the two-dimensional position of the center of rotation of the eyeball. Here, the coefficient t in the above equation (1) is set in advance corresponding to the assumed eyeball model. The feature point detection unit 17 calculates the two-dimensional position of the rotation center of the two eyeballs of the subject in the same procedure.

  The face posture detection unit 18 detects the face posture of the subject A based on the two-dimensional position of the rotation center of the two eyeballs and the two-dimensional position of the two nostril centers calculated by the feature point detection unit 17. That is, by applying the method described in Japanese Patent No. 4431799 by the present inventor, based on the two-dimensional position of the center of rotation of the two eyes and the two-dimensional position of the center of the two nostrils, The three-dimensional position and the three-dimensional position of the two nostril centers are calculated. Further, by applying the method described in the patent, the face posture detection unit 18 selects a combination of three points of the rotation center of the two eyeballs and the center of the two nostrils as the first and second reference region groups. Then, a normal vector of a plane including each reference region group is calculated, and a combined vector of these normal vectors is derived as the face posture of the subject person A. In addition, the face posture detection unit 18 calculates the centroids of the three points included in the first and second reference part groups, and calculates the centroid between the two centroids as the face position of the subject A. Note that the face posture detection unit 18 may calculate the face posture and the face position using the midpoint of the two-dimensional positions of the two nostril centers, or only one of the two-dimensional positions of the two nostril centers. It may be used to calculate the face posture and the face position.

  Here, when detecting the face posture, the face posture detection unit 18 refers to a measured distance between three points constituting the reference region group from a storage unit such as the auxiliary storage unit 103 (FIG. 2) in the image processing apparatus 1. The Thereby, the three-dimensional positions of the three points constituting the reference part group can be obtained with high accuracy. For example, when using the midpoint of the two-dimensional position of two nostril centers, the measured distance between the rotation centers of two eyeballs and the midpoint between the two nostril centers and the rotation centers of the respective eyeballs are used. The measured distance is stored in advance and is referred to when detecting the face posture. These measured distances can be measured in advance with a measuring tool such as a ruler. For example, it is assumed that the measured distance between the rotation centers of two eyeballs is the distance between the pupils of both eyes when the subject A is viewed from the front, and the measured value is equal to the distance between the rotation centers of the eyeballs. Can be obtained. On the other hand, the measured distance between the rotation centers of the two eyeballs may be set to a predetermined value (for example, 65 mm) as an average human value. The accuracy of the measured distance from the midpoint between the nostril centers to the rotation center of each eyeball does not significantly affect the accuracy of the three-dimensional position of the pupil, but the detected subject A's head in the vertical and horizontal directions Affects attitude. If the detection accuracy of the face posture in the left-right direction need not be high, the measured distances between the midpoint between the nostril centers and the rotation centers of the respective eyeballs may be set equal. Further, if the detection accuracy of the face posture is not required to be high, a value obtained by actually measuring the distance between the straight line connecting the two pupils and the midpoint between the nostrils may be set in advance.

  The line-of-sight detection unit 19 is a cubic of the two-dimensional position of the two pupil centers of the subject A detected by the feature point detection unit 17 and the rotation center of the two eyeballs of the subject A detected by the face posture detection unit 18. Based on the original position, the three-dimensional position of the two pupils of the subject A is calculated. Here, the pupil image and the corneal reflection image appear on the image of the camera 2 at the same time as the angle formed by the straight line connecting the optical system 4 and the pupil center P of the subject A and the optical axis L2 of the eyeball (FIG. 4). It is assumed that θ is 30 degrees at the maximum. In the first place, the rotation center E of the eyeball is close to the pupil center P, and the distance between them is as small as about 9.4 mm in the above-described eyeball model. Therefore, even if the line of sight of the subject A changes by 30 degrees, the distance from the camera 2 at the center of the pupil only changes by (1-cos30 °) × 100 = 13%. Accordingly, it can be estimated that the pupil center P is present at a predetermined distance (for example, 9.4 mm) in front of the rotation center E of the eyeball as viewed from the camera 2. Using this, the line-of-sight detection unit 19 sets a plane that is perpendicular to the optical axis L1 of the camera 2 and that is located a predetermined distance (eg, 9.4 mm) from the center of rotation of the eyeball. Then, an intersection point between the plane and the center (pinhole) of the camera 2 obtained from the face image and a straight line passing through the pupil center P is obtained, and the intersection point is calculated as a three-dimensional position of the pupil center P. Similarly, the line-of-sight detection unit 19 calculates the three-dimensional position of the two pupil centers P.

  Furthermore, the line-of-sight detection unit 19 calculates a line-of-sight vector (line-of-sight direction) and a gazing point for each eye part of the subject A. That is, the present inventor uses the three-dimensional position of the pupil center P and the vector r from the pupil center P to the corneal reflection center calculated from the two-dimensional position of the pupil center P and the two-dimensional position of the corneal reflection center. In the same manner as in Japanese Patent No. 4517049, the angle θ formed by the straight line passing through the camera 2 and the center of the pupil and the optical axis L2 is obtained to calculate the line-of-sight vector and the gaze point on a predetermined plane. Then, the gaze detection unit 19 calculates the final gaze direction and gaze point using the two gaze vectors calculated for each of the two eyes.

  Note that the line-of-sight detection unit 19 may calculate the line-of-sight direction using the corneal sphere center C instead of the pupil center P. That is, in the eyeball model described above, the optical axis L2 passes through the rotation center E and the corneal ball center C of the eyeball, and the distance between them is about 5.6 mm in the eyeball model described above. Even if the line of sight of the subject A changes by 30 degrees, the distance seen from the camera 2 at the center of the corneal sphere only changes by 13%, so the corneal sphere center C seen from the camera 2 is more than the rotation center E of the eyeball. Can be estimated to be present at a predetermined distance (for example, 5.6 mm). Using this, the line-of-sight detection unit 19 sets a plane that is perpendicular to the optical axis L1 of the camera 2 and that is located a predetermined distance (eg, 5.6 mm) from the center of rotation of the eyeball. Then, an intersection point between the plane and the straight line passing through the center (pinhole) of the camera 2 obtained from the face image and the corneal reflection center Cr is obtained, and the intersection point is calculated as a three-dimensional position of the corneal sphere center C. Further, the line-of-sight detection unit 19 calculates a vector r from the pupil center P to the corneal reflection center Cr calculated from the three-dimensional position of the corneal sphere center C and the two-dimensional position of the pupil center P and the two-dimensional position of the corneal reflection center. , The line-of-sight vector and the gazing point on a predetermined plane can be calculated by obtaining the angle θ between the straight line passing through the camera 2 and the center of the corneal sphere and the optical axis L2.

  Next, the operation procedure of the above-described pupil detection device 10 will be described, and the pupil detection method according to the present embodiment will be described.

  First, when pupil detection processing is activated by an instruction input by the user using the input device 105 to the pupil detection device 10, the image processing device 1 illuminates the face of the subject A with the light sources 3a and 3b. Control is performed so that images are continuously captured by the camera 2 (imaging step).

  Thereafter, the two-dimensional position of the rotation center E of the two eyeballs is calculated by the detection unit 13 of the image processing apparatus 1 based on the bright pupil image and the dark pupil image acquired by the camera 2, and the two pupils are calculated based on the two-dimensional positions. A three-dimensional position of the center P is calculated (pupil detection step). In this pupil detection step, a two-dimensional position of the pupil center P and a direction on the image of the optical axis L2 of the eyeball of the subject A are detected based on the difference image generated from the bright pupil image and the dark pupil image, Based on these, the step of calculating the two-dimensional position of the rotation center E of the eyeball of the subject A, and the pupil center P based on the two-dimensional position of the center of rotation E of the eyeball and the two-dimensional position of the nostril center of the subject A Calculating a three-dimensional position.

  Finally, the detection unit 13 of the image processing apparatus 1 calculates the face posture of the subject A, calculates the gaze direction and gazing point of the subject A, and outputs the results to the output device 106 ( Face posture / gaze detection step).

  According to the pupil detection device 10 described above, two pupil positions based on the two-dimensional positions of the two pupil centers P on the face image acquired by the optical system 4 and the direction of the optical axis L2 of the eyeball on the image are used. The two-dimensional position of each rotation center E of the eyeball is calculated, and based on the calculated two-dimensional position of the rotation center E of the two eyeballs and the two-dimensional position of the nostril center, the three-dimensional position of the two pupil centers P Is calculated. Here, even when the viewing direction of the subject A changes, the positional relationship between the three points of the rotation center E of the two eyeballs and the nostril center is maintained. In other words, the eyeball is modeled as a sphere that rotates in the orbit, and the position of the center of rotation of the eyeball hardly changes even if the eyeball rotates by changing the line of sight. Therefore, even if there is a change in the line-of-sight direction, the variation of the eyeball with respect to the skeleton coordinate system is extremely small. On the other hand, since the nostrils are also fixed with respect to the skeletal coordinate system, as a result, the distance between the three points between the rotation center of the two eyeballs and the nostril center varies even if the line-of-sight direction varies with respect to the skeleton coordinate system It will not change. By utilizing this property, it is possible to detect the position of the pupil with high accuracy, further detect the face posture with high accuracy, and detect the gaze direction and the gazing point with high accuracy. In addition, according to the configuration of the pupil detection device 10, the number of cameras in the optical system can be reduced, and the configuration of the optical system is simplified.

  In the pupil detection device 10 having the above-described configuration, the eyeball is based on the two-dimensional position of each of the two pupil centers P and the two-dimensional position of the corneal reflection center Cr generated corresponding to each of the two pupil centers P. The direction of the optical axis L2 is obtained, and the two-dimensional position of the rotation center E is calculated based on the direction of the optical axis L2 of the eyeball. In this case, since the two-dimensional position of the rotation center E of the eyeball is calculated from the two-dimensional position of the pupil center P and the two-dimensional position of the corneal reflection center Cr, it is possible to detect the position of the pupil with high accuracy by simple calculation. .

Further, based on the two-dimensional position of the rotation center E of the two eyeballs and the two-dimensional position of the nostril center, the three-dimensional position of the rotation center E of the two eyeballs and the three-dimensional position of the nostril center are calculated. Thus, the face posture of the subject person A is detected, so that the face posture of the subject person A can be detected with high accuracy.
[Second Embodiment]

  Next, the configuration of the pupil detection device according to the second embodiment will be described. FIG. 6 is a block diagram illustrating a functional configuration of the image processing apparatus 1A that constitutes the pupil detection apparatus of the second embodiment. In the image processing apparatus 1A shown in the figure and the image processing apparatus 1 according to the first embodiment, the functions of the feature point detection unit 17A and the line-of-sight detection unit 19A included in the detection unit 13 are different.

  The line-of-sight detection unit 19A replaces the line-of-sight vector calculation method using the vector r from the pupil center P to the corneal reflection center in the first embodiment (hereinafter referred to as “pupil-corneal reflection method”). It has a function of executing a line-of-sight vector calculation method (hereinafter referred to as “pupil shape method”) using the pupil shape detected on the image.

The principle of the pupil shape method executed by the line-of-sight detection unit 19A will be described. FIG. 7A shows the eyeball EB of the subject A when the subject A is looking at the optical system 4 including the camera 2. Part (b) of FIG. 7 shows the eyeball EB of the subject A when the subject A is looking at a place different from the optical system 4. In FIG. 7A and FIG. 7B, it is assumed that the eyeball EB of the subject A is at infinity from the optical system 4 including the camera 2. As shown in part (a) of FIG. 7, when the subject A is looking at the optical system 4, the shape of the pupil EP in the face image obtained from the camera 2 is a circle. The diameter TD of this circle is the length A1. On the other hand, as shown in part (b) of FIG. 7 and FIG. 8, when the optical axis L2 of the eyeball EB is inclined by the inclination θ _{S with} respect to the axis A3 connecting the camera 2 and the pupil center, The shape of the pupil EP is an ellipse. Major axis _{L A} and minor axis _{L B} is formula of the ellipse (2), as indicated by (3).
Long diameter L _A = A1 (2)
Minor axis L _B = A1 · cos (θ _S ) (3)

Here, the ellipticity R is defined. When the shape of the pupil EP of the image is elliptical, ellipticity R is represented by a major axis L _A and minor axis L _B (formula (4) refer).
Ellipticity = major axis L _A / minor axis L _B (4)

And according to Formula (2)-(4), inclination (theta) _S is shown by Formula (5). This inclination θ _S corresponds to the magnitude | θ | of the angle θ formed by the straight line passing through the camera 2 and the pupil center and the optical axis L2.
θ _S = cos ⁻¹ (minor axis L _B / major axis L _A ) = cos ⁻¹ (1 / ellipticity) (5)

  FIG. 8 shows the eyeball EB viewed from the optical system 4. For example, the pupil Pa shows a state when the subject A is looking at the optical system 4. The pupil Pb shows a state when the subject A is looking above the optical system 4. The other pupils Pc respectively show a state when the subject A is looking at a position different from the optical system 4. An arrow D indicates the direction of the line of sight. As shown in FIG. 8, the minor axis direction of the ellipse indicating the shape of the pupil EP changes depending on the direction of the line of sight (arrow D). The minor axis direction is indicated by the inclination φ of the direction of the line of sight (arrow D) with respect to the horizontal axis X ′ of the world coordinate system.

Using the above principle, the line-of-sight detection unit 19A is based on the shape of the two pupils EP on the face image specified by the ellipse fitting by the feature point detection unit 17, and the ellipticity of each shape and the shortness of the ellipse. Calculate the direction of the axis. The line-of-sight detection unit 19A obtains the inverse cosine component (inclination θ _S ) of the reciprocal of ellipticity and the direction of the minor axis (inclination φ), thereby identifying the line of sight specified by the inclination angle θ _S and the inclination φ. The vector PT is calculated for two eyes. At this time, the line-of-sight detection unit 19A sets the start point of the line-of-sight vector PT to the three-dimensional position of the rotation center E of the eyeball detected by the face posture detection unit 18.

Note that the calculation result obtained by the pupil shape method is the direction of the optical axis L2. Therefore, in the above calculation method, it is assumed that the optical axis L2 corresponds to the line of sight. There is a deviation between the visual axis (axis P through the center of the pupil and the axis passing through the fovea) and the optical axis L2 (the axis of symmetry of the eyeball optical system). Therefore, if the result obtained by the pupil shape method is calibrated, the output result of the pupil detection device can be brought close to the actual line of sight. The line-of-sight detection unit 19A introduces a deviation angle θ ₀ indicating the deviation of the optical axis L2 with respect to the visual axis. That is, the line-of-sight detection unit 19A can also calculate the tilt θ _Sa after calibration by calibrating the tilt θ _S by the following equation (6).
θ _Sa = θ _S −θ ₀ (6)
In the above equation (6), the inclination θ _Sa corresponds to a line-of-sight vector indicating the visual axis. The shift angle θ ₀ is a vector indicating the shift of the optical axis L2 with respect to the visual axis. The deviation angle θ ₀ is defined by the magnitude | θ ₀ | of the deviation angle θ ₀ and the inclination φ ₀ and is represented as θ ₀ = (| θ ₀ |, φ ₀ ).

The feature point detection unit 17A calculates the two-dimensional position of the rotation center E of the two eyeballs based on the direction of the line-of-sight vector calculated by the line-of-sight detection unit 19A. Specifically, the feature point detection unit 17A is a constant velocity motion model such as a Kalman filter based on the three-dimensional coordinates of the rotation center E of the eyeball calculated by the face posture detection unit 18 for the face image of the immediately preceding image frame. The three-dimensional coordinates of the center of rotation E of the eyeball in the image frame to be processed are estimated using an estimation method using. Next, the feature point detection unit 17A calculates the three-dimensional position of the pupil center P in the same manner as the line-of-sight detection unit 19 of the first embodiment. At this time, assuming that a corneal reflection image does not appear on the face image, the feature point detection unit 17A has an angle θ formed by a straight line connecting the optical system 4 and the pupil center P and the optical axis L2 of the eyeball is 30 degrees. Assuming that the pupil center P is located at a predetermined distance (for example, 9.4 mm × cos 30 ° = 8.1 mm) before the rotation center E of the eyeball as viewed from the camera 2, the three-dimensional position of the pupil center P is calculated. . Further, the feature point detection unit 17A includes a position vector W _P indicating the three-dimensional position of the calculated pupil center P, a unit that indicates the direction of the optical axis L2 of the eyeball is calculated by using the pupil shape method by line-of-sight detection section 19A based on the vector W _0, it calculates the position vector W _E showing the three-dimensional position of the center of rotation E of the eye by the following formula (7).
W _E = W _P −κ _E · W ₀ (7)
In the above equation (7), κ _E is a numerical value indicating the distance between the preset pupil center P and the rotation center E. Then, the feature point detection unit 17A converts the obtained three-dimensional position of the rotation center E of the eyeball into a two-dimensional position on the face image using the pinhole model of the camera 2. In the same procedure, the feature point detection unit 17 calculates the two-dimensional position of the rotation center E of the two eyeballs. The calculated two-dimensional position of the rotation center E of the two eyeballs is used for detection of the face posture by the face posture detection unit 18 in the image frame to be processed.

According to the pupil detection device of the second embodiment described above, the pupil position and line of sight can be detected relatively accurately with respect to the irregular movement of the head of the subject A. In the pupil detection device according to the first embodiment, the pupil position is detected by using both the pupil image and the cornea reflection image that appear in the face image, but the camera 2 is used to detect the nostril image. When installed below the front (30 degrees below the front), when the subject A looks above the front, the angle θ exceeds 30 degrees, so that the corneal reflection does not appear depending on the subject A. On the other hand, the ellipticity of the pupil observed from the camera 2 becomes high, and the change rate of the ellipticity with respect to the change of the gaze angle becomes high, and as a result, the detection resolution of the angle θ when using the pupil shape method becomes high. . At the same time, as the ellipticity increases, the direction of the minor axis can be obtained with high accuracy. As a result, when the subject A looks in a direction away from the direction of the camera 2, the accuracy of pupil position and line-of-sight detection is increased by using the pupil shape method of the second embodiment.
[Third Embodiment]

  Next, the configuration of the pupil detection device according to the third embodiment will be described. FIG. 9 is a block diagram illustrating a functional configuration of the image processing apparatus 1B that configures the pupil detection apparatus of the third embodiment. The image processing apparatus 1B shown in the figure and the image processing apparatus 1 according to the first embodiment are different in the functions of the feature point detection unit 17B and the line-of-sight detection unit 19B included in the detection unit 13.

  That is, the line-of-sight detection unit 19B includes a processing unit 22a having a function of calculating a line-of-sight vector using the pupil-cornea reflection method in the same manner as the line-of-sight detection unit 19 of the first embodiment, and the line-of-sight detection unit of the second embodiment. Similarly to 19A, it has a processing unit 22b having a function of calculating a line-of-sight vector using the pupil shape method. Finally, the line-of-sight vector is calculated by taking into account the processing result of the processing unit 22a and the processing result of the processing unit 22b. calculate. For example, the average value of the processing results of both processing units is calculated, or the line-of-sight vector is calculated by weighted addition of the processing results of both processing units. Further, the line-of-sight detection unit 19B may calculate the line-of-sight vector by selecting and using one of the processing results of the processing unit 22a and the processing unit 22b. For example, when the calculated ellipticity of the pupil shape is a predetermined value or more, the processing result of the processing unit 22b is selected, and when the ellipticity of the pupil shape is less than the predetermined value, the processing result of the processing unit 22a is selected. The

  The feature point detection unit 17B includes a processing unit 21a that calculates the two-dimensional position of the rotation center E of the two eyeballs in the same manner as the feature point detection unit 17 in the first embodiment, and two feature points in the same manner as in the second embodiment. The processing unit 21b that calculates the three-dimensional position of the pupil center P and the two-dimensional position of the rotation center E of the two eyeballs may be used by taking into account the processing results of the two processing units 21a and 21b. For example, the face posture detection unit 18 takes into account the three-dimensional positions of the two pupil centers P calculated using the processing result of the processing unit 21a and the three-dimensional positions of the two pupil centers P calculated by the processing unit 21b. Then, the final three-dimensional positions of the two pupil centers P are calculated. For example, the average value of both processing results is calculated, or the three-dimensional position of the pupil center P is calculated by weighting and adding both processing results. Further, the face posture detection unit 18 may obtain the three-dimensional position or the face posture of the pupil center P by selecting and using one of the processing results of the processing unit 21a and the processing unit 21b. For example, when the ellipticity of the pupil shape calculated by the line-of-sight detection unit 19B is equal to or greater than a predetermined value, the processing result of the processing unit 21b is selected, and when the ellipticity of the pupil shape is less than the predetermined value, the processing unit 21a A processing result is selected.

  According to the pupil detection device according to the third embodiment described above, by adding the calculation result using the two-dimensional position of the corneal reflection image and the calculation result using the ellipticity related to the pupil image detected on the image, More accurate pupil position detection and line-of-sight detection are realized. Moreover, by selecting and using the calculation result using the two-dimensional position of the cornea reflection image and the calculation result using the ellipticity of the pupil image detected on the image, more accurate pupil position detection and gaze detection can be performed. Realized.

  The present invention is not limited to the embodiment described above. In the said embodiment, you may change into the following structure.

  For example, the detection of the face posture by the face posture detection unit 18 in each of the above-described embodiments is performed using the method described in Japanese Patent No. 4501003 by the present inventor and the direction of the normal line of the plane composed of both eyes and the right nostril And detecting the direction of the normal line of the plane composed of both eyes and the left nostril. Furthermore, in the detection of the face posture by the face posture detection unit 18, in order to improve the tracking of both eyes and nostrils, a differential position correction method and a feature point tracking method using the face posture (patent no. The method described in No. 5429885 may be used.

  A method for calculating a three-dimensional pupil position and a line-of-sight vector by the feature point detection unit 17, the face posture detection unit 18, and the line-of-sight detection unit 19 in the first embodiment for both eyes of the subject A, and a second embodiment The method of calculating the three-dimensional position of the pupil and the line-of-sight vector by the feature point detection unit 17A, the face posture detection unit 18, and the line-of-sight detection unit 19A in the form may be used separately at the same time.

  Further, the calculation of the three-dimensional position of the pupil center P in the feature point detection unit 17A according to the second embodiment may be performed based on the direction of the optical axis L2 calculated by the line-of-sight detection unit 19B. Specifically, the feature point detection unit 17A sets a plane at a predetermined distance in front of the rotation center E along the direction of the optical axis L2, and the plane and the pinhole of the camera 2 obtained from the face image An intersection point with a straight line passing through the pupil center P may be obtained, and the intersection point may be calculated as a three-dimensional position of the pupil center P.

  In the image processing apparatus 1B of the third embodiment, when the position on the corneal reflection image is detected only for one eye of the subject A and the position of the corneal reflection of the other eye is not detected, the processing unit 22a The line-of-sight vector may be determined giving priority to the processing result. For example, the line-of-sight vector may be calculated using only the line-of-sight vector detected by the processing unit 22a for one eye, or the line-of-sight vector may be calculated by increasing the weight on the processing result by the processing unit 22a.

  In addition, when the line-of-sight detection unit 19B of the third embodiment calculates the line-of-sight vector using the pupil shape method, the line-of-sight vector is directed rightward or leftward with respect to the camera 2 as follows. You may determine whether you are facing. That is, when the pupil shape method is simply used, it cannot be determined which direction the subject A is facing the camera, so the direction of the line-of-sight vector is determined by the following method.

  10 and 11 are plan views showing the state of the subject A with respect to the optical system 4. Part (a) of FIG. 10 shows a case where the subject A is looking slightly rightward toward the optical system 4 with respect to the optical system 4. In this case, since the rotation of the head of the subject A is small, the line-of-sight detection unit 19B can detect the line-of-sight vector using the pupil-cornea reflection method. 10B shows a case where the subject A moves his / her line of sight to the right side of the camera 2. In this case, since the angle θ is large, a corneal reflection image does not appear. Therefore, the line-of-sight detection unit 19B detects the line-of-sight vector using the pupil shape method, but it cannot be determined from the pupil shape whether the subject A is looking at the right side or the left side of the camera 2. However, unless the subject A intentionally shifts his / her line of sight, not only the line-of-sight vector DPT but also the face direction DP rotates to the right as shown in FIG. Part (a) of FIG. 11 shows a case where the subject A looks relatively left with respect to the optical system 4, and in this case, the line-of-sight detection unit 19B detects the line-of-sight vector using the pupil shape method. To do. In this case, not only the line-of-sight vector DPT but also the face direction DP rotates to the left. Using this tendency, when the line-of-sight detection unit 19B obtains the line-of-sight vector, it refers to the face direction detected by the face posture detection unit 18 so as to face the right direction or the left direction. Determine. Specifically, the line-of-sight detection unit 19B determines that when the absolute value of the angle θ exceeds 30 degrees, the line-of-sight vector is leftward when the face direction DP is inclined to the left by a predetermined angle Th or more. The line-of-sight vector is calculated as being inclined, and when the face direction DP is inclined rightward by a predetermined angle Th or more, the line-of-sight vector is calculated as if the line-of-sight vector is inclined rightward. Here, the angle Th as the threshold is set to about 5 degrees. At this time, the front direction of the subject A may be calibrated using the method described in Japanese Patent No. 5004099 by the present inventor. In other words, since the normal of the triangle consisting of the center of rotation of the two eyeballs and the midpoint between the nostrils and the front of the face are generally greatly displaced up and down, the deviation angle between them is obtained as a calibration value, Calibrate the face direction using the calibration value.

  Note that, as in the state shown in part (b) of FIG. 11, the optical axis of the eye part rather faces the right side rather than the head direction, even though the head of the subject A is relatively large looking left. It is possible that the subject A rotates his / her head while looking at one point as in the case where the user is present. However, in this case, there is no problem because the angle θ is small and the line-of-sight vector can be detected using the pupil-corneal reflection method. In the state shown in part (c) of FIG. 11, it is assumed that the subject A is looking further to the right than in the state shown in part (b) of FIG. 11. It is not only unnatural to turn the line of sight largely to the right, and even in this state, since the angle θ is 30 degrees or less, the line-of-sight vector can be detected using the pupil-cornea reflection method, and there is no problem.

  The above-described method for determining the direction of the line-of-sight vector by the line-of-sight detection unit 19B is applied not only to the horizontal direction about the optical system 4 but also to other point-symmetrical directions (for example, the vertical direction) about the optical system 4. May be.

  Further, in the pupil detection devices of the first to third embodiments described above, a plurality of optical systems 4 may be provided in advance in at least two systems. The combination of the two optical systems 4 of the plurality of optical systems 4 is preliminarily camera calibrated to enable three-dimensional measurement of the object by stereo matching. For example, you may provide the two optical systems 4 arrange | positioned in the direction of 15 degree | times on either side seeing from the center position of the subject person's A presence range. If two systems of optical systems capable of imaging in the range of 45 degrees in both the left and right directions are provided, for example, the subject A can image in the range of 120 degrees in the horizontal direction as a whole. In such image processing apparatuses 1, 1 </ b> A, and 1 </ b> B of the pupil detection device, the image is output from the two optical systems 4 in a state where the subject A is previously positioned so as to face the two optical systems 4 at the center position. Based on the image, the three-dimensional position of the two pupil centers and nostril centers (or the midpoint between the two nostril centers) of the subject A can be measured by stereo matching. Then, the image processing apparatuses 1, 1 </ b> A, 1 </ b> B, based on the measured three-dimensional positions of the two pupil centers and the nostril center of the subject A, the distance between the two pupil centers (first distance), and two The distance (second distance) between the pupil center and the nostril center is calculated in advance. Thereafter, the image processing apparatuses 1, 1 </ b> A, 1 </ b> B perform the three-dimensional position, face posture, and face position of the pupil of the subject A by the method described in the first to third embodiments based on the distances. , Line-of-sight vector, and gaze point can be detected. At this time, the second distance to be calculated may be a distance between each of the two pupil centers and the nostril center, or a distance between a straight line connecting the two pupil centers and the nostril center. May be. According to such a pupil detection device, it is possible to detect the face direction and the line-of-sight direction for the subject A located in a wide angle range. Moreover, according to the pupil detection device in which a plurality of combinations of the two optical systems 4 are arranged, the face direction and the line-of-sight direction can be detected in a wider angle range.

  DESCRIPTION OF SYMBOLS 1,1A, 1B ... Image processing device, 2 ... Camera (imaging device), 3a, 3b ... Light source (illumination device), 4 ... Optical system (optical system), 10 ... Pupil detection device, 13 ... Detection unit (processing part) ), 17, 17A, 17B ... feature point detector (rotation center calculator, pupil position calculator), 18 ... face posture detector (pupil position calculator), 19, 19A, 19B ... gaze detector (pupil position calculator) Part), 21a, 22a ... processing part (first processing part), 21b, 22b ... processing part (second processing part), A ... subject, C ... corneal sphere center, CB ... corneal sphere, Cr ... cornea Reflection center, E ... center of eyeball rotation, EB ... eyeball, EP ... pupil, P ... pupil center.

Claims (10)

An optical system including an imaging device that images the face of the subject and an illumination device that illuminates the face;
A processing unit that calculates a three-dimensional position of two pupils of the subject based on an image output by the imaging device;
With
The processor is
Based on the position of each of the two pupils detected on the image and the direction of the optical axis of the eyeball of the subject person, the position on the image of the rotation center of each of the eyeballs of the subject person is determined. A rotation center calculation unit to calculate,
A pupil position calculation unit that calculates a three-dimensional position of the two pupils based on the position on the image of the rotation center of the two eyeballs and the position of the nostril of the subject;
Having
Pupil detection device. The rotation center calculation unit obtains the direction of the optical axis of the eyeball based on each position of the two pupils and a position on the image of corneal reflection that occurs corresponding to each of the two pupils, and the eyeball Calculating the position of the center of rotation on the image based on the direction of the optical axis of
The pupil detection device according to claim 1. The rotation center calculation unit obtains the direction of the optical axis of the eyeball based on the ellipticity related to each of the two pupils detected on the image, and calculates the rotation center based on the direction of the optical axis of the eyeball. Calculating a position on the image;
The pupil detection device according to claim 1. The processing unit, as the rotation center calculation unit, is a direction of the optical axis of the eyeball based on each position of the two pupils and a position on the image of corneal reflection that occurs corresponding to each of the two pupils. A first processing unit that calculates the position of the rotation center on the image based on the direction of the optical axis of the eyeball, and an ellipticity relating to each of the two pupils detected on the image. A second processing unit that obtains the direction of the optical axis of the eyeball and calculates the position of the center of rotation on the image based on the direction of the optical axis of the eyeball,
Taking into account the processing result by the first processing unit and the processing result by the second processing unit to calculate the three-dimensional position of the two pupils;
The pupil detection device according to claim 1. The processing unit, as the rotation center calculation unit, is a direction of the optical axis of the eyeball based on each position of the two pupils and a position on the image of corneal reflection that occurs corresponding to each of the two pupils. A first processing unit that calculates the position of the rotation center on the image based on the direction of the optical axis of the eyeball, and an ellipticity relating to each of the two pupils detected on the image. A second processing unit that obtains the direction of the optical axis of the eyeball and calculates the position of the center of rotation on the image based on the direction of the optical axis of the eyeball,
Calculating a three-dimensional position of the two pupils by selecting and using any one of the processing results of the first processing unit and the second processing unit;
The pupil detection device according to claim 1. When the corneal reflection corresponding to one pupil of the subject is detected and the corneal reflection corresponding to the other pupil of the subject is not detected, the processing of the first processing unit and the second processing unit Among the results, the processing result of the first processing unit related to the one pupil is preferentially used.
The pupil detection device according to claim 4 or 5. The pupil position calculation unit, based on the position on the image of the rotation center of the two eyeballs and the position on the image of the nostril of the subject, and the three-dimensional position of the rotation center of the two eyeballs and the Detecting the face direction of the subject by calculating the three-dimensional position of the nostrils;
The pupil detection apparatus of any one of Claims 1-6. A line-of-sight detection unit that obtains the line-of-sight direction of the subject based on the direction of the optical axis of the eyeball;
The pupil position calculation unit, based on the position on the image of the rotation center of the two eyeballs and the position on the image of the nostril of the subject, and the three-dimensional position of the rotation center of the two eyeballs and the By detecting the three-dimensional position of the nostril and detecting the face direction of the subject,
The line-of-sight detection unit determines whether the line-of-sight direction of the subject person is one of two directions based on the face direction of the subject person.
The pupil detection device according to claim 3. Comprising at least two systems of the optical system;
The processing unit measures a first distance between the two pupils and a second distance between the pupils and the nostrils based on images output from at least two systems of the optical system. And using the first distance and the second distance to calculate a three-dimensional position of the two pupils,
The pupil detection device according to any one of claims 1 to 8. Imaging the face with an imaging device while illuminating the face of the subject with an illumination device;
Calculating a three-dimensional position of two pupils of the subject based on an image output by the imaging device;
With
The step of calculating the three-dimensional position includes:
Based on the position of each of the two pupils detected on the image and the direction of the optical axis of the eyeball of the subject person, the position on the image of the rotation center of each of the eyeballs of the subject person is determined. A calculating step;
Calculating a three-dimensional position of the two pupils based on a position on the image of the rotation center of the two eyeballs and a position on the image of the nostril of the subject;
Having
Pupil detection method.

Images