JP2017213191A - Sight line detection device, sight line detection method and sight line detection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a burden applied to a detection object person in calibration in sight line detection.SOLUTION: A sight line detection device comprises a storage part, a feature amount calculation part, a feature amount determination part, and a parameter calculation part. In the storage part, plural eyeball features containing a set of feature amounts of the eyeballs, detected when one gazes each of plural gaze points which are predetermined, from a view point in a positional relationship with a camera, in a mode of being associated with a radius of a cornea. The feature amount calculation part calculates a feature amount of eyeball of a person, based on an image of the person who gazes the first gaze point out of the plural gaze points, acquired from the camera. The feature determination part determines the eyeball feature of the person based on the feature amount of the eyeball corresponding to the first gaze point, out of the plural eyeball features registered in the storage part, and the feature amount of the eyeball of the person. The parameter calculation part calculates a calibration parameter to a sight line of the person, based on the plural eyeball feature amounts included in the determined eyeball feature.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a line-of-sight detection device, a line-of-sight detection method, and a line-of-sight detection program.

  The line-of-sight detection technology for detecting a person's line of sight is applied to an input device that inputs information by line of sight, recording of the line of sight of a person driving a vehicle, and the like. As one of gaze detection methods, a pupil-cornea reflection method is known that detects a gaze of a person based on the position of the pupil and the position of corneal reflection in an image of the human eyeball.

  In the line-of-sight detection technique, in order to accurately detect the line of sight of a person, a process (calibration) for calibrating the direction of the line of sight based on the characteristics of the eyeball is performed.

  In the calibration in the pupil-corneal reflection method, the line of sight detected based on the position of the pupil and the position of the corneal reflection in a state in which a person is gazing at a predetermined gazing point, the direction of the gazing point viewed from the person, Based on the above, a calibration parameter used for line-of-sight calibration is calculated.

  When performing calibration and calculating calibration parameters, a person as a detection target is caused to gaze at a plurality of gazing points.

  In addition, as the above-described line-of-sight detection technique, a plurality of parameter sets are stored corresponding to a plurality of observers, and an optimal set of parameters is selected based on the characteristics of the eyeball of the current observer, and the line of sight is detected. A technique of calculating is known (see, for example, Patent Document 1). This type of line-of-sight detection technology makes it possible to easily, quickly and accurately respond to changes in the state of the eyeball of the observer that is the detection target due to changes in the observer.

  Further, as the above-described line-of-sight detection technique, a technique for improving the line-of-sight detection accuracy is known by dynamically repeating calibration even after initial calibration is performed and line-of-sight detection is started (for example, Patent Documents). 2).

Japanese Patent Application Laid-Open No. 07-035966 JP 2000-010723 A

  When calculating calibration parameters for the line of sight by performing calibration, a person who is a detection target is gazes at a plurality of gaze points (for example, 4 to 9 gaze points). That is, in one calibration, a process of specifying a gazing point to cause a person to gaze and acquiring an image in a state where the person is gazing at the gazing point is performed a plurality of times. It is a burden for the person to be detected to gaze at a plurality of gaze points in this way.

  In one aspect, an object of the present invention is to reduce a burden on a detection target person during calibration in line-of-sight detection.

  In one aspect, the visual line detection device includes a storage unit, a feature amount calculation unit, a feature determination unit, and a parameter calculation unit. In the storage unit, a plurality of eyeball features including a set of eyeball feature amounts detected when each of a plurality of gaze points determined in advance from a viewpoint having a predetermined positional relationship with the camera is associated with the corneal radius. It is registered. The feature amount calculation unit calculates the feature amount of the eyeball of the person based on the image of the person gazing at the first gazing point among the plurality of gazing points acquired from the camera. The feature determination unit determines the eyeball feature of the person based on the feature amount of the eyeball corresponding to the first gazing point among the plurality of eyeball features registered in the storage unit and the calculated feature amount of the eyeball of the person. decide. The parameter calculation unit calculates a calibration parameter for the line of sight of the person based on the feature amounts of the plurality of eyeballs included in the determined eyeball feature.

  According to the above-described aspect, the burden on the person to be detected at the time of calibration in line-of-sight detection is reduced.

It is a figure which shows the functional structure of the gaze detection apparatus which concerns on 1st Embodiment. It is a figure which shows the example of an eyeball characteristic database. It is a figure explaining the feature-value of an eyeball and the distance between pupils. It is a figure which shows the functional structure of a database preparation apparatus. It is a flowchart explaining the creation process of the eyeball feature database which concerns on 1st Embodiment. It is a figure which shows the example of application of a gaze detection apparatus. It is a figure explaining the acquisition method of the image at the time of eyeball feature database preparation. It is a flowchart explaining the process which the gaze detection apparatus which concerns on 1st Embodiment performs. It is a flowchart explaining the content of the calibration parameter determination process which concerns on 1st Embodiment. It is a flowchart explaining the content of a gaze detection process. It is a figure explaining the correlation with the position of a pupil, the position of corneal reflection, and a corneal radius. It is a figure explaining the correlation with the position of a pupil on an image, the position of corneal reflection, and a corneal radius. It is a figure which shows another example of application of a gaze detection apparatus. It is a figure which shows the modification of a gaze detection system. It is a flowchart explaining the content of the calibration parameter determination process which concerns on 2nd Embodiment. It is a flowchart explaining the content of the process which correct | amends the feature-value of an eyeball. It is a figure explaining the relationship between the direction of a face, and the feature-value of an eyeball. It is a figure explaining the correction method of the feature-value of an eyeball when a face is not facing a camera. It is a figure which shows the functional structure of the gaze detection apparatus which concerns on 3rd Embodiment. It is a figure which shows the example of the eyeball characteristic database which concerns on 3rd Embodiment. It is a figure explaining the feature-value and iris radius of an eyeball. It is a flowchart explaining the content of the calibration parameter determination process which concerns on 3rd Embodiment. It is a figure explaining the correlation with an iris radius and eyeball shape. It is a figure explaining the example of the determination method of the eyeball feature which concerns on 3rd Embodiment. It is a flowchart explaining the content of the process which determines the eyeball characteristic in the calibration parameter determination process which concerns on 4th Embodiment. It is a figure which shows the example of the some record extracted from an eyeball characteristic database. It is a figure explaining the calculation method of the feature-value. It is a figure which shows the hardware constitutions of a computer.

[First Embodiment]
FIG. 1 is a diagram illustrating a functional configuration of the visual line detection device according to the first embodiment.

  As shown in FIG. 1, the line-of-sight detection device 1 according to the present embodiment includes an image acquisition unit 101, a gaze instruction unit 102, a feature amount calculation unit 103, a process switching unit 104, a feature amount correction unit 105, A feature determination unit 106, a parameter calculation unit 107, and a line-of-sight calculation unit 108 are provided. Furthermore, the line-of-sight detection device 1 includes an eyeball feature database 121 and a line-of-sight storage unit 122. The line-of-sight detection apparatus 1 according to the present embodiment detects the line of sight of a person who is a line-of-sight detection target (hereinafter referred to as a target person) by the pupil-corneal reflection method.

  The image acquisition unit 101 acquires an infrared image captured by the infrared camera 201 (hereinafter simply referred to as an image). The infrared camera 201 is an imaging device used for imaging an image including an eyeball of a target person who observes a predetermined spatial region or a display surface of a predetermined display device. The infrared camera 201 is used in combination with a light source that emits infrared rays, such as an infrared light emitting diode (LED) 202. In the set of the infrared camera 201 and the infrared LED 202 (hereinafter referred to as the line-of-sight sensor 2), the infrared LED 202 emits infrared rays toward a space including the imaging range of the infrared camera 201 and is installed in the vicinity of the infrared camera 201. Is done. The infrared camera 201 captures an image including the eyeball of the target person irradiated with infrared rays emitted from the infrared LED 202 and transmits the image to the image acquisition unit 101.

  Further, when performing the process (calibration) of calibrating the line of sight of the target person in the line-of-sight detection device 1, the image acquisition unit 101 causes the gaze instruction unit 102 to execute a process of gazing at the point of interest of the target person. Note that the process of calibrating the line of sight in this specification means a process of determining calibration parameters used in the process of detecting the line of sight of the target person.

  Based on the information input from the input device 3, the image acquisition unit 101 determines whether or not to perform processing for causing the target person to watch the gazing point. The input device 3 is a button switch, a touch panel device, or the like, and is used, for example, for inputting a command for causing the line-of-sight detection device 1 to start a line-of-sight detection process, a command for performing line-of-sight calibration, and the like. When the information input from the input device 3 is a command for executing processing including line-of-sight calibration, the image acquisition unit 101 causes the gaze instruction unit 102 to execute processing for causing the target person to gaze at the gaze point.

  The gaze instruction unit 102 uses the output device 4 to perform processing for notifying the target person of the position (gaze point) at which the target person is gazed. For example, when the output device 4 is a display device, the gaze instruction unit 102 displays an image indicating a gazing point on the display surface of the display device. When the output device 4 is a speaker, the gaze instruction unit 102 outputs a prepared audio signal to the speaker.

  Based on the image acquired from the infrared camera 201, the feature amount calculation unit 103 calculates the feature amount of the eyeball of the target person shown in the image. The feature amount calculation unit 103 extracts the center of corneal reflection and the center of the pupil in each of the right eye and the left eye from the image, and calculates the positional relationship between them as the feature amount of the eyeball. The positional relationship between the position of the corneal reflection and the position of the pupil is, for example, a two-dimensional coordinate representing the center position of the pupil viewed from the center position of the corneal reflection in the image.

  The process switching unit 104 switches between the process of calibrating the line of sight and the process of detecting the line of sight based on the information input from the input device 3 and the information input from the parameter calculation unit 107. When the information input from the input device 3 includes information instructing execution of the process of calibrating the line of sight, and the calibration parameter after the input of the information is not determined, the process switching unit 104 detects the line of sight. The processing performed by the apparatus 1 is switched to calibration (processing for calibrating the line of sight). On the other hand, when the information input from the input device 3 does not include information for instructing the execution of the process of calibrating the line of sight, or when the calibration parameter has already been determined, the process switching unit 104 is the line-of-sight detection apparatus 1. The processing to be performed is switched to the processing for calculating the line of sight.

  The feature amount correcting unit 105 calculates the interpupillary distance of both eyes based on the pupil position extracted by the feature amount calculating unit 103, and then corrects the feature amount of the eyeball as necessary. When the calculated interpupillary distance is different from the preset interpupillary distance, the feature amount correcting unit 105 determines the eyeball based on the ratio between the calculated interpupillary distance and the preset interpupillary distance. Correct the feature value. The preset interpupillary distance is registered in the eyeball feature database 121.

  Based on the eyeball feature amount calculated by the feature amount calculation unit 103 (or the corrected eyeball feature amount when corrected by the feature amount correction unit 105) and the eyeball feature database 121, the feature determination unit 106 Determine the eye characteristics of the person. In the eyeball feature database 121, eyeball feature amounts when a plurality of persons with different eyeball features gaze at each of a plurality of gazing points are registered. The feature determination unit 106 includes the feature amount of the eyeball (or the feature amount of the eyeball corrected by the feature amount correction unit 105) calculated from the image when the target person is gazing at any one of a plurality of gazing points. Based on the eyeball feature database 121, the eyeball feature of the target person is determined. That is, the feature determination unit 106 determines that the target person has a plurality of gazing points based on the feature amount of the eyeball when the target person is gazing at any one of the plurality of gazing points and the eyeball feature database 121. The feature amount of the eyeball when gazing at each is determined.

  The parameter calculation unit 107 calculates a calibration parameter for the line of sight of the target person based on the eyeball feature determined by the feature determination unit 106. After calculating the calibration parameter, the parameter determination unit 107 transmits the calculated calibration parameter to the line-of-sight calculation unit 108 and notifies the processing switching unit 104 that the calibration parameter has been calculated.

  The line-of-sight calculation unit 108 calculates the line of sight of the target person based on the feature amount of the eyeball calculated by the feature amount calculation unit 103 and the calibration parameter. The line-of-sight calculation unit 108 causes the line-of-sight storage unit 122 to store the calculated line of sight of the target person in time series.

  FIG. 2 is a diagram illustrating an example of an eyeball feature database. FIG. 3 is a diagram for explaining the feature amount of the eyeball and the inter-pupil distance.

  As shown in FIG. 2, the eyeball feature database 121 includes an identifier (ID), a feature amount (PG1 to PG5) of the eyeball when gazing at the gazing point, and an interpupillary distance (PD).

  The identifier (ID) is a value for identifying a record including an eyeball feature registered in the eyeball feature database 121. In the eyeball feature database 121 of FIG. 2, the eyeball feature for one person is divided into the eyeball feature of the right eye and the eyeball feature of the left eye, and a separate identifier is assigned to each.

  The feature amount of the eyeball when gazing at the gazing point is a two-dimensional coordinate representing the relative position of the center of the pupil viewed from the center of corneal reflection in the image captured when gazing at the gazing point. . In the eyeball feature database 121 in FIG. 2, eyeball feature amounts for each of the five gazing points PG1 to PG5 are registered as eyeball features for one identifier (record).

  The interpupillary distance (PD) is the distance between the right eye pupil and the left eye pupil in the image used to create the eyeball feature database 121. The eyeball feature database 121 is created using a human image captured in a state where the distance from the infrared camera 201 to the person is a predetermined distance (for example, 600 mm). That is, the interpupillary distance in the eyeball feature database 121 is the interpupillary distance when the distance from the infrared camera 201 to the person is a predetermined distance.

  As described above, the plurality of records registered in the eyeball feature database 121 include the feature amount of the eyeball and the interpupillary distance when one person gazes each of the five gazing points PG1 to PG5. .

  The image acquired by the line-of-sight detection device 1 is an image including the eyeball of the target person imaged by the infrared camera 201 in a state where the face of the target person is irradiated with infrared rays as described above. Therefore, as shown in FIG. 3A, a cornea reflection 601 by infrared rays is reflected in the portion of the eyeball 6 in the acquired image 5.

  In the line-of-sight detection device 1 that detects the line of sight by the pupil-corneal reflection method, for example, the line of sight is detected based on the relative position of the center Q2 of the pupil 602 viewed from the reference point Q1 with the center Q1 of the corneal reflection 601 as the reference point. To do. At this time, the line-of-sight detection apparatus 1 performs calibration for reducing the error between the line of sight calculated based on the image 5 and the actual line of sight of the target person and detecting the line of sight with high accuracy.

  In the line-of-sight detection device 1 according to the present embodiment, an image in which the target person is gazing at any one of a plurality of predetermined gazing points is acquired, and the eyeball feature amount and the eyeball feature database in the image are acquired. Perform calibration based on the above. The gazing point to be watched by the target person is any one of a plurality of gazing points set when the eyeball feature database is created. When performing calibration using the eyeball feature database 121 of FIG. 2, the line-of-sight detection device 1 acquires an image in which the target person is gazing at any one of the five gazing points PG1 to PG5. Perform calibration.

  The line-of-sight detection device 1 acquires an image 5 captured while gazing at one point (gaze point) designated by the target person, and calculates the feature amount of the right eye and the feature amount of the left eye in the acquired image 5. To do. Thereafter, the line-of-sight detection apparatus 1 corrects the right eye feature amount and the left eye feature amount in the image 5 as necessary, based on the interpupillary distance in the image 5 and the interpupillary distance in the eyeball feature database 121. As shown in FIG. 3B, the right eye 6R and the left eye 6L of the target person are shown in the image 5 acquired from the infrared camera 201. The inter-pupil distance PD is a distance (pixel difference) between the center Q2R of the pupil 602R of the right eye 6R and the center Q2L of the pupil 602L of the left eye 6L on the image 5. In the image 5, a pixel including the center Q2R of the pupil 602R of the right eye 6R is represented by coordinates (PRx, PRy), and a pixel including the center Q2L of the pupil 602L of the left eye 6L is represented by coordinates (PLx, PLy). Then, the interpupillary distance PD is calculated by the following equation (1).

  For example, the pixel including the center Q2R of the pupil 602R of the right eye 6R in the image 5 is the pixel at coordinates (173, 124), and the pixel including the center Q2L of the pupil 602L of the left eye 6L is the pixel at coordinates (275, 131). In this case, the inter-pupil distance PD is 102.2 (pix). In this case, the interpupillary distance calculated from the image 5 is different from the interpupillary distance in the image feature database 121. Therefore, the line-of-sight detection device 1 (feature amount correction unit 105) corrects the calculated feature amount of the eyeball based on the ratio between the calculated interpupillary distance and the interpupillary distance in the database. For example, when the right eye feature amount calculated from the image 5 is (0, −2), the feature amount correction unit 105 calculates the feature amount to (0 × 100/102) based on the ratio of the inter-pupil distance. .2, -2 × 100 / 102.2) ≈ (0, −1.95).

  When the feature amount of the eyeball is corrected, the line-of-sight detection device 1 (feature determination unit 106) refers to the feature amount of the eyeball of the gazing point that is focused on the target person in the eyeball feature database 121 and matches the corrected feature amount. To extract records that contain feature quantities. The extracted record includes the feature amount of the eyeball when gazing at the five gazing points PG1 to PG5. Therefore, the feature determination unit 106 determines the feature quantities of the five eyeballs included in the extracted record as the feature quantities of the eyeball when the target person gazes at the five gazing points PG1 to PG5, respectively. The feature determination unit 106 determines the feature amount of the eyeball when the target person gazes at the five gazing points PG1 to PG5 with each of the right eye and the left eye. Thereafter, the line-of-sight detection device 1 (parameter calculation unit 107) calculates a calibration parameter based on the determined feature amounts of the five eyeballs for the right eye and the feature amounts of the five eyeballs for the left eye.

  As described above, in the line-of-sight detection device 1 according to the present embodiment, the target person has each of a plurality of gaze points based on the image when the target person is gazing at one gaze point and the image feature database 121. A feature amount of the eyeball when gazing at is determined, and a calibration parameter is calculated. That is, the gaze detection apparatus 1 according to the present embodiment may have only one gaze point (one point) at which the target person gazes at one calibration. For this reason, according to the present embodiment, it is possible to reduce the burden on the target person to watch the gazing point during calibration.

  The above eyeball feature database 121 is created by, for example, the database creation device 7 shown in FIG.

FIG. 4 is a diagram illustrating a functional configuration of the database creation device.
As illustrated in FIG. 4, the database creation device 7 includes a gaze instruction unit 701, an image acquisition unit 702, a feature amount calculation unit 703, a distance calculation unit 704, a registration unit 705, and an eyeball feature database 121. Prepare.

  When a command for starting eye feature registration processing is input from the input device 3, the gaze instruction unit 701 generates information indicating a gaze point to be watched by the target person, and transmits the generated information to the output device 4. . The gaze instruction unit 701 causes the image acquisition unit 702 to acquire an image of the target person after transmitting information indicating the gaze point to the output device 4.

  The image acquisition unit 702 acquires an image including the eyeball of the target person captured by the infrared camera 201 of the line-of-sight sensor 2. As the line-of-sight sensor 2 (infrared camera 201 and infrared LED 202), a sensor capable of capturing an image under the same conditions as the line-of-sight sensor 2 used in combination with the line-of-sight detection device 1 is used.

  The feature amount calculation unit 703 extracts the pupil and corneal reflection from the image acquired from the infrared camera 201, and calculates the feature amount of the eyeball of the target person shown in the image. Similar to the feature quantity calculation unit 103 of the eye gaze detection apparatus 1, the feature quantity calculation unit 703 of the database creation device 7 uses the position of the center of the cornea reflection and the center of the pupil in each of the right eye and the left eye as the feature quantity of the eyeball. A value representing the relationship is calculated. The positional relationship between the center of the corneal reflection and the center of the pupil is, for example, two-dimensional coordinates representing the relative position of the center of the pupil viewed from the center of the corneal reflection.

  The distance calculation unit 704 calculates the inter-pupil distance (pixel difference) between the center of the right eye pupil and the center of the left eye pupil in the image.

  The registration unit 705 registers a set of the eyeball feature amount calculated by the feature amount calculation unit 703 and the interpupillary distance calculated by the distance calculation unit 704 in the eyeball feature database 121. The registration unit 705 assigns an identifier for the right eye to the set of the right eye feature amount and the interpupillary distance, and registers the set in the eye feature database 121 as one record. Also, the registration unit 705 assigns an identifier for the left eye to the set of the left eye feature amount and the interpupillary distance, and registers it in the eyeball feature database 121 as one record.

  When the operator of the database creation device 7 or the target person operates the input device 3 to input a command for starting the eye feature registration process, the database creation device 7 starts the process shown in FIG.

  FIG. 5 is a flowchart for explaining eyeball feature database creation processing according to the first embodiment.

  First, the database creation device 7 designates one gazing point among a plurality of gazing points (step S11), and acquires an image including the eyeball of the person gazing at the designated gazing point (step S12). . Thereafter, the database creation device 7 determines whether or not all of the plurality of gazing points have been designated (step S13). If there is a point of interest that has not been specified (step S13; No), the database creation device 7 repeats the processes of steps S11 and S12. The gaze instruction unit 701 performs the process in step S11. The processing in step S12 is performed by the image acquisition unit 702 in cooperation with the infrared camera 201. The determination in step S13 is performed by the gaze instruction unit 701 or the image acquisition unit 702.

  In step S11, the gaze instruction unit 701 designates one gaze point among a plurality of predetermined gaze points according to a predetermined designation rule. The designation rule is, for example, a rule of designating a first gaze point in a preset designation order among gaze points that are not designated in the process for one person.

  After designating the gaze point, the gaze instruction unit 701 notifies the person of the gaze point designated using the output device 4 and causes the gaze point to be watched. For example, when the output device 4 is a speaker, the gaze instruction unit 701 outputs to the output device 4 an audio signal including a sound that causes the designated gaze point to be watched. When the output device 4 is a display device, the gaze instruction unit 701 outputs an image signal including an image representing the gazing point to the output device 4. The gaze instruction unit 701 causes the person to gaze at the gaze point using the output device 4, and then causes the image acquisition unit 702 to obtain an image including the eyeball of the person who gazes at the gaze point.

  In step S12, the image acquisition unit 702 causes the infrared camera 201 of the line-of-sight sensor 2 to capture an image and acquires an image captured by the infrared camera 201.

  The gaze instruction unit 701 and the image acquisition unit 702 specify all of a plurality of predetermined gazing points, and obtain a plurality of images in which one person is gazing at each of the plurality of gazing points. Repeat the process. When the determination in step S13 is performed by the gaze instruction unit 701, the image acquisition unit 702 notifies the gaze instruction unit 701 that the image has been acquired every time an image is acquired. Upon receiving the notification from the image acquisition unit 702, the gaze instruction unit 701 performs the determination in step S13. When there is an unspecified gaze point (step S13; No), the gaze instruction unit 701 designates the next gaze point, causes the target person to gaze at the gaze point, and causes the image acquisition unit 702 to acquire an image. . On the other hand, when all the gazing points have been designated (step S13; Yes), the database creation device 7 next extracts the pupil and corneal reflection from each of the plurality of images and calculates the feature amount of the eyeball. (Step S14).

  On the other hand, when the determination in step S13 is performed by the image acquisition unit 702, the image acquisition unit 702 has acquired, as the determination in step S13, an image in which a single person is gazing at each of all gazing points. Judgment is made. When there is a gaze point for which an image has not been acquired (step S13; No), the image acquisition unit 702 causes the gaze instruction unit 701 to specify the next gaze point. On the other hand, when the image is acquired by gazing all the gazing points (step S13; Yes), the database creation device 7 next extracts the pupil and corneal reflection from each of the plurality of images, and the feature amount of the eyeball. Is calculated (step S14).

  The feature amount calculation unit 703 performs the process in step S14. The feature amount calculation unit 703 extracts the right eye pupil and corneal reflection and the left eye pupil and corneal reflection from the acquired image according to a known pupil-corneal reflection extraction method in the known pupil-corneal reflection method. . After that, the feature amount calculation unit 703, based on the positional relationship between the center of the corneal reflection of the right eye and the center of the pupil, the feature amount of the right eye's eyeball when gazing at each gazing point (feature of the right eye) Amount). The feature amount extraction unit 703 also determines the feature amount of the left eye's eyeball when gazing at each gazing point based on the positional relationship between the center of the corneal reflection of the left eye and the center of the pupil (feature of the left eye). Amount). The feature amount calculation unit 703 associates the right eye feature amount and the left eye feature amount extracted from each of the plurality of images with the gazing point that the person is gazing at the time of capturing each image, and the distance calculation unit 704. Send to.

  When the processing in step S14 is completed, the database creation device 7 next calculates the interpupillary distance based on the position of the pupil extracted from each image (step S15). The distance calculation unit 704 performs the process in step S15. For example, the distance calculation unit 704 first calculates the inter-pupil distance (number of pixels) between the center of the right eye pupil and the center of the left eye pupil for each of a plurality of images with different gazing points. Thereafter, the distance calculation unit 704 calculates the average value of the calculated plurality of inter-pupil distances, and sets this as the inter-pupil distance of the person shown in the image. The distance calculation unit 704 transmits the right and left eye feature amounts received from the feature amount calculation unit 703 and the interpupillary distance calculated by the distance calculation unit 704 to the registration unit 705.

  When the processing of step S15 is completed, the database creation device 7 then registers the right eye feature amount, left eye feature amount, and interpupillary distance calculated from each image in the registration unit 705 in the eye feature database 121. Is performed (step S16). The registration unit 705 assigns an identifier for the right eye to the set of the feature amount of the right eye and the interpupillary distance when each gazing point is being watched, and registers it in the eyeball feature database 121 as one record. . At this time, the feature amount of the right eye is registered in the gaze point field associated with the feature amount of each right eye among the plurality of gaze points in the eyeball feature database 121. In addition, the registration unit 705 assigns an identifier for the left eye to the set of the feature amount of the left eye and the interpupillary distance when gazing at each gazing point, and stores it in the eyeball feature database 121 as one record. sign up. At this time, the feature amount of the left eye is registered in the gaze point field associated with the feature amount of each left eye among the plurality of gaze points in the eyeball feature database 121.

  When the process of step S16 is completed, the database creation device 7 determines whether or not to continue the creation process of the eyeball feature database (step S17). In the determination in step S17, the database creation device 7 waits for input of information indicating whether or not to continue the creation process, for example. For example, when the operator inputs information indicating that the creation process is continued by operating the input device 3, that is, when the creation process is continued (step S17; Yes), the database creation device 7 performs steps S11 to S16. Process. On the other hand, when the operator inputs information indicating that the creation process is to be ended by operating the input device 3, that is, when the creation process is not continued (step S17; No), the database creation apparatus 7 ends the creation process. .

  The eyeball feature database 121 created in the above procedure is transferred to the line-of-sight detection device 1 via a portable recording medium such as a memory card or an optical disk mounted with a flash memory, or a transmission cable, and stored in the line-of-sight detection device 1. Store in the department.

  When the eyeball feature database 121 is created, as described above, when the plurality of persons having different corneal radii (eyeball sizes) gaze at a plurality of gazing points, the feature amount of the eyeball and between the pupils Calculate and register the distance. At this time, for example, it is preferable that the corneal radius is in the range of 6.0 mm to 9.0 mm at intervals of 0.1 mm, and the feature amount of the eyeball at each corneal radius is registered in the eyeball registration database 121. The corneal radius can be measured by a dedicated measuring device such as a keratometer, for example. In addition, the feature amount of the eyeball registered in the eyeball feature database 121 may be calculated using, for example, an eyeball model reproduced on a computer, instead of calculating from an image obtained by capturing a human eyeball.

  In addition, the eyeball feature information including a set of eyeball feature quantities when each of a plurality of gaze points is prepared is not limited to the database 121 shown in FIG. It may be registered (stored) in the storage unit in this manner.

FIG. 6 is a diagram illustrating an application example of the visual line detection device.
The line-of-sight detection device 1 of the present embodiment is applied to a line-of-sight detection system that detects the line of sight 9S of a target person (driver) 9 who drives the vehicle 8, as shown in FIGS. 6 (a) and 6 (b), for example. Is possible. This type of gaze detection system includes a gaze detection device 1, a gaze sensor 2, an input device 3, and an output device such as a speaker 4.

  The line-of-sight sensor 2 is installed, for example, on the dashboard 801 of the vehicle 8 so as to face the target person 9. The line-of-sight detection device 1, the input device 3, and the speaker 4 are installed, for example, in the center console of the vehicle 8. Note that the line-of-sight detection device 1 may be incorporated in, for example, a drive recorder (event data recorder) that records the driving state of the vehicle 8 and information around the vehicle 8.

  The line of sight 9S of the driver (target person 9) when driving the vehicle 8 is directed not only to the front of the vehicle but also to instruments (instrument panel) (not shown), a room mirror 802 in the vehicle interior, a door mirror (not shown), etc. It is done. The line-of-sight detection system detects, for example, the line of sight 9S during driving of the target person 9, and holds the line-of-sight history. The line-of-sight history held by the line-of-sight detection system is used, for example, to determine whether or not the target person 9 is performing appropriate line-of-sight movement for safe driving.

  The eyeball feature database 121 used in the line-of-sight detection device 1 that detects the line of sight 9S of the driver (target person 9) driving the vehicle 8 acquires, for example, an image of a person seated in the driver's seat 803 of the vehicle 8. Create.

FIG. 7 is a diagram illustrating an image acquisition method when creating an eyeball feature database.
When detecting the line of sight 9S of the target person 9 who is driving the vehicle 8 and using the line-of-sight detection device 1, the eyeball feature database 121 shows that the person seated in the driver's seat 803 in a vehicle of the same vehicle type as the vehicle 8 is gazing at the gazing point. Get the image when you are doing. At this time, the line-of-sight sensor 2 is installed at a position that is substantially the same as the installation position of the line-of-sight sensor 2 in the line-of-sight detection system, that is, a position facing the person 9 seated on the driver's seat 803 on the dashboard 801. Further, as shown in FIG. 7A, the driver seat 803 has a predetermined distance (for example, 600 mm) from the infrared camera 201 of the line-of-sight sensor 2 to the eyeballs 6R and 6L of the person 9 seated on the driver seat 803. Adjust the position so that.

  In this state, the line-of-sight sensor 2, the input device 3, the speaker 4, and the like are connected to the database creation device 7, and the eyeball feature database 121 is created according to the flowchart of FIG. At this time, for example, the database creation device 7 sequentially gazes a plurality of gazing points set at positions in front of the vehicle with respect to the eyeballs 6R and 6L of the person 9, and the person 9 when gazing at each gazing point. Get the image. For example, as shown in FIG. 7B, the plurality of gazing points include four corners PG1, PG2, PG4, and PG5 in the field of view 10 when the person 9 is looking in front of the vehicle, Let it be 5 points of the center PG3. The five gazing points PG1 to PG5 are provided, for example, by sticking seal-like markers 11A to 11E to respective points in the passenger compartment.

  For example, the database creation device 7 first causes the person 9 to gaze at the upper left gazing point PG1, and acquires an image captured by the infrared camera 201 in that state. Next, the database creation device 7 causes the person 9 to gaze at the upper right gazing point PG2, and acquires an image captured by the infrared camera 201 in that state. Thereafter, the database creation device 7 looks at the image when the person 9 is gazing at the center PG3 of the visual field 10, the image when gazing at the lower left gazing point PG4, and the lower right gazing point PG5. Images are acquired sequentially. Then, the database creation device 7 calculates the feature amount of the right eye and the left eye from each of the acquired five images, calculates the inter-pupil distance, and registers it in the eye feature database 121.

  Note that the gazing points PG1 to PG5 shown in FIG. 7B are merely examples, and the positions and shapes of the gazing points PG1 to PG5 and the number of gazing points can be changed as appropriate.

  For example, when the target person (driver) 9 operates the input device 3 and inputs a command to start the eye gaze detection process, the eye gaze detection device 1 provided with the eyeball feature database 121 created by the above method is shown in FIG. The process shown in FIG.

  FIG. 8 is a flowchart for describing processing performed by the visual line detection device according to the first embodiment.

  As shown in FIG. 8, when the line-of-sight detection device 1 starts operating, first, calibration parameter determination processing (step S1) is performed. In step S <b> 1, the line-of-sight detection device 1 designates one of the plurality of gazing points set when creating the eyeball feature database 121 (first gazing point), and the target person Acquire an image when gazing at the gazing point. Thereafter, the line-of-sight detection device 1 uses the eyeball feature database 121 based on the center position of the pupil and the center position of the corneal reflection in the image of the target person who is gazing at the designated single gazing point. To decide. The determined eyeball features include a plurality of eyeball feature amounts corresponding to the eyeball feature amounts when the target person gazes each of the plurality of gaze points. Therefore, after determining the eyeball characteristics, the line-of-sight detection apparatus 1 calculates the calibration parameters according to a known calculation method for calculating the calibration parameters by causing the target person to gaze a plurality of gazing points in the pupil-corneal reflection method.

  The processing in step S1 is performed by the image acquisition unit 101, the gaze instruction unit 102, the feature amount calculation unit 103, the feature amount correction unit 105, the feature determination unit 106, and the parameter calculation unit 107.

  After determining the calibration parameters in step S1, the line-of-sight detection device 1 performs a line-of-sight detection process (step S2). In step S2, the line-of-sight detection device 1 detects (calculates) the line of sight of the target person based on the center position of the pupil and the center position of corneal reflection in the image of the target person and the calibration parameter determined in step S1. . In step S <b> 2, the line-of-sight detection apparatus 1 calculates the direction and position of the line of sight of the target person according to a known line-of-sight calculation method in the pupil-membrane reflection method.

  The processing in step S2 is performed by the image acquisition unit 101, the feature amount calculation unit 103, and the line-of-sight calculation unit 108.

  FIG. 9 is a flowchart for explaining the contents of the calibration parameter determination process according to the first embodiment.

  In the calibration parameter determination process (step S1), the line-of-sight detection device 1 first acquires an image including the eyeball of the target person that is captured while the target person is gazing at one gaze point (step S101). . The process of step S101 is performed by the image acquisition unit 101 and the gaze instruction unit 102. In step S <b> 101, the image acquisition unit 101 first causes the gaze instruction unit 102 to perform processing for gaze the target person at a predetermined one gaze point (first gaze point). The gaze instruction unit 102 outputs to the output device (speaker) 4 an audio signal that gazes at any one of the gaze points PG1 to PG5 (for example, the gaze point PG3) of the eyeball registration database 121, and sets the gaze point PG3 to the target person. Let them watch. On the other hand, the image acquisition unit 101 causes the infrared camera 201 of the line-of-sight sensor 2 to capture an image in a state where the target person is gazing at a predetermined gazing point (gazing point PG3), and an image captured by the infrared camera 201 To get. Thereafter, the image acquisition unit 101 transmits the acquired image to the feature amount calculation unit 103.

  When the image of the target person is acquired, the line-of-sight detection device 1 then calculates the feature amount of the eyeball by extracting the pupil and corneal reflection from the acquired image in the feature amount calculation unit 103 (step S102). Similar to the feature extraction unit 703 of the database creation device 7, the feature extraction unit 103 extracts the center of the pupil and the center of corneal reflection in each of the right eye and the left eye in the image, and viewed from the center of corneal reflection. The relative position of the center of the pupil is calculated as the feature amount of the eyeball. The feature amount calculation unit 103 extracts the pixel including the center of the pupil and the pixel including the center of corneal reflection in the image according to a known extraction method in the pupil-corneal reflection method, and then the pupil viewed from the center of corneal reflection. The relative position of the center of is calculated.

  The feature amount calculation unit 103 transmits the calculated feature amount of the eyeball to the process switching unit 104. The process switching unit 104 switches the transmission destination of the feature amount of the eyeball depending on whether the process performed by the visual line detection device 1 is the calibration parameter determination process (step S1) or the visual line detection process (step S2). . When the process performed by the line-of-sight detection device 1 is the calibration parameter determination process (step S <b> 1), the process selection unit 104 transmits the feature amount of the eyeball to the feature amount correction unit 105.

  Therefore, when the calibration parameter determination process is performed, the line-of-sight detection apparatus 1 next calculates the inter-pupil distance in the image based on the position of the pupil extracted from the image in the feature amount correction unit 105 (step S103). ). The feature amount correcting unit 105 calculates the expression based on the coordinates indicating the position of the pixel including the center of the right eye pupil and the coordinates indicating the position of the pixel including the center of the left eye pupil in one acquired image. The interpupillary distance (pixel difference) is calculated from (1).

  Next, the feature amount correcting unit 105 determines whether or not the calculated interpupillary distance matches the interpupillary distance in the eyeball feature database 121 (step S104). When the calculated interpupillary distance does not match the interpupillary distance in the eyeball feature database 121 (step S104; No), the feature amount correcting unit 105 corrects the feature amount of the eyeball based on the ratio of the interpupillary distance (step S105). ), And transmits the corrected feature amount of the eyeball to the feature determination unit 106. On the other hand, when the calculated interpupillary distance matches the interpupillary distance in the eyeball feature database 121 (step S104; Yes), the feature amount correcting unit 105 skips the process of step S105 and calculates the eyeball feature calculated in step S103. The amount is transmitted to the feature determination unit 106.

  When the processes of steps S103 to S105 are completed, the line-of-sight detection device 1 next determines eye features used for calibration with reference to the eye feature database 121 in the feature determination unit 106 (step S106). The feature determination unit 106 refers to the feature amount of the eyeball when gazing at the gazing point designated in step S <b> 101 among the feature amounts of the eyeball registered in the eyeball feature database 121. Then, the feature determination unit 106 extracts a record including the feature amount of the eyeball that matches the feature amount of the eyeball received from the feature amount correction unit 105, and uses the eyeball feature amount set included in the record for calibration. To decide. The feature determination unit 106 transmits the determined eyeball feature (a set of eyeball feature amounts associated with a certain identifier) to the parameter calculation unit 107.

  When the process of step S106 is completed, the line-of-sight detection device 1 then calculates a calibration parameter based on the determined eye feature in the parameter calculation unit 107 (step S107), and uses the calculated calibration parameter as the calibration parameter of the target person. To decide. The parameter calculation unit 107 calculates a calibration parameter according to a known calculation method in the pupil-corneal reflection method.

  When the process of step S107 is completed, the parameter calculation unit 107 transmits the calculated (determined) calibration parameter to the line-of-sight detection unit 108 and notifies the process switching unit 104 that the calibration parameter for the current target person has been determined. Notice. Thereby, the calibration parameter determination process in the line-of-sight detection device 1 ends.

  After determining the calibration parameters in the calibration parameter determination process (steps S101 to S107), the line-of-sight detection device 1 performs the process shown in FIG. 10 as the line-of-sight detection process (step S2).

FIG. 10 is a flowchart for explaining the contents of the line-of-sight detection process.
When the line-of-sight detection process is started, the line-of-sight detection device 1 first acquires an image including the eyeball of the target person (step S201). The process of step S201 is performed by the image acquisition unit 101. In addition, after finishing the process of step S101 of FIG. 9, the image acquisition part 101 may continue acquiring the image imaged with the infrared camera 201 regularly, or an infrared ray will be triggered when the calibration parameter is determined. Acquisition of an image captured by the camera 201 may be started. The image acquisition unit 101 transmits the acquired image to the feature amount calculation unit 103.

  When the image of the target person is acquired, the line-of-sight detection device 1 then calculates the feature amount of the eyeball by extracting the pupil and corneal reflection from the acquired image in the feature amount calculation unit 103 (step S202). In step S202, the feature amount calculation unit 103 extracts the center of the pupil and the center of corneal reflection in each of the right eye and the left eye in the image, as in the process of step S102, and the pupil as viewed from the center of the corneal reflection. The relative position of the center is calculated as the feature amount of the eyeball. The feature amount calculation unit 103 extracts the pixel including the center of the pupil and the pixel including the center of corneal reflection in the image according to a known extraction method in the pupil-corneal reflection method, and then the pupil viewed from the center of corneal reflection. The relative position of the center of is calculated.

  The feature amount calculation unit 103 transmits the calculated feature amount of the eyeball to the process switching unit 104. As described above, the process switching unit 104 determines the feature amount of the eyeball depending on whether the process performed by the visual line detection device 1 is the calibration parameter determination process (step S1) or the visual line detection process (step S2). Switch the destination of. When the line-of-sight detection device 1 determines the calibration parameter as described above, the parameter determination unit 107 transmits information notifying the processing switching unit 104 that the calibration parameter has been determined. When the process switching unit 104 receives the information from the parameter determination unit 107, the process switching unit 104 determines that the process performed by the line-of-sight detection device 1 has been switched to the line-of-sight detection process. Then, when the line-of-sight detection device 1 is performing the line-of-sight detection process, the process selection unit 104 transmits the feature amount of the eyeball to the line-of-sight calculation unit 108.

  Therefore, when the line-of-sight detection process is being performed, the line-of-sight detection apparatus 1 next calculates the line of sight of the target person based on the calculated feature amount of the eyeball and the calibration parameter in the line-of-sight calculation unit 108 (step) S203). The line-of-sight calculation unit 108 calculates the direction and position of the line of sight of the target person according to a known line-of-sight calculation method in the pupil-cornea reflection method.

  When the gaze of the target person is calculated, the gaze calculation unit 108 stores the calculated gaze in the gaze storage unit 122 (step S204).

  When the processes of steps S201 to S204 are finished, the line-of-sight detection device 1 determines whether or not to continue the detection process (step S205). For example, when the target person operates the input device 3 and inputs information for ending the detection process, when the detection process is not continued (step S205; No), the line-of-sight detection device 1 ends the line-of-sight detection process. On the other hand, when continuing a detection process (step S205; Yes), the gaze detection apparatus 1 repeats a gaze detection process (step S201-S204).

FIG. 11 is a diagram illustrating the correlation between the position of the pupil and the position of corneal reflection and the corneal radius.
When the gaze of the target person is detected by the pupil-corneal reflection method as in the gaze detection apparatus 1 of the present embodiment, the position of the pupil and the corneal reflection in the image captured by the infrared camera 201, the corneal radius of the target person, and Have a correlation.

  For example, as shown in FIG. 11, when the eyeball centers O of two eyeballs 61 and 62 having different corneal radii are overlapped, the directions of the pupil centers Q21 and Q22 viewed from the eyeball center O (direction of the line of sight 9S) Are in the same direction regardless of the corneal radii R1, R2. The directions of the corneal reflection centers Q11 and Q12 viewed from the center O of the eyeball are not related to the corneal radii R1 and R2, and the line segment connecting the infrared LED 202 and the center O of the eyeball, the infrared camera 201 and the eyeball. This is the direction of the angle that bisects the angle θ formed by the line connecting the center O. Therefore, when the distance from the infrared camera 201 to the center O of the eyeball is the same, the distance from the center of corneal reflection to the center of the pupil is the distance in the eyeball 61 having a large corneal radius and the distance in the eyeball 62 having a small corneal radius. Longer than.

  Furthermore, in real space, the distance from the infrared camera 201 to the center O of the eyeball is very long compared to the cornea radii R1 and R2 of the eyeball. Therefore, the correlation between the distance from the center of corneal reflection on the imaging surface of the infrared camera 201 to the center of the pupil and the corneal radius can be expressed as shown in FIG.

  FIG. 12 is a diagram for explaining the correlation between the position of the pupil and the position of corneal reflection on the image and the corneal radius.

  The distance F from the imaging surface 203 of the infrared camera 201 in FIG. 12 to the center O of the eyeball is much longer than the cornea radii R1 and R2 of the eyeballs 61 and 62. Therefore, the center Q11. The infrared rays from Q12 toward the imaging surface 203 and the infrared rays from the pupil centers Q21 and Q22 in the eyeball toward the imaging surface 203 can be regarded as parallel regardless of the corneal radii R1 and R2.

  Here, it is assumed that the directions of the pupil centers Q21 and Q22 (the direction of the line of sight 9S) viewed from the center O of the eyeball are the same. In this case, the distance L1 from the corneal reflection center Q11 of the eyeball 61 to the pupil center Q21 on the imaging surface 203 is longer than the distance L2 from the corneal reflection center Q12 of the eyeball 62 to the pupil center Q22. As described above, even when a plurality of target persons see the same direction at the same position when viewed from the infrared camera 201, the feature amount of the eyeball in the image depends on the cornea radius (eyeball size) of each target person. Different values. That is, when the installation positions of the infrared camera 201 and the infrared LED 202 are known and the distance from the infrared camera 201 to the eyeball of the target person is a predetermined distance, the positional relationship between the center of corneal reflection and the center of the pupil is It depends only on the corneal radius.

  For example, when the distance from the center of corneal reflection on the imaging surface 203 (image) to the center of the pupil is L1, the cornea radius of the eyeball of the target person can be regarded as the same as the cornea radius R1 of the eyeball 61. . Therefore, if the eyeball feature amount for a person whose cornea radius is R1 is registered in the eyeball feature database 121, the eyeball feature amount when one gaze point is watched, the eyeball feature database 121, Based on the above, it is possible to determine a set of feature quantities of the eyeball of the target person.

  That is, if the positional relationship between the center of corneal reflection and the center of the pupil when the distance from the infrared camera 201 to the eyeball of the target person is a predetermined distance is known, the corneal radius of the target person ( The size of the eyeball) can be estimated. If the corneal radius of the target person can be estimated, the positional relationship between the center of the cornea reflection and the center of the pupil in the image of the eyeball of the target person looking in a certain direction can be determined based on the position in the image of the other person's eyeball. It can be estimated from the relationship. Therefore, by registering a set of eyeball feature quantities when a plurality of persons with different eyeball sizes (corneal radii) gaze at a plurality of gaze points in the eyeball feature database 121, one gaze point is registered. It is possible to determine the feature amount of the eyeball of the target person based on the above. Therefore, according to the present embodiment, it is possible to determine a plurality of feature quantities necessary for determining the calibration parameter of the line of sight simply by having the target person gaze at one gaze point, and the target person gazes at the gaze point. It becomes possible to reduce the burden.

  In addition, in the calibration parameter determination process according to the present embodiment, the calibration parameter is determined based on the image acquired using the set of one infrared camera 201 and one infrared LED 202 and the eyeball feature parameter 121. It is possible to calculate. Therefore, according to this embodiment, there is no need to provide a plurality of infrared cameras 201 and / or infrared illuminations, and an increase in the number of devices in the line-of-sight detection system including the line-of-sight detection device 1, the infrared camera 201, and the infrared LED 202 is suppressed. Is possible. Therefore, an increase in the introduction cost of the line-of-sight detection system can be suppressed. Further, in the calibration parameter determination process performed by the line-of-sight detection device 1, a record (eyeball feature) including a feature quantity that matches the feature quantity of the eyeball in one acquired image is extracted from the eyeball feature database 121, and the extracted eyeball feature is extracted. This is a process of determining the calibration parameter based on the above. Therefore, according to this embodiment, it is possible to determine the calibration parameter without performing complicated processing, and it is possible to suppress an increase in the processing load of the line-of-sight detection device 1.

  Note that the feature amount of the eyeball calculated from the image captured by the infrared camera 201 is not limited to the relative position of the center of the pupil viewed from the center of the corneal reflection, but is the relative position of the center of the corneal radius viewed from the center of the pupil. May be. Furthermore, the value used for determining whether or not to correct the feature amount of the eyeball is not limited to the inter-pupil distance, but may be the distance of corneal reflection of both eyes.

  In addition, as described above, the information on eyeball features including a set of eyeball feature quantities when each of a plurality of gazing points is prepared is not limited to the database 121 shown in FIG. May be registered (stored) in the storage unit in another manner that can be identified.

  The line-of-sight detection device 1 according to the present embodiment is not limited to the line-of-sight detection system that detects the line of sight of the driver (target person 9) who drives the vehicle 8, and can be applied to various line-of-sight detection systems.

FIG. 13 is a diagram illustrating another application example of the visual line detection device.
The line-of-sight detection device 1 of the present embodiment is also applicable to a line-of-sight detection system 14 that detects the line of sight 9S of a person 9 looking at the display device 12, as shown in FIG. In this type of line-of-sight detection system 14, for example, the line-of-sight detection device 1 can be incorporated into an information processing device 13 that performs various processes including display control of images, images, text data, and the like displayed on the display device 12. . Furthermore, although not shown in FIG. 13, the database creation device 7 can be incorporated in the information processing device 13.

  The process for creating the eyeball feature database 121 used in the line-of-sight detection system 14 in FIG. 13 may be the same as the process in FIG. 5. First, each of the plurality of gazing points PG1 to PG5 set on the display surface 1201 of the display device 12 is selected. An image of the target person 9 who is gazing is acquired (steps S11 to S13). At this time, the display surface 1201 of the display device 12 includes, for example, an image in which only the gazing point PG1 is displayed, an image in which only the gazing point PG2 is displayed, an image in which only the gazing point PG3 is displayed, and only the gazing point PG4. The displayed image and the image on which only the gazing point PG5 is displayed are sequentially displayed.

  After acquiring a plurality of images, the database creation device 7 calculates the feature amount of the right and left eyeballs for each of the acquired images and calculates the interpupillary distance (steps S14 and S15). Register in the feature database 121 (step S16).

  The created eyeball feature database 121 is stored in the line-of-sight detection device 1 (information processing device 13) via a portable recording medium such as a memory card or an optical disc, or a transmission cable. Also in the line-of-sight detection system 14, the line-of-sight detection process 1 storing the eyeball feature database 121 performs the processes shown in FIGS. 8 to 10.

  The line-of-sight detection system 14 in FIG. 13 can be used for line-of-sight input, for example. When the target person 9 gazes at a predetermined object displayed on the display surface 1201 of the display device 12, the target person 9 is gazing at the information processing apparatus 13 by detecting the line of sight. It is possible to execute processing associated with the object.

  Further, the line-of-sight detection system 14 of FIG. 13 can be used to investigate which information the target person 9 is interested in, for example, an apparatus that provides various types of information such as digital signage.

FIG. 14 is a diagram illustrating a modification of the line-of-sight detection system.
The line-of-sight detection device 1 according to the present embodiment is not limited to the above-described line-of-sight detection system. For example, as shown in FIG. 14A, the eyeball feature database 121 is stored in a management server 15 separate from the line-of-sight detection device 1. The present invention can also be applied to the line-of-sight detection system 17 being used. In this type of line-of-sight detection system 17, the line-of-sight detection device 1 and the management server 15 are communicably connected via a communication network 16 such as the Internet. Although not shown in FIG. 14A, the visual line detection device 1 is connected to the visual sensor 2, the input device 3, and the output device 4 such as a speaker and a display device 12.

  Furthermore, the line-of-sight detection system 17 to which the line-of-sight detection apparatus 1 is applied includes a line-of-sight detection apparatus 1 installed in each of the plurality of vehicles 8 (8A, 8B) as shown in FIG. The eyeball feature database 121 stored in the system may be shared. In this type of gaze detection system 17, for example, a radio communication unit and an antenna 804 are connected to the gaze detection device 1 of each vehicle 8, and communication between each gaze detection device 1 and the eyeball feature database 121 via the communication network 16. Enable. Further, in the line-of-sight detection system 17 shown in FIG. 14B, the line-of-sight information of the target person detected by each line-of-sight detection device 1 can be transferred to the line-of-sight history 1501 of the management server 15. Therefore, the eyeball feature database 121 can be shared by the plurality of eye gaze detection devices 1 and the gaze of a plurality of target persons can be collectively managed by the management device 15. Furthermore, although illustration is omitted, in the line-of-sight detection system 17 shown in FIG. 14B, for example, the calibration parameter creation processing described above may be performed on the management server. In this case, the line-of-sight detection device 1 transmits, for example, the image captured by the infrared camera 201 and the vehicle type information of the vehicle 8 in which the line-of-sight detection device 1 is installed to the management server 15 and calculates the calibration parameters to the management server 15. Let Thereafter, the line-of-sight detection device 1 acquires a calibration parameter from the management server 15 and calculates the line of sight of the driver (target person 9) based on the calibration parameter.

  In addition, the line-of-sight detection system illustrated in FIGS. 6, 13, and 14 is merely an application example of the line-of-sight detection apparatus 1 according to the present embodiment. That is, the line-of-sight detection apparatus 1 is not limited to these line-of-sight detection systems, and can be applied to various line-of-sight detection systems using the pupil-corneal reflection method. For example, the line-of-sight detection apparatus 1 can be applied to a line-of-sight detection system that collects information such as which products the target person (customer) is displaying in the store, such as which products are interested in the store. is there.

  Further, the functional configuration of the line-of-sight detection device 1 shown in FIG. 1 and the processing performed by the line-of-sight detection device 1 are merely examples. The line-of-sight detection device 1 according to the present embodiment may start the calibration parameter determination process when, for example, a human sensor or the like detects that the target person 9 has stopped at a predetermined position. . Moreover, when installing the gaze detection device 1 according to the present embodiment in the vehicle 8, for example, the calibration parameter determination process is started when the target person 9 (driver) starts the engine, and the target person 9 The line-of-sight detection process may be terminated when the engine is stopped.

[Second Embodiment]
In the present embodiment, a processing method for correcting the eyeball feature amount in consideration of the face direction of the target person and determining the calibration parameter in the line-of-sight detection device 1 will be described.

  Note that the functional configuration of the visual line detection device 1 according to the present embodiment may be the configuration illustrated in FIG. Further, the eyeball feature database 121 stored in the line-of-sight detection device 1 according to the present embodiment may have the configuration illustrated in FIG. 2, for example. However, when the eyeball feature database 121 according to the present embodiment is created, an image in which a person is gazing at a gazing point with the face facing the infrared camera 201 is acquired, and the eyeball calculated from the image is acquired. Are registered in the eyeball feature database 121.

  The line-of-sight detection apparatus 1 according to the present embodiment performs the calibration parameter determination process (step S1) and the line-of-sight detection process (step S2) shown in FIG. Note that the line-of-sight detection device 1 according to the present embodiment performs the processing illustrated in FIGS. 15 and 16 as the calibration parameter determination processing.

  FIG. 15 is a flowchart for explaining the contents of the calibration parameter determination process according to the second embodiment. FIG. 16 is a flowchart for explaining the content of processing for correcting the feature amount of the eyeball. Of the processing blocks in the flowcharts of FIGS. 15 and 16, blocks that perform the same processes as those in the flowchart of FIG. 9 are given the same step numbers (eg, S101, S102, etc.) as in FIG.

  In the calibration parameter determination process of the present embodiment, the line-of-sight detection device 1 first acquires an image including the eyeball of the target person that is captured while the target person is gazing at one gaze point (step S101). . The process of step S101 is performed by the image acquisition unit 101 and the gaze instruction unit 102. In step S101, the image acquisition unit 101 and the gaze instruction unit 102 each perform the processing described in the first embodiment. The gaze instruction unit 102 causes the target person to gaze at one gaze point using the output device 4. On the other hand, the image acquisition unit 101 causes the infrared camera 201 to capture an image including the eyeball of the target person who is gazing at one gazing point, and acquires the captured image. Further, the image acquisition unit 101 transmits the acquired image to the feature amount calculation unit 103.

  When the image of the target person is acquired, the line-of-sight detection device 1 then calculates the feature amount of the eyeball by extracting the pupil and corneal reflection from the acquired image in the feature amount calculation unit 103 (step S102). As described in the first embodiment, the feature amount calculation unit 103 extracts a pixel including the center of the pupil and a pixel including the center of corneal reflection in each of the right eye and the left eye in the image, and the center of corneal reflection is extracted. The relative position of the center of the pupil viewed from the above is calculated as the feature amount of the eyeball.

  The feature amount calculation unit 103 transmits the calculated feature amount of the eyeball to the process switching unit 104. When the process performed by the line-of-sight detection device 1 is the calibration parameter determination process, the process switching unit 104 transmits the feature quantity of the eyeball to the feature quantity correction unit 105 as described in the first embodiment.

  Thereafter, the line-of-sight detection apparatus 1 performs a process of correcting the feature quantity of the eyeball (step S110) in the feature quantity correction unit 105. In step S110, the feature amount correcting unit 105 first estimates the face orientation based on the feature amount of the right eye and the feature amount of the left eye. When the estimated face orientation does not face the infrared camera 201, the feature amount correction unit 105 determines that the feature amount of the right eye and the left eye is the feature amount when the face orientation faces the infrared camera 201. The feature amount is corrected so that Subsequently, the feature amount correcting unit 105 performs the processing in steps S103 to S105 described in the first embodiment, that is, the feature of the eyeball when the interpupillary distance on the image is different from the interpupillary distance in the eyeball feature database 121. Perform processing to correct the amount. The feature amount correction unit 105 transmits the corrected feature amount of the eyeball to the feature determination unit 106. If the feature amount of the eyeball is not corrected in step S110, the feature amount correction unit 105 transmits the feature amount of the eyeball calculated by the feature amount calculation unit 103 to the feature determination unit 106.

  When the processing of step S110 is completed, the line-of-sight detection device 1 next determines an eyeball feature to be used for calibration with reference to the eyeball feature database 121 in the feature determination unit 106 (step S106). The feature determination unit 106 refers to the feature amount of the eyeball when gazing at the gazing point designated in step S <b> 101 among the feature amounts of the eyeball registered in the eyeball feature database 121. Then, the feature determination unit 106 determines a set of feature amounts including the same feature amount as the feature amount of the eyeball received from the feature amount correction unit 105 as an eyeball feature used for calibration. The feature determination unit 106 transmits the determined eyeball feature (a set of eyeball feature amounts associated with a certain identifier) to the parameter calculation unit 107.

  When the process of step S106 is completed, the line-of-sight detection device 1 calculates a calibration parameter based on the determined eyeball feature in the parameter calculation unit 107 (step S107). The parameter calculation unit 107 calculates a calibration parameter according to a known calculation method in the pupil-corneal reflection method, and determines the calculated calibration parameter as the calibration parameter of the current target person. When the process of step S107 is completed, the parameter calculation unit 107 transmits the determined calibration parameter to the line-of-sight calculation unit 108 and notifies the process switching unit 104 that the calibration parameter for the current target person has been determined. Thereby, the calibration parameter determination process in the line-of-sight detection device 1 ends.

  Next, with reference to FIG. 16, the process of correcting the feature quantity of the eyeball in step S110 performed by the feature quantity correction unit 105 will be described.

  In the process of step S110, the feature amount correcting unit 105 first estimates the orientation of the face of the target person based on the feature quantity of the eyeball (step S111), and whether or not the face of the target person faces the infrared camera 201. Is determined (step S112). In step S <b> 111, the feature amount correction unit 105 estimates the face orientation based on the magnitude relationship between the right eye feature amount and the left eye feature amount, for example. When the feature amount of the right eye and the feature amount of the left eye are substantially the same, the feature amount correction unit 105 estimates that the target person's face is directly facing the infrared camera 201. When the absolute value of the difference between the right eye feature amount and the left eye feature amount is equal to or greater than the threshold value, the feature amount correction unit 105 estimates that the face is tilted to the right or left.

  As a result of estimating the orientation of the face in step S111, if the face is not directly facing the infrared camera 201 (step S112; No), the feature amount correcting unit 105 next selects the right eye or the left eye based on the face orientation. The feature amount of the eyeball is corrected (step S113). In step S103, the feature amount correction unit 105 determines whether the right eye feature amount and the left eye feature amount are substantially the same based on the face orientation estimated in step S111. The center position of reflection and the center position of the pupil are corrected. After step S113, the feature amount correcting unit 105 calculates the interpupillary distance based on the corrected center position of the right eye pupil and the center position of the left eye pupil (step S114).

  On the other hand, when the face is directly facing the infrared camera 201 (step S112; Yes), the feature amount correction unit 105 skips the process of step S113 and performs the process of step S114. When the process of step S113 is skipped, the feature amount correcting unit 105 calculates the interpupillary distance based on the center position of the right eye pupil and the center position of the left eye pupil used for estimating the face orientation.

  After calculating the interpupillary distance, the feature amount correcting unit 105 next determines whether or not the calculated interpupillary distance matches the interpupillary distance in the eyeball feature database 121 (step S104). When the interpupillary distances do not match (step S104; No), the feature amount correcting unit 105 corrects the feature amount of the eyeball based on the ratio of the interpupillary distances (step S105). The processing performed by the feature amount correction unit 105 in step S105 is as described in the first embodiment. After step S105, the feature amount correcting unit 105 transmits the corrected feature amount of the eyeball to the feature determining unit 106 and ends the process of correcting the feature amount of the eyeball.

  On the other hand, when the interpupillary distances match (step S104; Yes), the feature amount correction unit 105 skips step S105 and transmits the eyeball feature amount to the feature determination unit 106 to correct the eyeball feature amount. Exit. When the interpupillary distances match, the feature amount correction unit 105 outputs the feature amount of the eyeball extracted by the feature amount extraction unit 103 or the feature amount of the eyeball corrected in step S113 to the feature determination unit 106.

FIG. 17 is a diagram for explaining the relationship between the orientation of the face and the feature amount of the eyeball.
FIG. 17A shows a case where the face direction V1 is the direction of the infrared camera 201 (that is, the face is facing the infrared camera 201) and the line-of-sight direction V2 is the direction of the infrared camera 201. 3 shows the feature amount of the eyeball and the inter-pupil distance on the imaging surface 203. In this case, the distance PL from the center of the corneal reflection 601R in the right eye to the center of the pupil 602R and the distance PR from the center of the corneal reflection 601L to the pupil 602L in the left eye viewed from the infrared camera 201 (imaging surface 203). The values are substantially the same (PR≈PL).

  In FIG. 17B, the feature amount of the eyeball and the inter-pupil distance on the imaging surface when the face direction V1 is the direction of the infrared camera 201 and the line-of-sight direction V2 is the direction of the infrared camera 201. It shows. Therefore, also in the case of FIG. 17B, the distance PL from the corneal reflection in the right eye to the pupil and the distance PR from the corneal reflection in the left eye to the pupil are substantially the same value (PR≈PL). .

  Thus, when the face is directly facing the infrared camera 201, the distance PR from the corneal reflection 601R to the pupil 602R in the right eye and the distance PL from the corneal reflection 601L to the pupil 602L in the left eye are substantially the same value. (PR≈PL).

  However, in FIG. 17B, the distance from the infrared camera 201 (imaging surface 203) to the eyeball (face) is longer than the distance from the infrared camera 201 to the eyeball in FIG. Therefore, even if the actual inter-pupil distance PD is the same in FIGS. 17A and 17B, the inter-pupil distance D2 in the imaging surface 203 in the case of (b) is the imaging surface in the case of (a). It becomes smaller than the inter-pupil distance D1 in 203 (D1> D2).

  On the other hand, in FIG. 17C, the feature amount of the eyeball when the face direction V1 is in a different direction from the infrared camera 201 and the line-of-sight direction V2 is the direction of the infrared camera 201 is shown. And the interpupillary distance on the imaging surface 203. In FIG. 17C, the distance from the infrared camera 201 to the right eye is longer than the distance from the infrared camera 201 to the left eye. Therefore, the distance PR from the center of the corneal reflection 601R to the center of the pupil 602R in the right eye is shorter than the distance PL from the center of the corneal reflection 601L to the center of the pupil 602L in the left eye (PR <PL). Further, in FIG. 17C, the face direction V <b> 1 of the target person is facing a direction different from that of the infrared camera 201. Therefore, the interpupillary distance D3 viewed from the infrared camera 201 (imaging plane 203) is calculated based on the actual interpupillary distance PD and the distance from the infrared camera 201 (imaging plane 203) to the eyeball. Shorter than. On the other hand, the feature amount of the eyeball and the interpupillary distance in the eyeball feature database 121 used in the present embodiment are values calculated based on an image captured with the face facing the infrared camera 201. Therefore, when the face is not directly facing the infrared camera 201, in order to correctly compare the feature amount of the eyeball calculated from the image with the feature amount of the eyeball in the eyeball feature database 121, the calculated feature amount is compared with the infrared camera. It is necessary to correct the feature amount in a state of directly facing 201.

  FIG. 18 is a diagram for explaining a method for correcting the feature amount of the eyeball when the face is not directly facing the camera.

  In FIG. 18, as an example of the case where the face is not directly facing the infrared camera 201, the feature amount of the eyeball when the distance from the imaging surface 203 to the right eye is longer than the distance from the imaging surface 203 to the left eye. Show. In this case, the distance from the center KRx of the corneal reflection 601R of the right eye on the imaging surface 203 to the center PRx of the pupil 602R is the distance L from the center KLx of the corneal reflection 601L of the left eye on the imaging surface 203 to the center PLx of the pupil 602L. Shorter than. Therefore, the feature amount correcting unit 105 determines the position of the corneal reflection 601R and the pupil 602R so that the distance from the center of the corneal reflection 601R of the right eye on the imaging surface 203 to the center of the pupil 602R matches the distance L of the left eye. The position is corrected (step S113 in FIG. 16). That is, in step S113, the feature amount correction unit 105 calculates points KRx ′ and PRx ′ on the imaging surface 203 that satisfy the following formula (2).

    KRx'-PRx '= KLx-PLx (2)

  In the equation (2), the point KRx ′ and the point PRx ′ are the virtual right eye on the imaging surface 203 when the target person faces the infrared camera 201 directly with the position of the left eye as a reference. The center of the corneal reflection 601R ′ and the center of the pupil 602R ′.

  When correcting the position of the corneal reflection 601R of the right eye and the position of the pupil 602R, as shown in FIG. 18, the distance from the center Px of the corneal reflection 601R of the right eye to the center Kx of the pupil 602R is the distance L of the left eye. A matching virtual plane 205 is set. Here, when the distance from the lens 204 of the infrared camera 201 to the imaging surface 203 is a distance h and the distance from the lens 204 to the virtual plane 205 is a distance H, the ratio h: H of the two distances h and H is the following number. It is represented by (3-1).

    h: H = (KRx-PRx): (KLx-PLx) (3-1)

  Therefore, the distance H from the lens 204 to the virtual plane 205 is calculated by the following number (3-2).

    H = d × {(KLx−PLx) / (KRx−PRx)} (3-2)

  On the other hand, h of two distances h and H; H is the center Ox of the imaging surface 203, the position KRx of the corneal reflection 601R of the right eye before correction on the imaging surface 203, and the virtual right eye on the imaging surface 203. When the position KRx ′ of the corneal reflection 601R ′ is used, it is represented by the following formula (4-1).

    h: H = (KRx-Ox): (KRx'-Ox) (4-1)

  Therefore, if the distance H from the lens 204 to the virtual plane 205 is known, the position KRx ′ of the virtual right-eye corneal reflection 601R ′ on the imaging surface 203 can be calculated based on the following number (4-2). It becomes possible.

    (KRx′−Ox) = (KRx−Ox) × (H / h) (4-2)

  The two distances h, H of h; H are the center Ox of the imaging surface 203, the position PRx of the right-eye pupil 602R before correction on the imaging surface 203, and the virtual right-eye pupil on the imaging surface 203. When the position PRx ′ of 602R ′ is used, it is represented by the following formula (5-1).

    h: H = (PRx−Ox): (PRx′−Ox) (5-1)

  Therefore, if the distance H from the lens 204 to the virtual plane 205 is known, the position PRx ′ of the virtual right-eye pupil 602R ′ on the imaging surface 203 can be calculated based on the following number (5-2). It becomes.

    (PRx′−Ox) = (PRx−Ox) × (H / h) (5-2)

  When the feature amount of the right eye is corrected based on Expression (3-2), Expression (4-2), and Expression (5-2), the feature amount correction unit 105, that is, the corrected eyeball feature amount, The interpupillary distance is calculated based on the positional relationship between the point PLx and the point PRx ′ on the imaging surface 203 (step S114). If the calculated interpupillary distance does not match the interpupillary distance in the eyeball feature table (step S104; No), the feature amount correcting unit 105 corrects the eyeball feature amount again based on the interpupillary distance ratio. (Step S105).

  By performing the process of correcting the feature amount of the eyeball, the feature amount of the eyeball calculated from the image is the face orientation and the interpupillary distance. When the eyeball feature table 121 is created, the face orientation of the person and the interpupillary distance The feature amount is consistent with. Therefore, according to the present embodiment, it is possible to suppress extraction of erroneous eyeball features due to changes in the eyeball feature amount (the distance between the corneal reflection and the pupil) and the interpupillary distance according to the face orientation. Thus, it is possible to further increase the accuracy of line-of-sight detection.

  15 and 16 are only examples of calibration parameter determination processing performed by the line-of-sight detection device 1 of the present embodiment. The calibration parameter determination process performed by the line-of-sight detection apparatus 1 according to the present embodiment can be changed as appropriate, and is not limited to the procedure shown in FIGS. .

  In addition, as described above, information on eyeball features including a set of eyeball feature quantities when each of a plurality of gazing points is prepared is not limited to the eyeball feature database, and each can be identified. It may be registered (stored) in the storage unit in another manner.

  The line-of-sight detection apparatus 1 of the present embodiment is applicable to various line-of-sight detection systems that detect the line of sight of a target person from an image, similar to the line-of-sight detection apparatus 1 of the first embodiment.

[Third Embodiment]
FIG. 19 is a diagram illustrating a functional configuration of the visual line detection device according to the third embodiment.

  As illustrated in FIG. 19, the line-of-sight detection apparatus 1 according to the present embodiment includes an image acquisition unit 101, a gaze instruction unit 102, a feature amount calculation unit 103, a process switching unit 104, a feature amount correction unit 105, A feature determination unit 106, a parameter calculation unit 107, and a line-of-sight calculation unit 108 are provided. The line-of-sight detection device 1 includes an eyeball feature database 123 and a line-of-sight storage unit 122. Furthermore, the line-of-sight detection device 1 includes an iris radius calculation unit 109.

  The functions of the image acquisition unit 101, the gaze instruction unit 102, the feature amount calculation unit 103, the process switching unit 104, the parameter determination unit 107, and the line-of-sight calculation unit 108 in the line-of-sight detection device 1 of the present embodiment are the first implementations. This is as described in the form.

  The iris radius calculation unit 109 extracts the iris of the eyeball in the image acquired by the image acquisition unit 101 from the infrared camera 201, and extracts the iris radius.

  The feature amount correction unit 105 corrects the feature amount and iris radius of the eyeball based on the interpupillary distance in the image acquired by the image acquisition unit 101 and the interpupillary distance registered in the eyeball feature database 123.

  Based on the eyeball feature amount and iris radius (or the corrected eyeball feature amount and iris radius) in the acquired image and the eyeball feature database 123, the feature determination unit 106 uses the eyeball feature (eyeball feature) of the target person. A set of quantities).

  FIG. 20 is a diagram illustrating an example of an eyeball feature database according to the third embodiment. FIG. 21 is a diagram for explaining the feature amount and iris radius of the eyeball.

  As shown in FIG. 20, the eyeball feature database 123 according to the present embodiment includes an identifier (ID), an iris radius (IR), and eyeball feature quantities (PG1 to PG5) when gazing at the gazing point. And interpupillary distance (PD). Among them, the identifier, the eyeball feature amount, and the interpupillary distance are the same values as the values in the eyeball feature database 121 described in the first embodiment.

  The iris radius (IR) is a radius in the iris image of each target person extracted from the image when the eyeball feature database 123 is created.

  The eyeball feature database 123 of FIG. 20 can be created, for example, by the database creation device 7 of FIG. When the eyeball feature database 123 is created by the database creation device 7, for example, the feature amount calculation unit 703 calculates the feature amount of the eyeball (relative position of the center of the pupil viewed from the center of corneal reflection) and calculates the iris radius. calculate. Then, in the registration unit 705, one identifier is assigned to the set of the eyeball feature value, the iris radius, and the interpupillary distance calculated by the distance calculation unit 704, and is registered in the eyeball feature database 123 as one record. To do.

  For example, as shown in FIG. 21, the feature amount calculation unit 703 extracts an annular region 603 surrounding the pupil 602 from the image 5 as an iris, and calculates an average distance (number of pixels) from the center Q2 of the pupil 602 to the outer periphery of the annular region 603. ) Is an iris radius IR. The feature amount of the eyeball extracted by the feature amount calculation unit 703 is as described in the first embodiment, and includes the center Q2 of the pupil 602 viewed from the pixel including the center Q1 of the corneal reflection 601 in the image 5. The value represents the relative position of the pixel.

  When the line-of-sight detection apparatus 1 of the present embodiment starts operation, the calibration parameter determination process (step S1) and the line-of-sight detection process (step S2) shown in FIG. 8 are performed. The line-of-sight detection device 1 of the present embodiment performs the process shown in FIG. 22 as the calibration parameter determination process.

  FIG. 22 is a flowchart for explaining the contents of the calibration parameter determination process according to the third embodiment. Of the processing blocks in the flowchart of FIG. 22, blocks that perform the same processing as in the flowchart of FIG. 9 are assigned the same step numbers (eg, S101, S102, etc.) as in FIG.

  In the calibration parameter determination process of the present embodiment, the line-of-sight detection device 1 first acquires an image including the eyeball of the target person that is captured while the target person is gazing at one gaze point (step S101). . The process of step S101 is performed by the image acquisition unit 101 and the gaze instruction unit 102. In step S101, the image acquisition unit 101 and the gaze instruction unit 102 each perform the processing described in the first embodiment. The gaze instruction unit 102 causes the target person to gaze at one gaze point using the output device 4. On the other hand, the image acquisition unit 101 causes the infrared camera 201 to capture an image including the eyeball of the target person who is gazing at one gazing point, and acquires the captured image. Further, the image acquisition unit 101 transmits the acquired image to the feature amount calculation unit 103.

  When the image of the target person is acquired, the line-of-sight detection device 1 then calculates the feature amount of the eyeball by extracting the pupil and corneal reflection from the acquired image in the feature amount calculation unit 103 (step S102). As described in the first embodiment, the feature amount calculation unit 103 extracts a pixel including the center of the pupil and a pixel including the center of corneal reflection in each of the right eye and the left eye in the image, and the center of corneal reflection is extracted. The relative position of the center of the pupil viewed from the above is calculated as the feature amount of the eyeball.

  The feature amount calculation unit 103 transmits the calculated feature amount of the eyeball and the acquired image to the process switching unit 104. When the process performed by the line-of-sight detection device 1 is the calibration parameter determination process (step S <b> 1), the process selection unit 104 transmits the eyeball feature amount and the image to the iris radius calculation unit 110.

  Next, the line-of-sight detection device 1 calculates the iris radius by extracting the iris from the image in the iris radius calculation unit 110 (step S120). For example, the iris radius calculation unit 110 extracts an annular region surrounding the pupil as an iris from each of the regions in the image where the right eye and the left eye are reflected. Subsequently, the iris radius calculation unit 110 calculates the average value of the distance between the center of the pupil and the outer periphery of the extracted annular region, and sets this as the iris radius. The iris radius calculation unit 110 transmits the calculated iris radius and the feature amount of the eyeball calculated by the feature amount calculation unit 103 to the feature amount correction unit 105.

  After step S120, the line-of-sight detection device 1 calculates the interpupillary distance based on the position of the pupil extracted from the image in the feature amount correction unit 105 (step S103). Based on the pixel position including the center of the pupil of the right eye in the image and the position of the pixel including the center of the pupil of the left eye extracted by the feature amount calculation unit 103, the feature amount correcting unit 105 The distance (pixel difference) is calculated.

  Next, the feature amount correcting unit 105 determines whether or not the calculated interpupillary distance matches the interpupillary distance in the eyeball feature database 123 (step S104). When the calculated interpupillary distance does not coincide with the interpupillary distance in the eyeball feature database 123 (step S104; No), the feature amount correction unit 105 corrects the feature amount and iris radius of the eyeball based on the ratio of the interpupillary distance. (Step S121). When the feature amount and iris radius of the eyeball are corrected, the feature amount correction unit 105 transmits the corrected feature amount and iris radius of the eyeball to the feature determination unit 106. On the other hand, when the calculated interpupillary distance matches the interpupillary distance in the eyeball feature database (step S104; Yes), the feature amount correcting unit 105 and the iris calculated in step S120 and the iris calculated in step S120. The radius is transmitted to the feature determination unit 106.

  When the feature amount and iris radius of the eyeball are transmitted to the feature determination unit 106, the line-of-sight detection device 1 determines the eyeball feature used for calibration with reference to the eyeball feature database 123 in the feature determination unit 106 (step S106). . The feature determination unit 106 refers to the feature amount of the eyeball when gazing at the gazing point designated in step S101 among the feature amounts of the eyeball registered in the eyeball feature database 123, and the iris radius (IR). Then, the feature determination unit 106 includes the feature included in the record in which the combination of the eyeball feature amount and the iris radius in the eyeball feature database 123 is the same as or closest to the eyeball feature amount and the iris radius received from the feature amount correction unit 105. A set of quantities is determined for the eye features used for calibration. The feature determination unit 106 transmits the determined eyeball feature (a set of eyeball feature amounts associated with a certain identifier) to the parameter calculation unit 107.

  When the process of step S106 is completed, the line-of-sight detection device 1 calculates a calibration parameter based on the determined eyeball feature in the parameter calculation unit 107 (step S107). The parameter calculation unit 107 calculates a calibration parameter according to a known calculation method in the pupil-corneal reflection method, and determines the calculated calibration parameter as the calibration parameter of the current target person. When the process of step S107 is completed, the parameter calculation unit 107 transmits the determined calibration parameter to the line-of-sight calculation unit 108 and notifies the process switching unit 104 that the calibration parameter for the current target person has been determined. Thereby, the calibration parameter determination process in the line-of-sight detection device 1 ends.

  After the calibration parameters are determined by the calibration parameter determination process of FIG. 22, the line-of-sight detection device 1 of the present embodiment performs, for example, the process shown in FIG. 10 as the line-of-sight detection process.

FIG. 23 is a diagram for explaining the correlation between the iris radius and the eyeball shape.
As main parameters in the eyeball model, for example, as shown in FIG. 23, in addition to the radius of the sphere 64 forming the cornea (corneal radius r), the radius R of the eyeball 65 that controls the rotation of the eye itself, and the cornea The distance D between the center of the sphere 64 and the center of the eyeball 65 is considered.

  When the cornea radius r is known in the eyeball model, there is a correlation between the distance D between the center of the sphere 64 forming the cornea and the center of the eyeball 65 and the iris diameter I (iris radius IR).

  For example, the distance D in the eyeball model shown in FIG. 23A is longer than the distance D in the eyeball model shown in FIG. At this time, the iris diameter I in the eyeball model in FIG. 23A is larger than the iris diameter I in the eyeball model in FIG. In this way, even when the corneal radius r is the same, when the iris diameter I (iris radius IR) is different, the distance between the cornea and the center of the eyeball 65 is different, so the distance between the center of corneal reflection and the center of the pupil is different. Differences occur (see, eg, FIGS. 11 and 12). Therefore, as in this embodiment, in addition to the positional relationship between the center of corneal reflection and the center of the pupil, it is possible to classify eyeball features more finely by using the iris radius, and the eyeball that is optimal for the target person Features can be selected.

FIG. 24 is a diagram illustrating an example of an eyeball feature determination method according to the third embodiment.
FIG. 24 shows a simplified eyeball feature database 123 according to the third embodiment. In addition, the eyeball feature database 123 of FIG. 24 shows the cornea radius (CR) along with the eyeball feature amounts (PG1 to PG5) and the iris radius (IR).

  In the eyeball model, as described above, even when the corneal radius is constant, the iris diameter I (iris radius) changes according to the distance D between the center of the sphere forming the cornea and the center of the eyeball. Then, when eyeball feature amounts are extracted from a plurality of persons having the same corneal radius and different iris radii, the eyeball feature amounts have different values depending on the iris radius.

  In the eyeball feature database 123 shown in FIG. 24, three types of eyeball feature amounts having different iris radii are registered as eyeball feature amounts in a person whose cornea radius is Rm. At this time, the eyeball feature amount of a person with an iris radius of IRm1, the eyeball feature amount of a person with an iris radius of IRm2, and the eyeball feature amount of a person with an iris radius of IRm3 are slightly different. For example, the feature quantity m13 when a person with an iris radius of IRm1 gazes at the gazing point PG3 is slightly different from the feature quantity m23 when a person with an iris radius of IRm2 gazes at the gazing point PG3. The feature amount m23 when a person with an iris radius of IRm2 gazes at the gazing point PG3 is slightly different from the feature amount m33 when a person with an iris radius of IRm3 gazes at the gazing point PG3. Further, the feature amount m13 when a person with an iris radius of IRm1 gazes at the gazing point PG3 is slightly different from the feature amount m33 when a person with an iris radius of IRm3 gazes at the gazing point PG3.

  In addition, as described above, the eyeball feature amounts of a plurality of persons having different corneal radii are different from each other. For example, a feature amount n3 when a person with a corneal radius of 4 gazes at the gazing point PG3, and a feature amount m13, m23, m33 when a person with a corneal radius of Rm (≠ 4) gazes at the gazing point PG3. Is a different value.

  Therefore, when the eyeball feature of the target person is specified based only on the feature amount of the eyeball extracted from a plurality of people with different corneal radii, even if the corneal radius is the same or substantially the same, the feature of the eyeball is different from the difference in the iris radius. There is a possibility that the amount of the eyeball is different and the proper eyeball feature cannot be selected.

  On the other hand, in the eyeball feature database 123 used in this embodiment, a plurality of eyeball features having the same corneal radius and different iris radii are registered. Therefore, in the process of determining the eyeball feature, for example, three types of eyeball features are first extracted as the eyeball feature for the target person whose cornea radius is Rm. At this stage, all of the three types of eyeball features are similar to the eyeball features of the target person, and can be determined as the eyeball features of the target person. However, in the present embodiment, the eyeball feature having the same or the closest iris radius is determined as the eyeball feature of the target person from the three types of eyeball features. As a result, it is possible to determine the eyeball feature of the target person for the eyeball feature whose cornea radius and iris radius are closest to the target person, and it is possible to determine the calibration parameter based on a more appropriate eyeball feature.

  Note that FIG. 22 is only an example of calibration parameter determination processing performed by the line-of-sight detection device 1 of the present embodiment. The calibration parameter determination process performed by the line-of-sight detection device 1 according to the present embodiment includes a process of correcting the feature amount and iris diameter of the eyeball according to the orientation of the face, as described in the second embodiment, for example. There may be.

  In addition, as described above, the eyeball feature information including a set of eyeball feature quantities when each of a plurality of gazing points is prepared is not limited to the database as described above. It may be registered (stored) in the storage unit in a manner.

  The line-of-sight detection apparatus 1 of the present embodiment is applicable to various line-of-sight detection systems that detect the line of sight of a target person from an image, similar to the line-of-sight detection apparatus 1 of the first embodiment.

[Fourth Embodiment]
In the present embodiment, an example will be described in which the eyeball feature is made closer to the feature amount of the eyeball of the target person in the process of determining the eyeball feature of the target person with reference to the eyeball feature databases 121 and 123. Note that the functional configuration of the visual line detection device 1 according to the present embodiment may be the calibration shown in FIG. Further, the eyeball feature database stored in the line-of-sight detection device 1 may be, for example, the eyeball feature database 121 of FIG. 2 or the eyeball feature database 123 of FIG.

  The line-of-sight detection apparatus 1 according to the present embodiment performs the calibration parameter determination process (step S1) and the line-of-sight detection process (step S2) shown in FIG. At that time, the line-of-sight detection device 1 performs the process shown in FIG. 9 or FIG. 15 as the calibration parameter determination process. The line-of-sight detection device 1 performs the process shown in FIG. 25 as the process of step S106 in the calibration parameter determination process, that is, the process of determining the eyeball feature with reference to the eyeball feature database.

  FIG. 25 is a flowchart for explaining the contents of processing for determining eyeball characteristics in the calibration parameter determination processing according to the fourth embodiment.

  The process of determining the eyeball feature (step S106) is performed by the feature determination unit 106 of the eye gaze detection apparatus 1. The feature determination unit 106 first refers to the eyeball feature database 121 (step S161) and determines whether there is a record that matches the calculated feature value or the corrected feature value (step S162). Here, the record is a set of feature quantities of a plurality of eyeballs associated with one identifier.

  When there is a record that matches the calculated feature value or the corrected feature value (step S162; Yes), the feature determination unit 106 determines the matched record as the eyeball feature of the target person (step S163). In this case, the feature determination unit 106 transmits the eyeball feature determined in step S163 to the parameter determination unit 107, and ends the process of determining the eyeball feature.

  On the other hand, when there is no record that matches the calculated feature value or the corrected feature value (step S162; No), the feature determination unit 106 next has a plurality of feature values close to the calculated feature value or the corrected feature value. Are extracted (step S164).

  Next, the feature determination unit 106 calculates the feature value of the eyeball of each gazing point based on the calculated feature value or the corrected feature value and the feature value of the extracted record (step S165). In step S165, the feature determination unit 106 calculates the feature amount of the eyeball of each gazing point based on the ratio of the distance between the calculated feature amount or the corrected feature amount and each of the extracted feature amounts of the plurality of records. To do.

  Next, the feature determination unit 106 determines the eyeball feature of each target point calculated in step S165 as the eyeball feature of the target person (step S166). After step S166, the feature determination unit 106 transmits the eyeball feature determined in step S163 to the parameter calculation unit 107, and ends the process of determining the eyeball feature.

  FIG. 26 is a diagram illustrating an example of a plurality of records extracted from the eyeball feature database. FIG. 26 shows an example of a right eye record registered in the eyeball feature database 121.

  As an example of the above processing performed by the feature determination unit 106, the right eye feature amount (0, -1.9) calculated from the image in which the target person is gazing at the gazing point PG3, and the eyeball feature database 121 of FIG. A case where eyeball characteristics are determined based on In the eyeball feature database 121 of FIG. 26, a record in which the feature amount of the gazing point PG3 is the feature amount (0, -1.9) is not registered. Therefore, the feature determination unit 106 determines that there is no record that matches the calculated feature amount (step S162; No), and then has a plurality of feature amounts that are close to the calculated feature amount (0.1-1.9). Are extracted (step S164). In step S164, the feature determination unit 106 extracts two records, a record having an identifier Rj and a record having an identifier Rk, as records having a feature amount close to the calculated feature amount.

  Next, the feature amount determination unit 106 calculates the feature amount of the eyeball of each gazing point based on the calculated feature amount or the corrected feature amount and the extracted feature amount of the record (step S165).

FIG. 27 is a diagram illustrating a feature amount calculation method.
When calculating the feature amount in step S165, for example, as shown in FIG. 27, the calculated feature amount P (0, -1.9) and the feature of the extracted record are displayed on the virtual plane expressing the feature amount. The quantities J (0, -1.8) and K (0.2, -2.2) are plotted.

  Next, a perpendicular line is drawn from the feature quantity P to the line segment JK connecting the feature quantity J and the feature quantity K of the extracted record, and the line segment JK and the line segment JN at the intersection N of the line segment JK and the perpendicular line are drawn. Divide JK. Then, based on the ratio of the divided line segment JN and line segment NK, the record with the identifier Rj, and the record with the identifier Rk, the eyeball when the target person gazes at each gaze point The feature amount is calculated.

  When the angle formed by the line segment JP and the line segment JN is θ, the following expressions (6-1) and (6-2) hold.

  In Expression (6-1), JN with an arrow at the top means a vector from the feature amount J toward the intersection N. In Expressions (6-1) and (6-2), JP with an arrow at the top means a vector from the feature quantity J to the feature quantity P. In Expression (6-2), JK with an arrow at the top means a vector from the feature quantity J to the feature quantity K.

  Moreover, Formula (6-3) is realized from Formula (6-1) and Formula (6-2).

  Therefore, the coordinates of the intersection N on the virtual plane can be calculated by the following equation (6-4).

  Here, using the feature quantity J = (0, −1.8), the vector JN = (Nx, Ny + 1.8), and the vector JK = (0.2, −0.4), Expression (6-4) Is solved, the coordinates (Nx, Ny) of the intersection N are (0.04, -1.88). Therefore, the ratio | JN |: | NK | of the line segment JN and the line segment NK is 0.09: 0.36 (= 1: 4).

  After obtaining the ratio between the line segment JN and the line segment NK, the feature determination unit 106 calculates, for each gaze point, the feature amount of the record with the identifier Rj and the feature amount of the record with the identifier Rk. Points that divide the connecting line segments by 1: 4 are calculated, and these points are used as feature amounts when the target person watches each gazing point. For example, the feature amount when gazing at the gazing point PG1 is (−4, −3) for the record with the identifier Rj, and (−4.5, −4) for the record with the identifier Rk. is there. Therefore, the feature determination unit 106 calculates the feature amount (X, Y) when the target person gazes at the gazing point PG1 using the following formulas (7-1) and (7-2).

X = -4 + {1 / (1 + 4)}. {-4.5-(-4)}
= -4.1 (7-1)
Y = −3 + {1 / (1 + 4)} · {−4 − (− 3)}
= -3.2 (7-2)

  As described above, in the present embodiment, when a feature quantity that matches the calculated feature quantity or the corrected feature quantity is not registered in the eyeball feature database, the calculated feature quantity or the feature quantity close to the corrected feature quantity is included. Based on the record, the eyeball characteristics of the target person are determined. For this reason, for example, it is possible to suppress an increase in the number of records registered in the eyeball feature database, and to suppress an increase in the processing load of the calibration parameter determination process due to the enlargement of the database. In this embodiment, the eyeball feature of the target person is calculated based on the distance from the calculated feature value or the corrected feature value to the feature value of the extracted record. Therefore, for example, the feature quantity closer to the actual eyeball feature of the target person is calculated compared to the case where the record having the feature quantity closest to the calculated feature quantity or the corrected feature quantity is determined as the eyeball feature of the target person. This improves the accuracy of eye-gaze detection.

  Note that the flowchart of FIG. 25 is merely an example of processing for determining eye features in the calibration parameter determination processing. The process of determining the eyeball feature can be changed as appropriate, and is not limited to the procedure shown in FIG.

  In addition, as described above, the eyeball feature information including a set of eyeball feature quantities when each of a plurality of gazing points is prepared is not limited to the database as described above. It may be registered (stored) in the storage unit in a manner.

  The line-of-sight detection apparatus 1 of the present embodiment is applicable to various line-of-sight detection systems that detect the line of sight of a target person from an image, similar to the line-of-sight detection apparatus 1 of the first embodiment.

  The line-of-sight detection device 1 according to each of the above embodiments can be realized using, for example, a computer and a program executed by the computer. Hereinafter, the line-of-sight detection apparatus 1 realized using a computer and a program will be described with reference to FIG.

FIG. 28 is a diagram illustrating a hardware configuration of a computer.
As shown in FIG. 28, the computer 21 includes a processor 2101, a main storage device 2102, an auxiliary storage device 2103, an input device 2104, an output device 2105, an input / output interface 2106, a communication control device 2107, a medium And a driving device 2108. These elements 2101 to 2108 in the computer 21 are connected to each other by a bus 2110 so that data can be exchanged between the elements.

  The processor 2101 is a central processing unit (CPU), a micro processing unit (MPU), or the like. The processor 2101 controls the overall operation of the computer 21 by executing various programs including an operating system. Further, the processor 2101 performs, for example, the calibration parameter determination process and the line-of-sight detection process shown in FIGS.

  The main storage device 2102 includes a read only memory (ROM) and a random access memory (RAM) not shown. In the ROM of the main storage device 2102, for example, a predetermined basic control program read by the processor 2101 when the computer 21 is started is recorded in advance. The RAM of the main storage device 2102 is used as a working storage area as needed when the processor 2101 executes various programs. The RAM of the main storage device 2102 can be used for storing, for example, an image acquired from the infrared camera 201, an eyeball feature amount calculated based on the image, and a calibration parameter.

  The auxiliary storage device 2103 is, for example, a non-volatile memory such as a flash memory (including a solid state drive (SSD)) or a hard disk drive (HDD). The auxiliary storage device 2103 can be used to store various programs executed by the processor 2101 and various data. The auxiliary storage device 2103 can be used for storing a program including a calibration parameter determination process and a line-of-sight detection process, for example. In addition, the auxiliary storage device 2103 can be used for storing, for example, images acquired from the eyeball feature databases 121 and 123 and the infrared camera 201, calibration parameters, and history of detected lines of sight.

  The input device 2104 is, for example, a keyboard device or a touch panel device. When an operator (user) of the computer 21 performs a predetermined operation on the input device 2104, the input device 2104 transmits input information associated with the operation content to the processor 2101. The input device 2104 can be used as, for example, the input device 3 in FIGS. 1 and 19.

  The output device 2105 is, for example, a display device or a speaker. The output device 2105 can be used, for example, for notification of a gazing point for gazing at a target person. That is, the output device 2105 can be used as the output device 4 in FIGS. 1 and 19.

  The input / output interface 2106 connects the computer 21 and other electronic devices. The input / output interface 2106 includes, for example, a universal serial bus (USB) standard connector. The input / output interface 2106 can be used, for example, for connection between the computer 21 and the infrared camera 201 of the line-of-sight sensor 2.

  The communication control device 2107 is a device that connects the computer 21 to a communication network and controls various types of communication between the computer 21 and other electronic devices via the communication network. The communication control device 2107 can be used, for example, for acquiring the eye feature database created by the database creation device 7 and storing it in the auxiliary storage device 2103 or the like. Further, the communication control device 2107 can be used for communication between the management server 15 and the computer 21 (gaze detection device 1) such as the gaze detection system 17 shown in FIG.

  The medium driving device 2108 reads programs and data recorded in the portable storage medium 22 and writes data stored in the auxiliary storage device 2103 to the portable storage medium 22. As the medium driving device 2108, for example, a memory card reader / writer corresponding to one type or a plurality of types of standards can be used. When a memory card reader / writer is used as the medium driving device 2108, the portable storage medium 22 is a standard compatible with the memory card reader / writer, such as a Secure Digital (SD) standard memory card (flash memory). ) Etc. can be used. Further, as the portable recording medium 22, for example, a flash memory having a USB standard connector can be used. The portable recording medium 22 can be used to store a program including the calibration parameter determination process and the line-of-sight detection process, eyeball feature databases 121 and 123, calibration parameters, and the detected line of sight.

  Further, when the computer 21 is equipped with an optical disk drive that can be used as the medium driving device 2108, various optical disks that can be recognized by the optical disk drive can be used as the portable recording medium 22. Examples of the optical disc that can be used as the portable recording medium 22 include a Compact Disc (CD), a Digital Versatile Disc (DVD), and a Blu-ray Disc (Blu-ray is a registered trademark).

  When the computer 21 inputs a command for starting the processing of FIG. 8, the processor 2101 reads and executes a program including calibration parameter determination processing and line-of-sight detection processing from the auxiliary storage device 2103 and the like. At this time, the processor 2101 functions as an image acquisition unit 101, a gazing point instruction unit 102, a feature amount calculation unit 103, a feature amount correction unit 105, a feature determination unit 106, a parameter calculation unit 107, and a line-of-sight calculation unit 108 (operations) To do). Furthermore, when the computer 21 executes the processing described in the third embodiment, the processor 2101 also functions (operates) as the iris radius calculation unit 110. The RAM of the main storage device 2102, the auxiliary storage device 2103, and the like function as a storage unit that stores the eyeball feature databases 121 and 123 and a line-of-sight storage unit 122 that stores the detected line of sight.

  Furthermore, the computer 21 can be operated not only as the line-of-sight detection device 1 but also as the database creation device 7. When the computer 21 is operated as the database creation device 7, the processor 2101 is caused to execute a program including the creation process of FIG. In this case, the processor 2101 functions (operates) as the gaze instruction unit 701, the image acquisition unit 702, the feature amount calculation unit 703, the distance calculation unit 704, and the registration unit 705 illustrated in FIG. In addition, the RAM of the main storage device 2102, the auxiliary storage device 2103, and the like include a storage unit that stores information indicating a plurality of gazing points, a storage unit that stores images acquired from the infrared camera 201, and a storage that stores an eyeball feature database 121. It functions as a part.

  Note that the computer 21 that operates as the line-of-sight detection device 1 or the database creation device 7 does not need to include all the elements 2101 to 2108 shown in FIG. 28, and some elements may be omitted depending on applications and conditions. Is possible. For example, the computer 21 may be one in which the communication control device 2107 or the medium driving device 2108 is omitted.

The following additional notes are disclosed for each of the embodiments described above.
(Appendix 1)
A storage unit in which a plurality of eyeball features including a set of eyeball feature amounts detected when each of a plurality of gaze points determined in advance from viewpoints in a predetermined positional relationship with the camera is associated with a corneal radius are registered When,
For a person who is gazing at a first gazing point of the plurality of gazing points, a feature amount of the eyeball of the person is calculated based on an image captured by a camera having a predetermined positional relationship with the person. A feature amount calculation unit;
Based on the feature amount of the eyeball corresponding to the first gazing point and the calculated feature amount of the eyeball of the person among the plurality of eyeball features registered in the storage unit, the eyeball feature of the person is obtained. A feature determining unit to determine;
A parameter calculation unit that calculates a calibration parameter for the line of sight of the person based on the feature amounts of the plurality of eyeballs included in the determined eyeball feature;
A line-of-sight detection device comprising:
(Appendix 2)
The feature amount of the eyeball is a value representing a positional relationship between the center of the pupil and the center of corneal reflection in the eyeball of the person shown in the image.
The line-of-sight detection apparatus according to Supplementary Note 1, wherein
(Appendix 3)
Each of the plurality of eyeball features registered in the storage unit further includes information representing a distance from the camera to the viewpoint,
The line-of-sight detection device calculates a distance from the camera to the person based on the image, and calculates the feature amount based on the calculated distance and information indicating the distance registered in the storage unit A feature amount correction unit that corrects the feature amount of the eyeball calculated by the unit;
The line-of-sight detection apparatus according to Supplementary Note 2, wherein:
(Appendix 4)
The information representing the distance from the camera to the viewpoint, and the distance calculated by the feature amount correction unit is a distance of the pupil of the eyes of the person in the image or a distance of the corneal reflection.
The line-of-sight detection apparatus according to Supplementary Note 3, wherein
(Appendix 5)
The feature amount of the eyeball is a value representing the positional relationship between the center of the pupil in the eyeball of the person shown in the image and the center of corneal reflection,
The line-of-sight detection device, based on a difference between a feature amount of the eyeball in the right eye and a feature amount of the eyeball in the left eye, which are captured in the acquired image, of the eyeball calculated by the feature amount calculation unit A feature amount correction unit for correcting the feature amount;
The line-of-sight detection apparatus according to Supplementary Note 1, wherein
(Appendix 6)
The feature amount of the eyeball registered in the storage unit is a feature amount when each of the plurality of gazing points is gazeed in a state where a face is facing the camera,
When the difference between the feature amount of the eyeball in the right eye and the feature amount of the eyeball in the left eye is greater than or equal to a threshold value, the feature amount correction unit and the feature amount of the eyeball in the right eye and the left eye Performing correction to match one of the feature quantities of the eyeball in the eye with the other feature quantity;
The line-of-sight detection device according to appendix 5, wherein
(Appendix 7)
Each of the plurality of eyeball features registered in the storage unit includes an iris radius of the eyeball,
The line-of-sight detection device further includes an iris radius calculation unit that calculates an iris radius of the human eyeball based on the acquired image,
The feature determination unit determines the eyeball feature of the person based on the feature amount of the eyeball corresponding to the first gazing point and the iris radius.
The line-of-sight detection apparatus according to Supplementary Note 1, wherein
(Appendix 8)
The feature determination unit includes:
Of the plurality of eyeball features registered in the storage unit, the eyeball feature amount corresponding to the first gaze point matches the calculated eyeball feature amount of the person, or the calculated person Determining the eyeball feature closest to the eyeball feature amount to the eyeball feature of the person.
The line-of-sight detection apparatus according to Supplementary Note 1, wherein
(Appendix 9)
The feature determination unit includes:
Two eyeball features whose eyeball feature values corresponding to the first gazing point among the plurality of eyeball features registered in the storage unit are close to the calculated human eyeball feature values are extracted,
The distance between the eyeball feature amount in one eyeball feature of the extracted eyeball feature and the calculated eyeball feature amount, and the eyeball feature amount in the other eyeball feature of the extracted eyeball feature and the calculated eyeball feature Based on the distance to the amount, the feature amount of the eyeball when the person gazes each of the plurality of gazing points is estimated,
Determining a set of estimated eyeball features as the human eyeball features;
The line-of-sight detection apparatus according to Supplementary Note 1, wherein
(Appendix 10)
Computer
Based on the image of the person gazing at the first gazing point among a plurality of predetermined gazing points obtained from the camera, the feature amount of the eyeball of the person is calculated,
A plurality of eyeball features including a set of eyeball feature quantities detected when each of the plurality of gazing points from a viewpoint having a predetermined positional relationship with the camera is associated with the corneal radius in the storage device. With reference to the registration information, the eyeball feature of the person is determined based on the feature amount of the eyeball corresponding to the first gazing point among the plurality of eyeball features and the calculated feature amount of the eyeball of the person And
Calculating a calibration parameter for the line of sight of the person based on the feature quantities of the plurality of eyeballs included in the determined eyeball feature;
A line-of-sight detection method characterized by executing processing.
(Appendix 11)
The plurality of eyeball features in the registration information each include an interpupillary distance between a right eye and a left eye,
In the gaze detection method, after calculating the feature amount of the eyeball of the person, based on the interpupillary distance of the person calculated by the computer based on the image and the interpupillary distance of the eyeball feature, Including a process of correcting the calculated characteristic amount of the eyeball of the person,
When the feature amount of the eyeball is corrected, the computer determines the eyeball feature of the person based on the corrected feature amount of the eyeball and the registration information in the process of determining the eyeball feature of the person. ,
The line-of-sight detection method according to supplementary note 10, characterized by:
(Appendix 12)
In the process of determining the eyeball characteristics of the person, the computer
Two eyeball features whose eyeball feature values corresponding to the first gaze point among the plurality of eyeball features in the registration information are close to the calculated human eyeball feature values are extracted,
The distance between the eyeball feature amount in one eyeball feature of the extracted eyeball feature and the calculated eyeball feature amount, and the eyeball feature amount in the other eyeball feature of the extracted eyeball feature and the calculated eyeball feature Based on the distance to the amount, the feature amount of the eyeball when the person gazes each of the plurality of gazing points is estimated,
Determining a set of estimated eyeball features as the human eyeball features;
The line-of-sight detection method according to supplementary note 10, characterized by:
(Appendix 13)
Based on the image of the person gazing at the first gazing point among a plurality of predetermined gazing points obtained from the camera, the feature amount of the eyeball of the person is calculated,
A plurality of eyeball features including a set of eyeball feature quantities detected when each of the plurality of gazing points from a viewpoint having a predetermined positional relationship with the camera is associated with the corneal radius in the storage device. With reference to the registration information, the eyeball feature of the person is determined based on the feature amount of the eyeball corresponding to the first gazing point among the plurality of eyeball features and the calculated feature amount of the eyeball of the person And
Calculating a calibration parameter for the line of sight of the person based on the feature quantities of the plurality of eyeballs included in the determined eyeball feature;
A line-of-sight detection program that causes a computer to execute processing.

DESCRIPTION OF SYMBOLS 1 Gaze detection apparatus 101,702 Image acquisition part 102,701 Gaze instruction | indication part 103,703 Feature-value calculation part 104 Process switching part 105 Feature-value correction | amendment part 106 Feature determination part 107 Parameter calculation part 108 Eye-gaze calculation part 121,123 Eyeball characteristic database 122 Line of sight storage unit 2 Line of sight sensor 201 Infrared camera 202 Infrared LED
3 Input device 4 Output device 5 Image 6 Eyeball 601, 601 R, 601 L Corneal reflection 602, 602 R, 602 L Pupil 7 Database creation device 8, 8 A, 8 B Vehicle 9 Person 12 Display device 13 Information processing device 15 Management server 16 Communication network 21 Computer 2101 Processor 2102 Main storage device 2103 Auxiliary storage device 2104 Input device 2105 Output device 2106 Input / output interface 2107 Communication control device 2108 Medium drive device 22 Portable recording medium

Claims (9)

A storage unit in which a plurality of eyeball features including a set of eyeball feature amounts detected when each of a plurality of gaze points determined in advance from viewpoints in a predetermined positional relationship with the camera is associated with a corneal radius are registered When,
For a person who is gazing at a first gazing point of the plurality of gazing points, a feature amount of the eyeball of the person is calculated based on an image captured by a camera having a predetermined positional relationship with the person. A feature amount calculation unit;
Based on the feature amount of the eyeball corresponding to the first gazing point and the calculated feature amount of the eyeball of the person among the plurality of eyeball features registered in the storage unit, the eyeball feature of the person is obtained. A feature determining unit to determine;
A parameter calculation unit that calculates a calibration parameter for the line of sight of the person based on the feature amounts of the plurality of eyeballs included in the determined eyeball feature;
A line-of-sight detection device comprising: The feature amount of the eyeball is a value representing a positional relationship between the center of the pupil and the center of corneal reflection in the eyeball of the person shown in the image.
The line-of-sight detection apparatus according to claim 1. Each of the plurality of eyeball features registered in the storage unit further includes information representing a distance from the camera to the viewpoint,
The line-of-sight detection device calculates a distance from the camera to the person based on the image, and calculates the feature amount based on the calculated distance and information indicating the distance registered in the storage unit A feature amount correction unit that corrects the feature amount of the eyeball calculated by the unit;
The line-of-sight detection apparatus according to claim 2. The feature amount of the eyeball is a value representing the positional relationship between the center of the pupil in the eyeball of the person shown in the image and the center of corneal reflection,
The line-of-sight detection device, based on a difference between a feature amount of the eyeball in the right eye and a feature amount of the eyeball in the left eye, which are captured in the acquired image, of the eyeball calculated by the feature amount calculation unit A feature amount correction unit for correcting the feature amount;
The line-of-sight detection apparatus according to claim 1. Each of the plurality of eyeball features registered in the storage unit includes an iris radius of the eyeball,
The line-of-sight detection device further includes an iris radius calculation unit that calculates an iris radius of the human eyeball based on the acquired image,
The feature determination unit determines the eyeball feature of the person based on the feature amount of the eyeball corresponding to the first gazing point and the iris radius.
The line-of-sight detection apparatus according to claim 1. The feature determination unit includes:
Of the plurality of eyeball features registered in the storage unit, the eyeball feature amount corresponding to the first gaze point matches the calculated eyeball feature amount of the person, or the calculated person Determining the eyeball feature closest to the eyeball feature amount to the eyeball feature of the person.
The line-of-sight detection apparatus according to claim 1. The feature determination unit includes:
Two eyeball features whose eyeball feature values corresponding to the first gazing point among the plurality of eyeball features registered in the storage unit are close to the calculated human eyeball feature values are extracted,
The distance between the eyeball feature amount in one eyeball feature of the extracted eyeball feature and the calculated eyeball feature amount, and the eyeball feature amount in the other eyeball feature of the extracted eyeball feature and the calculated eyeball feature Based on the distance to the amount, the feature amount of the eyeball when the person gazes each of the plurality of gazing points is estimated,
Determining a set of estimated eyeball features as the human eyeball features;
The line-of-sight detection apparatus according to claim 1. Computer
Based on the image of the person gazing at the first gazing point among a plurality of predetermined gazing points obtained from the camera, the feature amount of the eyeball of the person is calculated,
A plurality of eyeball features including a set of eyeball feature quantities detected when each of the plurality of gazing points from a viewpoint having a predetermined positional relationship with the camera is associated with the corneal radius in the storage device. With reference to the registration information, the eyeball feature of the person is determined based on the feature amount of the eyeball corresponding to the first gazing point among the plurality of eyeball features and the calculated feature amount of the eyeball of the person And
Calculating a calibration parameter for the line of sight of the person based on the feature quantities of the plurality of eyeballs included in the determined eyeball feature;
A line-of-sight detection method characterized by executing processing. Based on the image of the person gazing at the first gazing point among a plurality of predetermined gazing points obtained from the camera, the feature amount of the eyeball of the person is calculated,
A plurality of eyeball features including a set of eyeball feature quantities detected when each of the plurality of gazing points from a viewpoint having a predetermined positional relationship with the camera is associated with the corneal radius in the storage device. With reference to the registration information, the eyeball feature of the person is determined based on the feature amount of the eyeball corresponding to the first gazing point among the plurality of eyeball features and the calculated feature amount of the eyeball of the person And
Calculating a calibration parameter for the line of sight of the person based on the feature quantities of the plurality of eyeballs included in the determined eyeball feature;
A line-of-sight detection program that causes a computer to execute processing.

Images