JP2012055428A - Visual line detector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve detection accuracy of a visual line direction and reliability of a detection result.SOLUTION: When a first visual line direction and a second visual line direction are detected, the visual line direction selection part 30 of a visual line detector 10 selects the first visual line direction in the case that the absolute value of an angle difference between the first visual line direction and the second visual line direction is smaller than a first predetermined value, and selects the second visual line direction in the case that the absolute value of the angle difference is equal to or larger than the first predetermined value. When only the first visual line direction is detected, the visual line direction selection part 30 selects the first visual line direction of a present frame in the case that the angle difference between the first visual line direction (the first visual line direction of the present frame) and the first visual line direction of the frame one before (the first visual line direction of the previous frame) is smaller than a second predetermined value, and selects the visual line direction of the previous frame as the visual line direction of the crew of a vehicle in the case that the angle difference is equal to or larger than the second predetermined value. When only the second visual line direction is detected, the visual line direction selection part 30 selects the second visual line direction as the visual line direction of the crew of the vehicle.

Description

  The present invention relates to a line-of-sight detection device.

Conventionally, for example, when measuring the change in luminance of the retina reflection image when the distance of a predetermined gazing point that the driver gazes at is changed and storing the measurement result and detecting the gazing point of the driver A gaze point measuring apparatus that measures the position of a driver's gaze point using the measured luminance of the retinal reflection image and a measurement result stored in advance is known (for example, see Patent Document 1).
Conventionally, for example, driving using a face orientation direction estimated from a driver's face image, and a model indicating a correspondence relationship between a time-series change in face orientation and a gaze target registered for each of a plurality of drivers. A gaze direction estimation device that estimates the gaze direction of a person is known (see, for example, Patent Document 2).

JP-A-7-35543 JP 2008-146356 A

By the way, according to the gaze point measuring apparatus according to the above-described prior art, it is difficult to measure the gaze point position when the retinal reflection image and the corneal reflection image cannot be appropriately detected by external light, for example. Problems arise.
Further, according to the gaze direction estimation device according to the above-described conventional technology, the gaze direction is only obtained by estimation based on the learning result of the visual behavior for each driver, and it is difficult to improve the gaze direction estimation accuracy. The problem that there is.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a visual line detection device capable of improving the detection accuracy of the visual line direction and the reliability of the detection result.

  In order to solve the above-described problems and achieve the object, the line-of-sight detection device according to the first aspect of the present invention is an image pickup means for picking up a human face and outputting a face image (for example, an occupant in the embodiment). The position of the pupil from the face image output from the camera 12), the light source (for example, the light source 11 in the embodiment) arranged so as to be able to irradiate light in the direction in which the person exists, and the imaging means And position detection means (for example, pupil position detection unit 24, corneal reflection image detection unit 25 in the embodiment) that detects the position of the corneal reflection image by the light emitted from the light source and outputs a detection result; First detection means (for example, the first gaze direction detection unit 27 in the embodiment) that detects the gaze direction of the person based on the detection result output from the position detection means and outputs the detection result; Gaze detection device The position of the pupil of the detection result output from the position detection means and the position of a predetermined part in the face image not including the corneal reflection image of the detection result output from the position detection means (for example, Second detection means (for example, the second gaze direction detection unit 28 in the embodiment) that detects the gaze direction of the person and outputs a detection result based on the position of the eye in the embodiment), and Comparison means for comparing the detection result output from the first detection means and the detection result output from the second detection means and outputting the difference between the detection results as a comparison result (for example, in the embodiment) And when the difference between the comparison results output from the comparison means is less than a predetermined value, the person is based on the line-of-sight direction of the detection results output from the first detection means. Gaze direction And when the difference between the comparison results output from the comparison means is greater than or equal to a predetermined value, the line-of-sight direction of the person is detected based on the line-of-sight direction of the detection results output from the second detection means. Gaze direction detection means (for example, gaze direction selection unit 30 in the embodiment).

  Furthermore, in the line-of-sight detection device according to the second aspect of the present invention, the position detection means determines the position of the pupil for each frame obtained by dividing the face image sequentially output from the imaging means at predetermined time intervals. When detecting the position of the pupil in the frame immediately before a predetermined frame among the frames, the detection is performed in the predetermined frame and the frame detected in the previous frame is detected from the entire image. The position of the pupil is detected from a predetermined range smaller than the entire face image in the vicinity of the position of the pupil.

  Furthermore, in the visual axis detection device according to the third aspect of the present invention, the position detection means includes the position of the pupil and the cornea for each frame obtained by dividing the face image sequentially output from the imaging means at predetermined time intervals. The position of the reflected image is detected and a detection result is output, and when the detection result is output from only one of the first detection means and the second detection means, When the detection result is output from the second detection unit, or when the detection result is output from the first detection unit in an appropriate frame among the frames, the first detection is performed in the appropriate frame. A difference between the line-of-sight direction of the detection result output from the means and the line-of-sight direction detected by the line-of-sight direction detection means in a frame immediately before the appropriate frame. If it is less than value, to detect the gaze direction of the person on the basis of the sight line direction of the detection result.

According to the line-of-sight detection device according to the first aspect of the present invention, the first detection unit that detects the line-of-sight direction based on the position of the pupil and the position of the cornea reflection image with respect to the detection result sequentially output from each detection unit. Then, the detection accuracy of each detection result is likely to be higher than that of the second detection unit that detects the line-of-sight direction without using the cornea reflection image, but the reliability of each detection result and the stability of a plurality of detection results May be lower.
For this reason, when the difference between the detection results of the detection means is less than a predetermined value, it can be determined that the detection result of the first detection means is highly reliable and stable based on the detection result of the second detection means. Can improve the detection accuracy of the visual line direction by detecting the visual line direction based on the detection result of the first detection means.
In addition, when it can be determined that the reliability and stability of the detection result of the first detection means is low because the difference between the detection results of the detection means is equal to or greater than a predetermined value, the detection result of the second detection means By detecting the line-of-sight direction based on the above, desired reliability and stability can be ensured for the detection result of the line-of-sight direction.

  Furthermore, according to the visual axis detection device according to the second aspect of the present invention, when the position of the pupil is detected in the previous frame in an appropriate frame, a search is made within a predetermined range near the position of the pupil. Therefore, the processing load can be reduced as compared with the case where the entire face image is searched.

Furthermore, according to the visual line detection device according to the third aspect of the present invention, when the detection result is output only from the second detection means, the visual line direction is detected by detecting the visual line direction based on the detection result. Desired reliability and stability can be ensured for the result.
Even if the detection result is output only from the first detection means, if the difference between the detection result and the gaze direction detected by the gaze direction detection means in the previous frame is less than a predetermined value, Thus, it can be determined that the detection result of the first detection means has the desired reliability and stability, and the line-of-sight direction can be detected while ensuring the desired reliability and stability based on the detection result.

It is a lineblock diagram of a look detection device concerning an embodiment of the invention. It is a figure which shows the example of the predetermined range in the vicinity of the center position of the pupil or iris detected in the front frame by the pupil position detection part of the gaze detection apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the gaze detection apparatus which concerns on embodiment of this invention.

Hereinafter, an embodiment of a visual line detection device of the present invention will be described with reference to the accompanying drawings.
The line-of-sight detection device 10 according to the present embodiment includes, for example, a light source 11, an occupant camera 12, and a processing device 13, as shown in FIG.

The light source 11 irradiates a subject to be imaged (for example, a driver's face seated in the driver's seat) with light such as visible light and infrared light.
The occupant camera 12 includes at least an occupant's face seated on the seat in the imaging region as an imaging target, for example, a face that can be imaged in the visible light region and the infrared region and includes the occupant's face obtained by sequentially capturing images. Output an image.

  The processing device 13 includes, for example, an irradiation control unit 21, an occupant imaging control unit 22, a face image acquisition unit 23, a pupil position detection unit 24, a cornea reflection image detection unit 25, an eye-head detection unit 26, and a first A line-of-sight direction detection unit 27, a second line-of-sight direction detection unit 28, a comparison unit 29, a line-of-sight direction selection unit 30, and a line-of-sight direction storage unit 31 are configured.

The occupant imaging control unit 22 controls imaging by the occupant camera 12.
The irradiation control unit 21 controls light irradiation by the light source 11.
The face image acquisition unit 23 acquires face images sequentially output from the occupant camera 12.

  The pupil position detection unit 24 performs feature amount calculation and shape determination using the left and right eyeballs of the occupant as detection objects for each of a plurality of frames obtained by dividing the face images sequentially acquired by the face image acquisition unit 23 at predetermined time intervals. Recognition processing is performed, and based on the processing result, the center position of the pupil of the eye or the iris is detected.

Note that the pupil position detection unit 24 detects the pupil or iris center position in the previous frame when the pupil or iris center position is detected in the previous frame (previous frame) with respect to the frame to be processed. The center position of the pupil or iris is detected only within a predetermined range in the vicinity of the center position, that is, within a predetermined range smaller than the entire face image (for example, the left range L and the right range R shown in FIG. 2).
On the other hand, if the center position of the pupil or iris is not detected in the previous frame (previous frame), the center position of the pupil or iris is detected with the entire face image being processed.

  The cornea reflection image detection unit 25 is a corneal surface of light emitted from the light source 11 around the left and right eyes of the occupant for each of a plurality of frames obtained by dividing the face images sequentially acquired by the face image acquisition unit 23 at predetermined time intervals. The reflection point at (for example, the center position of the Purkinje image, which is an infrared reflection image) is detected.

  The eye detection unit 26 recognizes feature amount calculation and shape determination using the left and right eyeballs of the occupant as detection objects for each of a plurality of frames obtained by dividing the face images sequentially acquired by the face image acquisition unit 23 at predetermined time intervals. Processing is performed, and the position of the eye's eye is detected based on the processing result.

  The first line-of-sight direction detection unit 27 includes, for each of a plurality of frames, the eye pupil or iris center position detected by the pupil position detection unit 24, and the corneal surface reflection point detected by the corneal reflection image detection unit 25. The line-of-sight direction (first line-of-sight direction) is detected on the basis of the relative distance.

  The second line-of-sight direction detection unit 28 sets the relative distance between the center position of the eye pupil or iris detected by the pupil position detection unit 24 and the eye position detected by the eye detection unit 26 for each of a plurality of frames. Based on this, the gaze direction (second gaze direction) is detected.

For each frame, the comparison unit 29 has a gaze direction (first gaze direction) detected by the first gaze direction detection unit 27 and a gaze direction (second gaze direction) detected by the second gaze direction detection unit 28. To determine whether or not the absolute value of the difference between the first line-of-sight direction and the second line-of-sight direction is less than a first predetermined value.
Note that the difference between the first line-of-sight direction and the second line-of-sight direction is the difference in angle between the respective line-of-sight directions with reference to a predetermined direction (for example, the front direction of an occupant seated on the seat).

  The line-of-sight direction selection unit 30 calculates the absolute value of the difference between the comparison results by the comparison unit 29 when each line-of-sight direction is detected by the first line-of-sight direction detection unit 27 and the second line-of-sight direction detection unit 28 for each frame. When it is less than the first predetermined value, the first line-of-sight direction is selected as the line-of-sight direction of the vehicle occupant, and when the absolute value of the comparison result by the comparison unit 29 is greater than or equal to the first predetermined value, the second line of sight is selected. The direction is selected as the line-of-sight direction of the vehicle occupant.

Further, the line-of-sight direction selection unit 30 detects the first line-of-sight direction (the first line-of-sight direction) when the line-of-sight direction (first line-of-sight direction) is detected only by the first line-of-sight direction detection unit 27 for each frame. The difference between the gaze direction (gaze direction of the previous frame) selected by the gaze direction selection unit 30 and stored in the gaze direction storage unit 31 in the previous frame (previous frame) is a second predetermined value. If it is less, the first viewing direction of the current frame is selected as the viewing direction of the vehicle occupant.
On the other hand, when the difference between the first line-of-sight direction of the current frame and the line-of-sight direction of the previous frame is greater than or equal to a second predetermined value, the line-of-sight direction of the previous frame is selected as the line-of-sight direction of the vehicle occupant.

The difference between the first line-of-sight direction of the current frame and the line-of-sight direction of the front frame is a difference in angle between the respective line-of-sight directions with reference to a predetermined direction (for example, the front direction of an occupant seated on the seat). The 2 predetermined value is a value larger than the first predetermined value.
Further, when the line-of-sight direction (second line-of-sight direction) is detected only by the second line-of-sight direction detection unit 28 for each frame, the line-of-sight direction selection unit 30 uses the second line-of-sight direction as the line of sight of the vehicle occupant. Select as direction.

  The gaze direction storage unit 31 stores the gaze direction for each frame selected by the gaze direction selection unit 30.

  The line-of-sight detection apparatus 10 according to this embodiment has the above-described configuration. Next, the operation of the line-of-sight detection apparatus 10 will be described.

First, for example, in step S01 shown in FIG. 3, the first line-of-sight direction detection unit 27 or the second line-of-sight direction detection unit 28 in the frame (previous frame) immediately before the frame that is the current processing target (current frame). It is determined whether the line-of-sight direction is detected.
If this determination is “YES”, the flow proceeds to step S 02, and this step S 02
In this case, the pupil is limited to a predetermined range near the center position of the pupil or iris detected in the previous frame, that is, a predetermined range smaller than the entire face image (for example, the left range L and the right range R shown in FIG. 2). Or it sets to the search range of the center position of an iris.
On the other hand, if this determination is “NO”, the flow proceeds to step S 03, where the entire face image is set as the search range for the center position of the pupil or iris.

  Next, in step S04, the line-of-sight direction is based on the relative distance between the center position of the eye pupil or iris detected by the pupil position detection unit 24 and the reflection point of the corneal surface detected by the corneal reflection image detection unit 25. A first gaze detection process for detecting (first gaze direction) is executed.

  Next, in step S05, the line-of-sight direction (second line of sight) is based on the relative distance between the center position of the pupil or iris of the eye detected by the pupil position detector 24 and the position of the eye head detected by the eye detector 26. A second line-of-sight detection process for detecting (direction) is executed.

Next, in step S06, it is determined whether or not each gaze direction is undetected in both the first gaze detection and the second gaze detection.
If this determination is “YES”, the flow proceeds to step S 07, and in this step S 07, a signal indicating that the sight line direction of the vehicle occupant is not detected is output, and the flow proceeds to return.
On the other hand, if this determination is “NO”, the flow proceeds to step S08.

In step S08, it is determined whether or not each gaze direction is detected by both the first gaze detection and the second gaze detection.
If this determination is “NO”, the flow proceeds to step S 12 described later.
On the other hand, if this determination is “YES”, the flow proceeds to step S 09.
In step S09, it is determined whether or not the absolute value of the difference between the angle in the first line-of-sight direction and the angle in the second line-of-sight direction is less than a first predetermined value.
If this determination is “YES”, the flow proceeds to step S 10, in which the first line-of-sight direction is adopted as the line-of-sight direction of the vehicle occupant and the flow proceeds to return.
On the other hand, if this determination is “NO”, the flow proceeds to step S 11, in which the second line-of-sight direction is adopted as the line-of-sight direction of the vehicle occupant and the flow proceeds to return.

In step S12, it is determined whether or not the first gaze direction is detected by the first gaze detection.
If this determination is “NO”, the flow returns to step S 11 described above.
On the other hand, if the determination is “YES”, the flow proceeds to step S13.
In step S13, it is determined whether or not the absolute value of the angle difference between the current frame and the previous frame in the first line-of-sight direction is less than a second predetermined value.
If this determination is “YES”, the flow proceeds to step S 14, and in this step S 14, the first gaze direction of the current frame is adopted as the gaze direction of the vehicle occupant, and the flow proceeds to return.
On the other hand, if this determination is “NO”, the flow proceeds to step S 15, in which the line-of-sight direction of the previous frame is adopted as the line-of-sight direction of the vehicle occupant, and the flow proceeds to return.

As described above, according to the line-of-sight detection device 10 according to the present embodiment, the first line-of-sight direction detection unit 27 that detects the first line-of-sight direction based on the position of the pupil and the position of the cornea reflection image generates the cornea reflection image. Regardless of the second gaze direction detection unit 28 that detects the second gaze direction, there is a high possibility that the gaze direction detection accuracy is high, but the reliability and stability of the detection direction may be low.
For this reason, when the absolute value of the difference between the first line-of-sight direction and the second line-of-sight direction is less than the first predetermined value, it is determined that the reliability and stability of the first line-of-sight direction are high depending on the second line-of-sight direction. If possible, the detection accuracy of the line-of-sight direction can be improved by detecting the line-of-sight direction of the occupant based on the first line-of-sight direction.
In addition, when the absolute value of the difference between the first line-of-sight direction and the second line-of-sight direction is equal to or greater than the first predetermined value, it can be determined that the reliability and stability of the first line-of-sight direction are low. By detecting the sight line direction of the occupant based on the two sight line directions, it is possible to ensure desired reliability and stability with respect to the detection result of the sight line direction.

  Further, when the pupil position detection unit 24 detects the position of the pupil in the previous frame (previous frame) with respect to the frame to be processed, the pupil position detection unit 24 searches within a predetermined range in the vicinity of the pupil position. Therefore, the processing load can be reduced as compared with the case where the entire face image is searched.

Furthermore, when only the second line-of-sight direction is detected, the second line-of-sight direction is used as the occupant's line-of-sight direction, thereby ensuring desired reliability and stability for the detection result of the line-of-sight direction. Can do.
Even when only the first line-of-sight direction is detected, the detection result (first line-of-sight direction in the current frame) and the line-of-sight direction in the previous frame (line-of-sight direction in the previous frame) If the difference is less than the second predetermined value, it can be determined that the first line-of-sight direction in the current frame has the desired reliability and stability, and the first line-of-sight direction in the current frame is determined as the line-of-sight direction of the occupant. As a result, it is possible to ensure the desired reliability and stability and detect the line-of-sight direction.
On the other hand, if the difference between the detection result (first line-of-sight direction in the current frame) and the line-of-sight direction in the previous frame (line-of-sight direction in the previous frame) is greater than or equal to the second predetermined value, By adopting the line-of-sight direction as the line-of-sight direction of the occupant, the line-of-sight direction can be detected while ensuring the desired reliability and stability.

  Note that the second gaze direction detection unit 28 is not limited to the position of the head of the eye, for example, another position of the eye such as the position of the corner of the eye, and the center position of the pupil or iris of the eye detected by the pupil position detection unit 24. The line-of-sight direction (second line-of-sight direction) may be detected on the basis of the relative distance.

  In step S15 described above, the line-of-sight direction of the front frame is adopted as the line-of-sight direction of the vehicle occupant. However, the present invention is not limited to this, and a signal indicating that the line-of-sight direction of the vehicle occupant is not detected is output. May be.

DESCRIPTION OF SYMBOLS 10 Eye-gaze detection apparatus 11 Light source 12 Crew camera (imaging means)
24 Pupil position detection unit (position detection means)
25 Cornea reflection image detection unit (position detection means)
27 1st gaze direction detection part (1st detection means)
28 Second visual line direction detection unit (second detection means)
29 Comparison part (comparison means)
30 Gaze direction selection unit (Gaze direction detection means)

Claims (3)

Imaging means for imaging a human face and outputting a face image, a light source arranged so as to be able to irradiate light in a direction in which the person exists, and a position of a pupil from the face image output from the imaging means And a position detection means for detecting a position of a corneal reflection image by the light emitted from the light source and outputting a detection result, and detecting the gaze direction of the person based on the detection result output from the position detection means A line-of-sight detection device comprising first detection means for outputting a detection result.
Based on the position of the pupil of the detection result output from the position detection means and the position of a predetermined part in the face image not including the corneal reflection image of the detection result output from the position detection means. Second detection means for detecting the direction of the line of sight of the person and outputting a detection result;
A comparison unit that compares the detection result output from the first detection unit with the detection result output from the second detection unit and outputs a difference between the detection results as a comparison result;
When the difference between the comparison results output from the comparison unit is less than a predetermined value, the human gaze direction is detected based on the gaze direction of the detection results output from the first detection unit, Gaze direction detection that detects the gaze direction of the person based on the gaze direction of the detection result output from the second detection means when the difference of the comparison results output from the comparison means is greater than or equal to a predetermined value. A line-of-sight detection device. The position detection means detects the position of the pupil from the entire face image for each frame obtained by dividing the face image sequentially output from the imaging means at predetermined time intervals, and a predetermined frame among the frames. When the position of the pupil is detected in the frame immediately before, the predetermined range is smaller than the entire face image in the vicinity of the pupil position detected in the previous frame in the predetermined frame. The line-of-sight detection device according to claim 1, wherein the position of the pupil is detected from within. The position detection means detects the position of the pupil and the position of the cornea reflection image for each frame obtained by dividing the face image sequentially output from the imaging means at predetermined time intervals, and outputs a detection result. ,
When the detection result is output from only one of the first detection unit and the second detection unit,
When the detection result is output from the second detection unit, or when the detection result is output from the first detection unit in an appropriate frame among the frames, the first detection is performed in the appropriate frame. The detection is performed when a difference between the line-of-sight direction of the detection result output from the means and the line-of-sight direction detected by the line-of-sight direction detection means in a frame immediately before the appropriate frame is less than a predetermined value. The gaze detection apparatus according to claim 1, wherein the gaze direction of the person is detected based on the gaze direction of the result.

Images