WO2017203769A1

WO2017203769A1 - Sight line detection method

Info

Publication number: WO2017203769A1
Application number: PCT/JP2017/007189
Authority: WO
Inventors: 山下　龍麿; 正行中西
Original assignee: アルプス電気株式会社
Priority date: 2016-05-23
Filing date: 2017-02-24
Publication date: 2017-11-30
Also published as: JP6767482B2; JPWO2017203769A1

Abstract

[Problem] To provide a sight line detection method which, while ensuring precision in sight line direction detection, is capable of reducing computational processing load and achieving greater speed. [Solution] Provided is a sight line detection method, comprising a first assessment step of periodically assessing whether eye region images of a subject are included in an image of a prescribed range which is acquired in order to extract the eye region images. If the eye region images of the subject are included in the image of the prescribed range in the first assessment step, the eye region images are extracted and the sight line direction of the subject is detected on the basis of the extracted eye region images. If the eye region images of the subject are not included in the image of the prescribed range in the first assessment step, a new overall image is acquired, a face image of the subject is detected from the overall image, eye region images of the subject are extracted from the detected face image, the sight line direction of the subject is detected on the basis of the extracted eye region images, and furthermore, a range which includes the extracted eye region images is updated as the prescribed range.

Description

Gaze detection method

The present invention relates to a gaze detection method for detecting a gaze direction of a subject.

In the line-of-sight detection device described in Patent Document 1, first, the center position of the face, the center position of the parts constituting the face, the organ position such as the pupil position, and the like are detected from the acquired image data, and the detected center Using the position and the organ position, normalization is performed so that the size of the face is a predetermined size and the orientation of the face is upright. After that, using the normalized image data, the feature amount corresponding to the face direction and the feature amount of the eye region are extracted, and the gaze direction is estimated using these feature amounts.

JP 2012-037934 A

However, in the gaze detection device described in Patent Document 1, every time the gaze direction estimation is updated, the calculation process from the normalization process to the extraction of the feature quantity corresponding to the face direction and the feature quantity of the eye area is performed. Therefore, the amount of processing increases each time, and it is difficult to speed up the line-of-sight detection process. Furthermore, in recent years, it has been demanded to improve the accuracy of eye direction estimation by increasing the accuracy of eye area feature amount extraction. Become.

Therefore, an object of the present invention is to provide a gaze detection method capable of increasing the speed by suppressing the burden of calculation processing while ensuring the accuracy of gaze direction detection.

In order to solve the above-described problem, the eye gaze detection method of the present invention determines whether or not an eye region image of a subject is included in a predetermined range of images acquired for extracting an eye region image at a constant cycle. A first discriminating step for discriminating; when the eye region image of the subject is included in the image of the predetermined range in the first discriminating step, the eye region image is extracted, and the extracted eye region image is Based on this, the direction of the subject's line of sight is detected, and when the eye area image of the subject is not included in the image in the predetermined range in the first determination step, the whole image is newly acquired, and the subject A face image of the subject, the eye area image of the subject person is extracted from the detected face image, the line-of-sight direction of the subject person is detected based on the extracted eye area image, and the range including the extracted eye area image Is updated as a predetermined range. It is set to.
As a result, as long as the eye area image is included in the image in the predetermined range, that is, unless the eye area image is lost from the predetermined range, the entire image is not acquired and the eye area image extracted from the image in the predetermined range is used. Since the gaze direction is calculated, the calculation processing load can be suppressed while maintaining the accuracy of the gaze direction calculation, and the processing speed can be increased.

In the line-of-sight detection method of the present invention, independent of the first determination step, a second determination step of acquiring an entire image and determining whether or not the face image of the subject can be detected from the acquired entire image. The entire image acquired in the second determination step has a lower resolution than the image in the predetermined range determined in the first determination step, and the subject's face image from the entire image acquired in the second determination step Cannot be detected, a new whole image is acquired without waiting for the next first determination step, the face image of the subject is detected from this image, and the eye area image of the subject is detected from the detected face image. It is preferable to extract and detect the gaze direction of the subject based on the extracted eye region image, and further update the range including the extracted eye region image as a predetermined range.
Thereby, since it can discriminate | determine with an image with small data amount, the burden of arithmetic processing can be reduced, ensuring the accuracy of a gaze detection.

In the line-of-sight detection method of the present invention, image acquisition is performed by an image sensor in which a plurality of pixels are arranged in a horizontal direction and a vertical direction and driven by a rolling shutter system, and the predetermined range is aligned in the horizontal direction of the image sensor. It is preferable to be composed of one or two or more lines.
As a result, the cost of the image sensor can be reduced, and the burden of calculation processing can be reduced, and high-speed and high-precision gaze direction detection can be realized.

According to the line-of-sight detection method of the present invention, it is possible to reduce the processing load and ensure high-speed processing while ensuring the accuracy of line-of-sight detection.

It is a functional block diagram which shows the structure of the gaze detection apparatus which concerns on 1st Embodiment of this invention. It is a functional block diagram which shows the structure of the image acquisition part of 1st Embodiment of this invention. It is a functional block diagram which shows the structure of the gaze detection part of 1st Embodiment of this invention. It is a figure which shows the example of a subject's image. (A) is a figure which shows typically the image acquisition timing from an image pick-up element, FIG.5 (B) is a figure which shows typically the light emission period of a 1st light source and a 2nd light source. It is a flowchart which shows the flow of a gaze detection based on 1st Embodiment of this invention. It is a flowchart which shows the flow of a gaze detection based on 2nd Embodiment of this invention.

Hereinafter, a gaze detection method according to an embodiment of the present invention will be described in detail with reference to the drawings.
<First Embodiment>
<Configuration of eye gaze detection device>
With reference to FIG. 1 to FIG. 3, a visual line detection device used in the visual line detection method according to the first embodiment will be described. Here, FIG. 1 is a functional block diagram showing the configuration of the line-of-sight detection device 10 according to the first embodiment, FIG. 2 is a functional block diagram showing the configuration of the image acquisition unit 20 of the first embodiment, and FIG. It is a functional block diagram which shows the structure of the gaze detection part 60 of 1st Embodiment. FIG. 4 is a diagram illustrating an example of an image of a subject.

As shown in FIG. 1, the line-of-sight detection device 10 according to the first embodiment includes a control unit 11, a memory 12, an image acquisition unit 20, a face detection unit 30, a normalization processing unit 40, and an eye region image. An acquisition unit 50 and a line-of-sight detection unit 60 are provided. The line-of-sight detection device 10 is installed, for example, on an instrument panel in an automobile interior or an upper part of a windshield so as to face the driver's face as a subject.

In the line-of-sight detection device 10, the face detection unit 30 extracts the face image A2 (FIG. 4) from the entire image A1 (FIG. 4) of the subject SB acquired by the image acquisition unit 20, for example, an image in a range corresponding to the upper body. The normalization processing unit 40 performs normalization processing on the face image A2. For the normalized face image, a predetermined range A3 (FIG. 4) including the eye region is set in the eye region image acquisition unit 50, and an eye region image within this predetermined range is extracted and output to the line-of-sight detection unit 60. Is done. The line-of-sight detection unit 60 extracts a feature amount based on the received image, and detects the gaze direction of the subject based on the feature amount. Processing from image acquisition by the image acquisition unit 20 to detection of the line-of-sight direction by the line-of-sight detection unit 60 is executed according to control by the control unit 11, information necessary for the processing, processing results, and the like are stored in the memory 12 and necessary. Reads accordingly.

The predetermined range set by the eye area image acquisition unit 50 is stored in the memory 12, and after the gaze direction is detected by the line-of-sight detection unit 60, the next image is acquired within this predetermined range, and this image includes the eye area image. It is determined by the control unit 11 as a determination unit. If an eye area image is included in the acquired image, the line-of-sight direction is detected in the same manner as the above-described processing. As a result of the determination by the control unit 11, when the eye area image is not included in the image in the predetermined range, the entire image is acquired by the image acquisition unit 20, and the detection of the face image and the normalization processing are performed based on this image. Later, a predetermined range is newly set, and the data in the predetermined range stored in the memory 12 is updated with this range. Further, when an eye area image is included in the image acquired in the predetermined range, the line-of-sight direction is detected based on the feature amount extracted from the eye area image. Hereinafter, each constituent member / block will be described.

As shown in FIG. 2, the image acquisition unit 20 includes a first light source 21, a second light source 22, a first camera 23, a second camera 24, an exposure control unit 25, and a light source control unit 26. .

The first light source 21 includes a plurality of LED (light emitting diode) light sources. These LED light sources are arranged outside the lens of the first camera 23 so as to surround the lens.
The second light source 22 is also composed of a plurality of LED light sources. These LED light sources are arranged outside the lens of the second camera 24 so as to surround the lens.

The LED light source of the first light source 21 and the LED light source of the second light source 22 emit infrared light (near infrared light) of 800 nm or more and 1000 nm or less, and this detection light can be given to the driver's eyes. Are arranged as follows. In particular, 850 nm is a wavelength with a low light absorption rate in the eyeball of a human eye, and this light is easily reflected by the retina at the back of the eyeball.

The

cameras

23 and 24 have, for example, CMOS (complementary metal oxide semiconductor) as an image sensor. This image sensor acquires an image of a face including the driver's eyes, and light is detected by a plurality of pixels arranged in the horizontal direction and the vertical direction.
In these

cameras

23 and 24, it is preferable to arrange band pass filters in accordance with the wavelengths of the detection lights emitted from the two

light sources

21 and 22. Thereby, the extraction of the pupil image in the bright pupil image detection unit 61 and the dark pupil image detection unit 62 and the calculation of the gaze direction in the gaze direction calculation unit 65 can be performed with high accuracy.

The

cameras

23 and 24 can switch the shooting range and resolution according to the control of the control unit 11.
The shooting range can be switched between, for example, an entire image and a partial image. For example, when the driver of the vehicle is the target, the entire image is an image of the upper body of the driver who has arrived at the driver's seat as a position that is the target of gaze detection. The partial image is an image in a predetermined range set by the eye region image acquisition unit 50 based on the entire image, that is, an image in a range corresponding to the driver's eye region.

The shooting resolution can be switched between high resolution and low resolution, for example. A high-resolution image is an image having a resolution capable of extracting at least a feature amount necessary for detecting the gaze direction, and a low-resolution image can detect at least a feature portion of the face. It is an image having a resolution that can be detected.

The distance between the optical axes of the LED light sources of the first camera 23 and the first light source 21 is determined based on the optical axes of the first camera 23 and the second camera 24 in consideration of the distance between the line-of-sight detection unit 60 and the driver as the driver. The distance is sufficiently short. Therefore, the first light source 21 can be regarded as having substantially the same optical axis as the first camera 23. Similarly, the distance between the optical axes of the LED light sources of the second camera 24 and the second light source 22 is sufficiently shorter than the distance between the optical axes of the first camera 23 and the second camera 24. It can be considered that the optical axes 22 of the second camera 24 are substantially coaxial with each other.

On the other hand, since the distance between the optical axes of the first camera 23 and the second camera 24 is sufficiently long, the optical axes of the first light source 21 and the first camera 23, the second light source 22 and the second light source 22. The optical axes of the camera 24 are not coaxial. In the following description, the above arrangement may be expressed as two members being substantially coaxial and the like, and the two members being non-coaxial.

The timing of lighting (light emission) of the first light source 21 and the second light source 22 is controlled by the light source control unit 26. The timing of this lighting is set by an instruction signal from the exposure control unit 25, and the exposure control unit 25 performs shooting described later so as to be synchronized with the lighting of the first light source 21 and the second light source 22 according to the control of the control unit 11. The first camera 23 and the second camera 24 are caused to perform imaging under conditions (bright pupil imaging conditions, dark pupil imaging conditions).

The face detection unit 30 performs downsizing on the entire image A1 (FIG. 4) acquired by the image acquisition unit 20 to reduce the number of pixels by binning processing or the like as preprocessing. This downsizing is to reduce the resolution and reduce the size of the image data by combining a predetermined number of adjacent pixels in the entire image A1 into one pixel. This downsizing process is set to a level at which a later face detection process can be performed, and the number of pixels to be combined into one pixel is determined in accordance with this level. As a result, the data size of the image is reduced, and the speed can be increased while ensuring the accuracy of the subsequent face detection processing.

Furthermore, the face detection unit 30 performs face detection by applying various detection methods to the image after the downsizing process. For example, initial detection is performed based on the Haar-like face detection method, and information on general facial feature parts registered in the memory 12 in advance, for example, eyebrow, eyeball, iris, nose, lip position, shape, The face is detected according to the collation result in comparison with the size data. In addition, as the three-dimensional face data, the face orientation is also detected by comparing the acquired image with information on each characteristic part for a plurality of face orientations, for example, front, diagonal right direction, and diagonal left direction.

The face detection unit 30 also uses a plurality of landmarks corresponding to each feature part, such as eyebrows, eyeballs, irises, lip contours, and nasal ridge lines, based on the color and brightness of the detected face image. Is detected.
In addition to or instead of general facial feature information, a combination of facial feature information of a specific individual and a name or other identification information that identifies the individual is registered in advance. The individual may be authenticated together with the face detection by collating with the image acquired by the image acquisition unit 20.

The normalization processing unit 40 maintains the relationship between the plurality of landmarks detected by the face detection unit 30, and converts the face to face forward and has a predetermined size by, for example, affine transformation, This normalizes the face image.

The eye region image acquisition unit 50 predetermines a range in which an image including both eyes is included in the image normalized by the normalization processing unit 40 based on the position / range information of the eyeball detected as a landmark. Set as a range. Furthermore, the eye area image acquisition unit 50 acquires, as the eye area image, a bright pupil image and a dark pupil image corresponding to a predetermined range among the images acquired by the image acquisition unit 20. The set predetermined range is stored in the memory 12, and the acquired eye region image is output to the line-of-sight detection unit 60.

Here, an example of setting the predetermined range will be described. FIG. 5 is a diagram schematically illustrating an example of image acquisition timing from a rolling shutter type imaging device and light emission timing of a light source. 5A shows image acquisition timing from the image sensor, and FIG. 5B shows light emission periods of the first light source 21 and the second light source 22.
The image sensor drive system includes a global shutter system and a rolling shutter system, and the

cameras

23 and 24 of the first embodiment can use any image sensor, but here the case of the rolling shutter system will be described. To do.

In FIG. 5A, H000, H100, H200, H300, H400, H500, H600, H700, and H800 are lines of pixels arranged in the horizontal direction in order from the top to the bottom in the vertical direction in the image sensor. Respectively. The image sensor is driven for each of these lines by a rolling shutter system. Further, “VSYNC” in FIG. 5A is a vertical synchronization signal output from the

cameras

23 and 24 and is determined by the frame rate of the camera, and the control unit 11 is synchronized with these vertical synchronization signals, Capture image data corresponding to a pixel line of an image sensor of a corresponding camera. B11 to B18, B21 to... Indicate the timing of capturing image data corresponding to each pixel line of the image sensor, which means horizontal synchronization signals.

FIG. 5B shows the detection light emission periods I11, I12, and I13 from the first light source 21, and the detection light emission periods I21 and I22 from the second light source 22, respectively. In the example shown in FIG. 5B, the light emission times of the

light sources

21 and 22 are the same, and light is emitted alternately at a constant cycle. In each period in which the detection light from the first light source 21 or the second light source 22 is emitted, the image sensor is driven frame by frame from the line H000 to the line H800. The image obtained by driving for one frame corresponds to the entire image of the subject, and one or a plurality of pixel lines can be set as a predetermined range corresponding to the eye region in the image.

The line-of-sight detection unit 60 is composed of a CPU and a memory of a computer, and the processing by each block shown in FIG. 3 is performed by executing software installed in advance. The gaze detection unit 60 includes a bright pupil image detection unit 61, a dark pupil image detection unit 62, a pupil center calculation unit 63, a corneal reflection light center detection unit 64, and a gaze direction calculation unit 65. .

The image given to the line-of-sight detection unit 60 is read into the bright pupil image detection unit 61 and the dark pupil image acquisition unit 62, respectively. The bright pupil image detection unit 61 detects an eye image when the light source and the camera are combined, which satisfies any of the following bright pupil imaging conditions (a). The dark pupil image detection unit 62 detects the following dark pupils: An eye image when the combination of the light source and the camera satisfies any one of the imaging conditions (b) is detected.
(A) Bright pupil photographing condition (a-1) An image is acquired by the first camera 23 substantially coaxial with the first light source 21 during the lighting period. (A-2) During the lighting period of the second light source 22 An image is acquired by the substantially coaxial second camera 24 (b) Dark pupil photographing condition (b-1) An image is acquired by the non-coaxial second camera 24 during the lighting period of the first light source 21 (b-2) During the lighting period of the second light source 22, an image is acquired by the first camera 23 that is non-coaxial with the first light source 22.

<Light pupil image and dark pupil image>
Since the wavelength 850 nm of the light emitted from the

light sources

21 and 22 has a low absorption rate in the eyeball reaching the retina of the driver's eye, light of this wavelength is easily reflected by the retina. For example, when the first light source 21 is turned on, in the image acquired by the first camera 23 that is substantially coaxial with the first light source 21, infrared light reflected by the retina is detected through the pupil, and the pupil looks bright. This image is extracted as a bright pupil image by the bright pupil image detection unit 61. The same applies to an image acquired by the second camera 24 that is substantially coaxial with the second light source 22 when it is turned on.

On the other hand, when the first light source 21 is turned on and an image is acquired by the second camera 24 that is non-coaxial with the first light source 21, the infrared light reflected by the retina is transmitted to the second camera 24. The pupil appears dark because it is hardly incident. Therefore, this image is extracted by the dark pupil image detection unit 62 as a dark pupil image. The same applies to an image acquired by the non-coaxial first camera 23 when the second light source 22 is turned on.

The pupil center calculation unit 63 subtracts the dark pupil image detected by the dark pupil image detection unit 62 from the bright pupil image detected by the bright pupil image detection unit 61 to generate a pupil image signal whose pupil shape is bright. To be acquired. In the pupil center calculation unit 63, the pupil image signal is image-processed and binarized, and an area image corresponding to the shape and area of the pupil is calculated. Further, an ellipse including this area image is extracted, and an intersection point between the major axis and the minor axis of the ellipse is calculated as a feature amount as the center position of the pupil. Alternatively, the center position of the pupil may be calculated from the luminance distribution of the pupil image.

The dark pupil image signal detected by the dark pupil image detection unit 62 is given to the corneal reflection light center detection unit 64. The dark pupil image signal includes a luminance signal by reflected light reflected from the reflection point of the cornea. The reflected light from the reflection point of the cornea forms a Purkinje image, and is acquired as a spot image with a very small area by the imaging devices of the

cameras

23 and 24. The corneal reflection light center detection unit 64 performs image processing on the spot image, and obtains the center of the reflected light from the reflection point of the cornea as a feature amount.

The pupil center calculated value calculated by the pupil center calculating unit 63 and the corneal reflected light center calculated value calculated by the corneal reflected light center detecting unit 64 are given to the gaze direction calculating unit 65. The line-of-sight direction calculation unit 65 detects the direction of the line of sight from the pupil center calculated value and the corneal reflection light center calculated value.

The line-of-sight direction calculation unit 65 calculates a linear distance α between the center of the pupil and the center of the reflection point from the cornea. In addition, XY coordinates with the center of the pupil as the origin are set, and an inclination angle β between the line connecting the center of the pupil and the center of the reflection point and the X axis is calculated. Further, the line-of-sight direction is calculated from the linear distance α and the inclination angle β. The calculated gaze direction data is output to the control unit 11 as a detection result by the gaze direction calculation unit 65.
The line-of-sight direction may be calculated using the iris center instead of the pupil center. For the iris center, for example, the difference between the reflectance of the iris (black eye) and white eye of the image satisfying the bright pupil photographing condition is used to extract the iris part into an ellipse or a circle, and the center of the extracted figure is calculated. Ask by.

<Flow of gaze detection>
A flow of gaze detection using the gaze detection device 10 of the first embodiment will be described with reference to FIGS. 4 and 6. FIG. 6 is a flowchart showing the flow of gaze detection according to the first embodiment.

First, the entire image A1 (FIG. 4) of the subject SB is acquired by the image acquisition unit 20 (step S11 in FIG. 6). Specifically, the first light source 21 and the second light source 22 emit light alternately, and the first camera 23 and the second camera 24 simultaneously capture images in synchronization with the lighting of the first light source 21. At this time, a bright pupil image is acquired by the first camera 23 and a dark pupil image is acquired by the second camera 24. Even during the period when the second light source 22 is lit, the first camera 23 and the second camera 24 capture images simultaneously in synchronization with the lighting. At this time, a dark pupil image is acquired by the first camera 23 and a bright pupil image is acquired by the second camera 24. The captured image data is stored in the memory 12, and the bright pupil image acquired by the first camera 23 or the second camera 24 is given to the face detection unit 30 as the entire image A1.

Next, the face detection unit 30 performs face detection processing on the entire image A1 (FIG. 4) given from the image acquisition unit 20 (step S12 in FIG. 6). Prior to face detection processing, the face detection unit 30 performs downsizing to reduce the number of pixels by binning processing or the like. The face detection unit 30 performs face detection by applying various detection methods to the downsized image, and extracts a face image A2. For example, initial detection is performed based on the Haar-like face detection method, and information on general facial feature parts registered in the memory 12 in advance, for example, the eyebrow BR and eyeball EB in the entire image A1 shown in FIG. The face image A2 is extracted by comparing the positions, shapes, sizes, etc. of the iris IR, the nose NS, the lips LP, etc. with each other. In addition, as the three-dimensional face data, the face orientation is also detected by comparing the acquired image with information on each characteristic part for a plurality of face orientations, for example, front, diagonal right direction, and diagonal left direction. Furthermore, the face detection unit 30 is based on the color and brightness of the detected face image, and a plurality of landmarks corresponding to each feature part, for example, the contour lines of the eyebrows BR, the eyeballs EB, the iris IR, and the lips LP. The ridgeline of the nose NS is detected. Detection information about the detected face image A2 and landmark is output to the normalization processing unit 40.

Subsequently, the normalization processing unit 40 maintains the relationship between the plurality of landmarks detected by the face detection unit 30 so that the face is directed frontward and has a predetermined size by, for example, affine transformation. The face image is normalized by this conversion (step S13). The normalized image data is sent to the eye region image acquisition unit 50, and the eye region image acquisition unit 50 generates an image including the eyeballs of both eyes based on the position / range information of the eyeballs detected as landmarks. The included range is set as the initial predetermined range A3 (FIG. 4) (step S14). Furthermore, the eye region image acquisition unit 50 reads the bright pupil image and the dark pupil image acquired by the image acquisition unit 20 from the memory 12, and extracts and acquires an image in a range corresponding to the predetermined range A3 in these images. (Step S15). The image acquired in this way is given to the control unit 11 as a determination unit.

Next, the control unit 11 as a determination unit determines whether or not an eye region image is included in the image received from the eye region image acquisition unit 50 (step S16, first determination step). This determination is performed by comparing general eye feature information registered in advance in the memory 12 with, for example, the position, shape, and size of the eyeball and iris. Here, in step S14 and S15, since the predetermined range A3 is set based on the image including the eyeballs of both eyes, the control unit 11 adds the eye region image to the image received from the eye region image acquisition unit 50. Is included (YES in step S16). The control unit 11 outputs the image received from the eye area image acquisition unit 50 to the line-of-sight detection unit 60.

In the line-of-sight detection unit 60 that has received the eye region image, first, the bright pupil image detection unit 61 detects the bright pupil image, and the dark pupil image detection unit 62 detects the dark pupil image. Further, the pupil center calculation unit 63 subtracts the dark pupil image from the bright pupil image to obtain a pupil image signal in which the shape of the pupil is bright, and based on this signal, a portion corresponding to the shape and area of the pupil The center position of the pupil is calculated as a feature amount from the ellipse including this area image (step S17). Further, the corneal reflection light center detection unit 64 performs image processing on the spot image included in the dark pupil image signal, and obtains the center of the reflection light from the reflection point of the cornea as a feature amount (step S17).

Next, the gaze direction calculation unit 65 detects the gaze direction from the pupil center calculation value calculated by the pupil center calculation unit 63 and the corneal reflection light center calculation value calculated by the corneal reflection light center detection unit 64. (Step S18).

After detecting the line-of-sight direction, an image is acquired within the predetermined range set in step S14 (step S15). In this image acquisition method, as in step S11, the first light source 21 and the second light source 22 are alternately turned on, and the pupil images corresponding to the imaging conditions are captured by the two

cameras

23 and 24. Here, not the entire image but only an image in a predetermined range is acquired, so that the data size can be kept small, and the subsequent processing can be executed at high speed. For the image acquired here, the control unit 11 determines whether or not an eye region image is included (step S16, first determination step).

When the eye region image is included as a result of the determination by the control unit 11 (step S16) (YES in step S16), the line-of-sight detection unit 60 extracts the positions of the pupil center and the corneal reflection light center as the feature amount. (Step S17). Based on this feature amount, the line-of-sight direction calculation unit 65 detects the line-of-sight direction (step S18).

On the other hand, if the eye area image is not included as a result of the determination by the control unit 11 (step S16) (NO in step S16), the entire image is acquired again (step S11). Then, face image detection (step S12) and normalization processing (step S13) are executed, and a new predetermined range is set for the normalized image. With this new predetermined range, the data of the predetermined range stored in the memory 12 is updated (step S14), and the subsequent processing after image acquisition (step S15) is performed. Here, when the eye region image is not included, when only the image of the eyeball of one eye is included, the image of the eyeball of both eyes has sufficient density and resolution for feature amount detection. The case where there was not.

With the configuration as described above, the visual line detection method according to the first embodiment has the following effects.
(1) In the first determination step (step S16 in FIG. 6), it is determined whether or not the eye area image of the subject is included in the image of the predetermined range. Without detecting the entire image, the line-of-sight direction is continuously detected based on the image in the predetermined range. For this reason, it is possible to suppress the data size of the image acquired every time while ensuring the accuracy of the line-of-sight detection, and to reduce the processing load and increase the processing speed.
(2) When an image sensor driven by a rolling shutter system is used, the cost of the image sensor can be reduced, and the burden of calculation processing can be reduced and high-speed and high-precision gaze direction detection can be realized. .

Second Embodiment
Next, a second embodiment of the present invention will be described. In the second embodiment, regardless of whether or not an eye region image is included in a predetermined range, the second determination step is executed by periodically acquiring a low resolution whole image. And different. The line-of-sight detection apparatus according to the second embodiment has the same configuration as the line-of-sight detection apparatus 10 according to the first embodiment. Hereinafter, detailed description of the same configuration, processing, action, and the like as in the first embodiment will be omitted.

FIG. 7 is a flowchart showing the flow of gaze detection according to the second embodiment.
First, similarly to the line-of-sight detection (FIG. 6) of the first embodiment, the image acquisition unit 20 acquires the entire image A1 (FIG. 4) of the subject SB (step S21 in FIG. 7), and uses this entire image A1. Then, the face detection unit 30 performs face detection processing and extracts the face image A2 (step S22). Further, the face detection unit 30 detects a face orientation and a plurality of landmarks corresponding to each feature part. Further, the normalization of the face image in the normalization processing unit 40 (step S23), the setting of the predetermined range A3 in the eye region image acquisition unit 50 (step S24), and the determination in the control unit 11 as the determination unit (step S26, The first determination step is the same as in the first embodiment. Here, in step S24 and S25, since the predetermined range A3 is set based on the image including the eyeballs of both eyes, the control unit 11 adds the eye region image to the image received from the eye region image acquisition unit 50. Is included (YES in step S26), and the image received from the eye region image acquisition unit 50 is output to the line-of-sight detection unit 60.

In the line-of-sight detection unit 60 that has received the eye region image, as in the first embodiment, first, the bright pupil image detection unit 61 detects the bright pupil image, and the dark pupil image detection unit 62 detects the dark pupil image. . Further, the pupil center calculation unit 63 subtracts the dark pupil image from the bright pupil image to obtain a pupil image signal in which the shape of the pupil is bright, and based on this signal, a portion corresponding to the shape and area of the pupil The center position of the pupil is calculated as a feature amount from the ellipse including this area image (step S27). In addition, the corneal reflection light center detection unit 64 performs image processing on the spot image included in the dark pupil image signal, and obtains the center of the reflected light from the reflection point of the cornea as a feature amount (step S27). Subsequently, the gaze direction calculation unit 65 detects the gaze direction from the pupil center calculation value calculated by the pupil center calculation unit 63 and the corneal reflection light center calculation value calculated by the corneal reflection light center detection unit 64. (Step S28).

Next, the entire image of the subject SB is acquired by the image acquisition unit 20 (step S29). This image has a resolution lower than that of the image acquired in step S21, and has a minimum resolution that enables simple face image detection described below. In the face detection unit 30, face image detection processing is executed based on this image (step S30, second determination step). In this face image detection, it is confirmed that the position and orientation of the face are not deviated by a predetermined amount or more with respect to the face image detection in step S22 by comparing feature parts, and the detection of landmarks is omitted. This predetermined amount is set as a reference amount in which the eye area image is again included in the predetermined range A3 once set in the general feature site arrangement.

When the face image is detected (YES in step S30), that is, when the face position and orientation are within the range of the deviation amount less than the predetermined amount with respect to the result of the face image detection in step S22, step S24 is performed. An image is acquired for the predetermined range set in (Step S25). The resolution of the image acquired here is as high as the image acquired in step S21, and is higher than the resolution of the image acquired in step S29. As in the image acquisition method, the first light source 21 and the second light source 22 are turned on alternately, and the pupil images corresponding to the imaging conditions are captured by the two

cameras

23 and 24, as in step S21. For the acquired image, the control unit 11 determines whether or not an eye area image is included (step S26, first determination step).

When the eye region image is included as a result of the determination by the control unit 11 (step S26) (YES in step S26), the line-of-sight detection unit 60 extracts the positions of the pupil center and the corneal reflection light center as the feature amount. (Step S27). Based on this feature amount, the line-of-sight direction calculation unit 65 detects the line-of-sight direction (step S28).

(1) As a result of the determination by the control unit 11 (step S26), when an eye area image is not included (NO in step S26), and (2) when a face image cannot be detected in step S30 ( If NO in step S30), that is, if the face position and orientation have deviated by a predetermined amount or more with respect to the result of face image detection in step S22, the entire image is acquired again (step S21). Face image detection (step S22) and normalization processing (step S23) are executed for the entire image, a new predetermined range is set for the normalized image, and the new predetermined range is stored in the memory 12. The data in the predetermined range is updated (step S24), and the subsequent processing after image acquisition (step S25) is performed.

Note that the second determination step shown in step S30 of FIG. 7 is executed every time the detection of the line-of-sight direction (step S28) ends, but this execution interval may be set every predetermined number of times instead of every time. Further, instead of the second determination step shown in step S30 of FIG. 7, the second determination step may be executed independently of the processing flow shown in FIG.

According to the line-of-sight detection method of the second embodiment, since it is possible to perform the determination with an image having a small data amount in the second determination step, it is possible to reduce the burden of calculation processing while ensuring the accuracy of line-of-sight detection. it can.
Other operations, effects, and modifications are the same as those in the first embodiment.
Although the present invention has been described with reference to the above embodiment, the present invention is not limited to the above embodiment, and can be improved or changed within the scope of the purpose of the improvement or the idea of the present invention.

As described above, the line-of-sight detection method according to the present invention is useful in that the processing load can be reduced and the processing speed can be increased while ensuring the accuracy of line-of-sight detection.

DESCRIPTION OF SYMBOLS 10 Eye-gaze detection apparatus 11 Control part 12 Memory 20 Image acquisition part 21 1st light source 22 2nd light source 23 1st camera 24 2nd camera 25 Exposure control part 26 Light source control part 30 Face detection part 40 Normalization process part 50 Eye area | region image Acquisition unit 60 Gaze detection unit 61 Bright pupil image detection unit 62 Dark pupil image detection unit 63 Pupil center calculation unit 64 Corneal reflection light center detection unit 65 Gaze direction calculation unit A1 Whole image A2 Face image A3 Predetermined range

Claims

A first determination step of determining at regular intervals whether or not the eye area image of the subject is included in an image of a predetermined range acquired to extract the eye area image;
When the target region's eye region image is included in the image of the predetermined range in the first determination step, the eye region image is extracted, and the subject's eye direction is based on the extracted eye region image Detect
If the eye area image of the subject is not included in the image of the predetermined range in the first determination step, a new whole image is obtained, and the face image of the subject is detected from the whole image, An eye region image of the subject is extracted from the detected face image, a line-of-sight direction of the subject is detected based on the extracted eye region image, and a range including the extracted eye region image is defined as the predetermined range. A line-of-sight detection method characterized by updating as a range.
Independently of the first determination step, there is a second determination step of acquiring an entire image and determining whether or not the subject's face image can be detected from the acquired entire image,
The entire image acquired in the second determination step has a lower resolution than the image in the predetermined range determined in the first determination step.
If the face image of the subject cannot be detected from the entire image acquired in the second determination step, a new entire image is acquired without waiting for the next first determination step, and the image is A face image of the subject is detected, the eye region image of the subject is extracted from the detected face image, a line-of-sight direction of the subject is detected based on the extracted eye region image, and the extracted The line-of-sight detection method according to claim 1, wherein a range including an eye region image is updated as the predetermined range.
Image acquisition is performed by an imaging device in which a plurality of pixels are arranged in a horizontal direction and a vertical direction and driven by a rolling shutter system.
The line-of-sight detection method according to claim 1, wherein the predetermined range includes one or more lines arranged in the horizontal direction of the image sensor.