JP5601179B2 - Gaze detection apparatus and gaze detection method - Google Patents

Description

  The present invention relates to a gaze detection apparatus and a gaze detection method that detect a gaze direction by detecting, for example, a cornea reflection image.

  2. Description of the Related Art Conventionally, a line-of-sight detection device that detects a user's line-of-sight direction by detecting a cornea reflection image (Purkinje image) of a light source by analyzing an image obtained by photographing a user's eyes has been developed. Such a gaze detection device cannot detect the gaze direction unless it can detect the Purkinje image. Therefore, it is preferable that the exposure amount of the camera that captures the user's eyes is appropriately adjusted so that the Purkinje image can be identified on the image.

  However, a device equipped with a line-of-sight detection device is not always used in an environment where the illuminance by ambient light is kept constant to some extent, such as indoors. For example, when such a device is used outdoors, sunlight may strike the user's face or sunlight may directly enter the camera. In such a case, the brightness of the other part is much higher than the brightness of the user's eyes. Therefore, when the exposure amount is adjusted according to the other part, the exposure amount is about the Purkinje image and its surroundings. Run short. As a result, the Purkinje image cannot be identified on the image. On the other hand, if the line-of-sight detection device has an illumination light source that emits light having a sufficient amount of light to illuminate the user's face, the Purkinje image and the entire face can be identified regardless of the brightness of the ambient light. An image can be generated. However, such a light source consumes a large amount of power, and it is not preferable to irradiate a user's face, particularly eyes, with light having a large amount of light because it involves danger.

  Therefore, the face direction is detected by detecting the contour of the user's face from the first image with a relatively small amount of exposure, while the user's eyes appear in the second image with a relatively large amount of exposure. There has been proposed a technique for detecting the line-of-sight direction using the above information (see, for example, Patent Document 1).

JP 2009-276849 A

  However, if the exposure amount at the time of shooting the first image or the second image is not appropriate, it is difficult to detect the face outline or eyes from these images. Therefore, when the exposure amount is inappropriate, the visual line detection device may have to re-photograph the user's face after adjusting the opening time of the electronic shutter or the gain of the image sensor, for example. When re-photographing is performed, the processing time required for eye-gaze detection is increased by the period required for re-photographing. Longer processing time is not preferable especially for applications that require the gaze direction to be known in real time as much as possible.

  In view of this, it is an object of the present specification to provide a gaze detection apparatus that can detect a user's gaze direction regardless of the brightness of the surrounding environment and can suppress an increase in processing time required for gaze detection.

  According to one embodiment, a line-of-sight detection device is provided. The line-of-sight detection device combines a light source that illuminates a user's face, an imaging unit that generates a plurality of original images by photographing the user's face multiple times under a predetermined exposure condition, and a first number of original images. An image generation unit that generates a first composite image and generates a second composite image by combining a second number of original images greater than the first number; Based on the positional relationship between the face detection unit for detecting the user's face from the image, the Purkinje image detection unit for detecting the Purkinje image of the user's pupil and light source from the second composite image, and the Purkinje image of the user's pupil and light source A line-of-sight detection unit that detects the line-of-sight direction of the user.

The objects and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.
It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

  The line-of-sight detection device disclosed in this specification can detect the user's line-of-sight direction regardless of the brightness of the surrounding environment, and can suppress an increase in processing time required for line-of-sight detection.

It is a hardware block diagram of the gaze detection apparatus by one Embodiment. It is a functional block diagram of a control part. (A) is a figure which shows an example of the 1st synthetic | combination image with the relatively few number of the original images used for the synthesis | combination, (b) is the eye part in the 1st synthetic | combination image shown to (a). FIG. (A) is a figure which shows an example of the 2nd image with relatively many original images used for the synthesis | combination, (b) is an eye part expansion in the 2nd synthesized image shown to (a). FIG. It is a figure which shows an example of a reference table. It is a figure which shows the operation | movement flowchart of a gaze detection process. It is a figure which shows the operation | movement flowchart of a gaze detection process.

Hereinafter, a gaze detection apparatus according to one embodiment will be described with reference to the drawings.
This line-of-sight detection device creates a plurality of original images by photographing a user's face a plurality of times under an exposure condition that is assumed to have a small exposure amount so that the outline of the face can be detected even under a bright surrounding environment. The line-of-sight detection apparatus detects the user's face by detecting the contour or part of the user's face from the first synthesized image obtained by synthesizing a relatively small number of original images. On the other hand, this gaze detection apparatus detects the user's gaze direction by detecting the Purkinje image and the center of gravity of the pupil from the second synthesized image obtained by synthesizing a relatively large number of original images.
Note that the synthesis is, for example, calculating an addition value of luminance values of corresponding pixels between a plurality of images and setting the addition value to the corresponding pixel on the synthesized image. In addition, when the addition value of the luminance value is equal to or higher than the upper limit value of the luminance value of the pixel in the addition, the upper limit value is set as the luminance value of the corresponding pixel on the composite image.

  The line-of-sight detection apparatus according to the present embodiment is mounted on various apparatuses that use the user's line-of-sight direction. For example, the line-of-sight detection device is mounted on a mobile terminal or an in-vehicle driving support device. When the line-of-sight detection device is mounted on a mobile terminal, for example, the mobile terminal specifies the position where the user gazes on the display of the mobile terminal based on information about the user's line-of-sight direction received from the line-of-sight detection device, A process corresponding to the icon displayed at the position is executed.

  Further, when the line-of-sight detection device is mounted on an in-vehicle driving support device, for example, the driving support device is configured so that the user's line-of-sight direction is a predetermined range in front of the vehicle based on information about the user's line-of-sight received from the line-of-sight detection device It is determined whether it is off. If the user's line-of-sight direction deviates from the predetermined range over a certain period while the vehicle is moving forward, the driving support device determines that the user is looking around and alerts the user through a speaker or the like. Make a sound. Further, the driving support device receives information on the face direction of the user from the line-of-sight detection device, determines whether or not the user is facing side based on the information, and warns when the user is facing sideways. May be issued.

  FIG. 1 is a hardware configuration diagram of a line-of-sight detection device according to one embodiment. The line-of-sight detection device 1 includes a light source 2, a camera 3, an interface unit 4, a memory 5, and a control unit 6. Note that FIG. 1 is a diagram for explaining components included in the line-of-sight detection device 1, and is not a diagram showing an actual arrangement of each component of the line-of-sight detection device 1. For example, each component included in the line-of-sight detection device 1 may be accommodated in one casing, and some components may be arranged separately from other components. For example, when the line-of-sight detection device 1 is mounted on an in-vehicle driving support device, the light source 2 is attached so as to emit light toward the user near the instrument panel or the ceiling in front of the vehicle interior. The camera 3 is also attached, for example, near an instrument panel or a ceiling in front of the passenger compartment with the user facing. The light source 2 and the camera 3 may be connected to a circuit board disposed on the back side of the instrument panel by mounting the interface unit 4, the memory 5, and the control unit 6 via signal lines.

The light source 2 illuminates the user's face, particularly the eyes and surroundings. Therefore, the light source 2 includes, for example, at least one infrared light emitting diode and a drive circuit that supplies power from a power source (not shown) to the infrared light emitting diode in accordance with a control signal from the control unit 6. The light source 2 emits illumination light while receiving a control signal for turning on the light source from the control unit 6.
Note that the number of light sources 2 included in the line-of-sight detection device 1 is not limited to one, and may be plural.

  The camera 3 is an example of an imaging unit, and generates an original image in which a user's face is shown. For this purpose, the camera 3 has an image sensor having a solid-state imaging device arranged in a two-dimensional manner having sensitivity to light from the light source 2 and an imaging optical system that forms an image of the user's face on the image sensor. . The camera 3 further includes a visible light cut filter between the image sensor and the imaging optical system in order to suppress detection of a reflection image by the iris and a corneal reflection image of light from a light source other than the light source 2. May be. The camera 3 is arranged so that an image of the light source 2 appears on the user's cornea on the original image.

  Further, the number of pixels of the image sensor of the camera 3 can be set to, for example, 1 million, 2 million, or 4 million so that the image has a resolution that can identify the Purkinje image of the light source 2. An image generated by the image sensor is a color image, and each pixel has three color components of red (R), green (G), and blue (B). Each color component takes any value from 0 to 255, for example. The larger the color component value, the higher the luminance of the color. Alternatively, the image generated by the image sensor may be a monochrome image in which the luminance of each pixel increases as the light intensity increases.

  While the line-of-sight detection process is being performed, the camera 3 continuously captures the user's face with a predetermined period and a predetermined exposure condition notified from the control unit 6, so that an original image showing the user's face is captured. Generate multiple. The camera 3 outputs the generated original image to the control unit 6 every time an original image is generated.

  The interface unit 4 includes an interface circuit for connecting the line-of-sight detection device 1 to another device, for example, a device on which the line-of-sight detection device 1 is mounted. For example, when the line-of-sight detection device 1 is mounted on an in-vehicle driving support device, the interface unit 4 transmits or receives a signal via a communication line conforming to a control area network (CAN), for example. Circuit. And the interface part 4 transmits the information regarding a user's gaze direction or face direction received from the control part 6, for example to the control board of the vehicle-mounted driving assistance apparatus connected with the gaze detection apparatus 1 via CAN.

The memory 5 includes, for example, a readable / writable nonvolatile semiconductor memory. The memory 5 stores a program for eye gaze detection processing executed on the control unit 6. The memory 5 is used to detect the user's line-of-sight direction or the user's face, such as a line-of-sight direction reference table that represents the relationship between the pupil's center of gravity relative to the center of gravity of the Purkinje image and the user's line-of-sight direction. Various types of data are stored.
The memory 5 stores a predetermined number of original images taken most recently, a first composite image used for detecting a user's face and the like, and a second composite image used for detecting a pupil and Purkinje image. To do. Note that the predetermined number is, for example, the maximum number of original images used for generating the second composite image. Details of the composite image will be described later.

The control unit 6 includes one or a plurality of processors and their peripheral circuits, and is connected to each unit of the visual line detection device 1 through a signal line. And the control part 6 controls each part of the visual line detection apparatus 1 by transmitting a control signal to each part of the visual line detection apparatus 1 through a signal line.
The control unit 6 executes the gaze direction detection process while the application using the information on the gaze direction of the user is being executed on the device on which the gaze detection device 1 is mounted. In addition, the control unit 6 determines the relative positional relationship between the Purkinje image of the light source and the center of the pupil and the user's line-of-sight direction at a specific timing, such as when the line-of-sight detection device 1 is started or when the above-described application is started. A calibration process for obtaining the relationship is executed.

FIG. 2 is a functional block diagram of the control unit 6. The control unit 6 includes a light source control unit 11, a camera control unit 12, an image generation unit 13, a face detection unit 14, a Purkinje image detection unit 15, a line-of-sight detection unit 16, and a calibration unit 17.
The light source control unit 11, the camera control unit 12, the image generation unit 13, the face detection unit 14, the Purkinje image detection unit 15, the line-of-sight detection unit 16, and the calibration unit 17 are computer programs executed on a processor included in the control unit 6. It is a functional module realized by.
Alternatively, the light source control unit 11, the camera control unit 12, the image generation unit 13, the face detection unit 14, the Purkinje image detection unit 15, the line-of-sight detection unit 16, and the calibration unit 17 are each separate from the processor included in the control unit 6. It may be formed as a circuit of Alternatively, the light source control unit 11, the camera control unit 12, the image generation unit 13, the face detection unit 14, the Purkinje image detection unit 15, the line-of-sight detection unit 16, and the calibration unit 17 are integrated with circuits corresponding to the respective units. The line-of-sight detection device 1 may be mounted as an integrated circuit.

  The light source control unit 11 turns on or off the light source 2. For example, the light source control unit 11 turns on the light source 2 during execution of the line-of-sight detection process. In addition, when a plurality of light sources 2 are included, the light source control unit 11 may transmit a control signal to each light source so that two or more light sources are turned on simultaneously. When a plurality of light sources are turned on at the same time, the Purkinje image also includes images of a plurality of light sources that are turned on. Therefore, the line-of-sight detection device 1 uses a light source in which the arrangement pattern of bright spots on the image is actually turned on. It can be checked whether or not it matches the arrangement pattern. Therefore, the line-of-sight detection device 1 can easily distinguish between an image of another light source such as an indoor lamp or the sun and an image of the light source included in the line-of-sight detection device 1.

  The camera control unit 12 controls the period and exposure conditions that the camera 3 captures. The camera control unit 12 is, for example, in a period in which the camera 3 can take a plurality of times while the user is gazing at a predetermined direction during the line-of-sight detection process, for example, a period of 30 msec to 300 msec. The camera 3 is continuously photographed. The camera control unit 12 is, for example, an exposure condition in which the luminance value of the pixel corresponding to the user's face on the original image does not saturate when the average illuminance within the photographing target range of the camera 3 is a maximum value. The camera 3 is caused to perform each shooting. This prevents the face from becoming unidentifiable on the original image due to saturation of the luminance value. Therefore, when creating the first composite image, the control unit 6 first sets the number of original images to be used for the composition to a small value, and then increases the number to detect the user's face. It is possible to create a composite image suitable for use in the above. This exposure condition is obtained by calculation based on, for example, the sensitivity of the imaging device of the camera 3 or experimentally. Further, the exposure condition is preferably set so that the Purkinje image and surrounding pixels do not have the minimum luminance value in each original image so that the Purkinje image can be detected on the second composite image.

  The camera control unit 12 notifies the camera 3 of the shooting cycle and exposure conditions. Note that the exposure condition includes, for example, a shutter speed and an F value. In addition, when the line-of-sight detection process is started, the camera control unit 12 transmits a control signal instructing the camera 3 to start shooting. On the other hand, every time an original image is received from the camera 3, the camera control unit 12 temporarily stores the original image in the memory 5.

  The image generation unit 13 generates a first synthesized image by synthesizing the original image of the first cumulative number stored in the memory 5. Further, the image generation unit 13 generates a second composite image by combining the original images having the second integration number larger than the first integration number stored in the memory 5.

  Since the exposure condition for each original image is constant, the greater the number of original images used for composition, the substantially the same image as the image captured under the exposure condition with a larger exposure amount. Therefore, if the number of original images is small, the contrast of an area with relatively low contrast in the original image, for example, a shadowed portion, is not sufficiently high on the composite image. On the other hand, a region having a relatively high contrast in the original image, for example, a face outline portion when the face is illuminated by ambient light, has a sufficient contrast even on a synthesized image in which the number of original images used for synthesis is small. . Therefore, such a composite image with a small number of original images is suitable for detection of a part having a relatively large luminance difference from the surroundings, such as a face outline and eyes, nose and mouth.

  On the other hand, in a composite image in which the number of original images used for composition is large, the values of pixels included in the area illuminated by the ambient light are saturated. Therefore, for example, when the face is illuminated by ambient light from one side, the brightness of the pixels in the illuminated area on the composite image becomes constant at the maximum value that can be taken, and on the composite image The part in the area becomes indistinguishable. However, in a region where the contrast is low in the original image, the luminance difference between neighboring pixels increases according to the number of original images, so that it is easy to identify a part that appears in such a region. Therefore, a composite image having a large number of original images is suitable for detecting a pupil, a Purkinje image, or the like that may be shaded by ambient light.

  FIG. 3A is a diagram showing an example of a first composite image in which the number of original images used for composition is relatively small, and FIG. 3B is an eye in the composite image shown in FIG. It is the figure which expanded the part. In the first composite image 300, the contour of the face and the parts such as the eyes, nose and mouth are shown with a distinguishable contrast. On the other hand, as shown in FIG. 3B, the cornea region 310 of the eye is almost uniformly dark, and therefore it is difficult to detect the Purkinje image from the composite image 300.

  FIG. 4A is a diagram showing an example of a second synthesized image in which the number of original images used for synthesis is relatively large, and FIG. 4B is an eye in the synthesized image shown in FIG. It is the figure which expanded the part. In the second composite image 400, since the luminance of the pixels included in the region 410 is saturated, it is difficult to detect the face outline included in the region 410. On the other hand, as shown in FIG. 4B, the cornea region 420 of the eye has a sufficient contrast, and the luminance difference between the Purkinje image 430 in the cornea region 420 and its surroundings is large. Therefore, it is easy to detect the Purkinje image from the composite image 400.

Thus, the first composite image is used to detect a face region, while the second composite image is used to detect a line-of-sight direction using a Purkinje image.
In order to determine the first and second integration numbers, for example, when the device on which the line-of-sight detection device 1 is mounted has an illuminometer that measures the illuminance around the device, the image generation unit 13 The illuminance measurement value is acquired from the illuminance meter via the interface unit 4. In this case, the memory 5 stores in advance an accumulated number reference table that represents the relationship between the illuminance and the first and second accumulated numbers. Then, the image generation unit 13 determines the first and second integration numbers corresponding to the measured illuminance value by referring to the integration number reference table.
Note that the first cumulative number is set so that, for example, a contrast is obtained experimentally so that the contour of the face and each part such as the eyes, nose, and mouth can be identified on the first composite image. On the other hand, the second integration number is set so that, for example, a contrast is obtained experimentally so that the pupil and the Purkinje image can be identified on the second composite image.

  Further, the image generation unit 13 may determine the first and second integration numbers based on any original image stored in the memory 5, for example, the average luminance value of a predetermined area on the latest original image. Good. The predetermined area has a width and height that is 1/4 to 1/2 of the width and height of the original image including the center of the original image, for example, an area where a face is likely to appear on the original image. It can be a partial area.

  Further, when notified by the face detection unit 14 that face detection or face orientation detection has failed, the image generation unit 13 corrects the first integration number to be increased by a predetermined number of 1 or more. Then, the image generation unit 13 generates the corrected first synthesized image by synthesizing the corrected first integrated number of original images, and the face detection unit again generates the corrected first synthesized image. 14 Similarly, when notified by the Purkinje image detection unit 15 that the detection of the pupil or the detection of the Purkinje image has failed, the image generation unit 13 corrects the second cumulative number to be increased by a predetermined number of 1 or more. Then, the image generation unit 13 generates a corrected second composite image by synthesizing the corrected second integrated number of original images, and the corrected second combined image is again generated by the Purkinje detection unit. Pass to 15.

In order to generate the first composite image, the image generation unit 13 obtains the sum of the values of the pixels of the original image of the first integration number for each pixel of the first composite image, and calculates the sum as the first composite image. The value of the pixel of the composite image is used. Similarly, in order to generate the second composite image, the image generation unit 13 calculates the sum of the values of the pixels of the original image of the second integration number for each pixel of the second composite image, and calculates the sum. It is set as the value of the pixel in the second composite image. When the original image is a color image, the image generation unit 13 calculates the sum of the color values for each pixel for each color.
The image generation unit 13 passes the first composite image to the face detection unit 14, while passing the second composite image to the Purkinje image detection unit 15.

The face detection unit 14 detects the user's face on the first composite image during the execution of the line-of-sight detection process or the calibration process.
In the present embodiment, at the time of shooting, the user's face is illuminated with infrared light from the light source 2, and the reflectance of the skin with respect to the infrared light is relatively high (for example, the reflectance of the skin is near infrared. (Several tens of percent in the wavelength range), the brightness of the pixels in which the skin of the face is reflected on the image is high. On the other hand, since the area of the hair or the user's back area on the image has a low reflectance with respect to infrared light or is far from the light source, the luminance of the pixel in which the hair or the area behind the user is reflected is relatively low.
Therefore, when the value of each pixel of the image is a color image represented by the RGB color system, the face detection unit 14 converts the value of each pixel into a value represented by the YUV color system. Then, the face detection unit 14 extracts pixels whose luminance component (Y component) value of each pixel is equal to or greater than a predetermined threshold as face area candidate pixels that may have a face. When the value of each pixel of the image is a monochrome image representing luminance, the face detection unit 14 compares the value of each pixel with a predetermined threshold value. For example, the predetermined threshold is set to a value obtained by multiplying the maximum value of the luminance component on the image by 0.8.

Further, the area occupied by the user's face on the first composite image is relatively large, and the size of the area occupied by the face on the first composite image is estimated to some extent.
Therefore, the face detection unit 14 performs a labeling process on the face area candidate pixels, and sets a set of face area candidate pixels adjacent to each other as a face candidate area. Then, the face detection unit 14 determines whether the size of the face candidate area is included in a reference range corresponding to the size of the user's face. If the size of the face candidate area is included in the reference range corresponding to the size of the user's face, the face detection unit 14 determines that the face candidate area is a face area in which the user's face is reflected.
Note that the size of the face candidate area is represented by, for example, the maximum number of pixels in the horizontal direction of the face candidate area. In this case, the reference range is set to, for example, not less than 1/4 and not more than 2/3 of the number of pixels in the horizontal direction of the first composite image. Alternatively, the size of the face candidate area may be represented by the number of pixels included in the face candidate area, for example. In this case, the reference range is set to, for example, 1/16 or more and 4/9 or less of the total number of pixels of the first composite image.

  The face detection unit 14 may add not only the size of the face candidate area but also the shape of the face candidate area to the determination condition for determining the face candidate area as the face area. A human face generally has a substantially elliptical shape. Therefore, the face detection unit 14, for example, when the size of the face candidate area is included in the reference range and the circularity of the face candidate area is equal to or greater than a predetermined threshold corresponding to a general face contour. Alternatively, the face candidate area may be a face area. The face detection unit 14 obtains the total number of pixels positioned on the contour of the face candidate area as the perimeter of the face candidate area, and multiplies the total number of pixels in the face candidate area by 4π as the square of the perimeter. The circularity can be calculated by dividing.

  Alternatively, the face detection unit 14 may approximate the face candidate region to an ellipse by applying the least square method by applying the coordinates of each pixel on the contour of the face candidate region to the elliptic equation. The face detection unit 14 may use the face candidate area as a face area when the ratio of the major axis to the minor axis of the ellipse is included in the range of the ratio of the major axis to the minor axis of the face. Note that when the shape of the face candidate region is evaluated by ellipse approximation, the face detection unit 14 performs an inter-pixel calculation on the luminance component of each pixel of the first composite image to detect an edge pixel corresponding to the edge. May be. In this case, the face detection unit 14 connects the edge pixels using, for example, a labeling process, and uses the edge pixels connected to a certain length or more as the contour of the face candidate region.

  The face detection unit 14 may detect the face area according to any of various other methods for detecting the face area shown on the image. For example, the face detection unit 14 performs template matching between the face candidate area and a template corresponding to a general face shape, calculates the degree of coincidence between the face candidate area and the template, and the degree of coincidence is predetermined. If the value is greater than or equal to the value, the face candidate area may be determined as a face area.

When the face detection unit 14 can extract the face area, the face detection unit 14 generates information representing the face area. For example, the information representing the face area may be a binary image having the same size as the image and having different pixel values for the pixels in the face area and the pixels outside the face area.
The face detection unit 14 passes information representing the face area to the Purkinje image detection unit 15. On the other hand, the face detection unit 14 notifies the image generation unit 13 to increase the first integration number when the detection of the face region fails, for example, when the detection condition of the face region described above is not satisfied.

  Furthermore, the face detection unit 14 may determine the user's face orientation. In this case, the face detection unit 14 detects, for example, characteristic parts such as eyes, nostrils, and mouths in the face area. For this purpose, for example, the face detection unit 14 performs template matching while changing the relative position between the face region and a template corresponding to the general shape of those parts, and matches the face region and the template. Calculate the degree. Then, when the degree of coincidence is a predetermined value or more at a specific position in the face area, the face detection unit 14 determines that a part corresponding to the template is captured at the specific position.

The face detection unit 14 obtains the center line of the face based on the position information of the part. For example, the face detection unit 14 obtains a first midpoint of a line connecting the centroids of the left and right nostrils. Further, the face detection unit 14 obtains a second midpoint of a line connecting the center of gravity of the left and right eyes. The face detection unit 14 sets a line connecting the first midpoint and the second midpoint as the face center line.
The face detection unit 14 obtains the ratio of the number of pixels of the left and right face areas of the face center line or the ratio of the maximum distance from a predetermined point on the face center line to the left and right end points of the face area. Then, the face detection unit 14 determines the orientation of the user's face by referring to a face orientation reference table that represents the relationship between the ratio and the orientation of the face. Note that the relationship between the ratio and the face orientation is determined in advance by experiments, and a face orientation reference table representing the relationship is stored in the memory 5 in advance.

  For example, the face detection unit 14 sets the face direction when facing the camera 3 to 0 °, and is positive when facing the right side when viewed from the camera 3, and when facing the left side. Create a horizontal angle value that represents the face orientation with a negative value. Similarly, the face detection unit 14 sets the face direction to 0 ° when facing the camera 3, and faces the upper side when facing the upper side when viewed from the camera 3. Create a vertical angle value representing the face orientation, sometimes with a negative value. Then, the face detection unit 14 outputs the combination of the horizontal angle value and the vertical angle value as information representing the face direction via the interface unit 4.

The Purkinje image detection unit 15 detects an area in which the user's eyes are reflected in the face area on the second composite image during execution of the line-of-sight detection process or the calibration process. Detect Purkinje image.
The luminance of the pixel corresponding to the eye is significantly different from the luminance of the pixel corresponding to the periphery of the eye. Therefore, the Purkinje image detection unit 15 detects edge pixels whose luminance changes in the vertical direction by performing a difference calculation between neighboring pixels in the vertical direction using, for example, a Sobel filter for each pixel in the face region. The Purkinje image detection unit 15 sets, for example, an eye region that is surrounded by two edge lines in which a predetermined number or more of edge pixels are connected in a substantially horizontal direction corresponding to the size of the eye.
Alternatively, the Purkinje image detection unit 15 detects a region that most closely matches the template in the face region by template matching between the template representing the image of the eye on the image and the face region, and uses the detected region as the eye region. It is good.

  Further, the Purkinje image detection unit 15 detects an area where the pupil is reflected in the eye area. In this embodiment, the Purkinje image detection unit 15 performs template matching between a template corresponding to a pupil and an eye region, and detects a region having the highest degree of matching with the template in the eye region. Then, the Purkinje image detection unit 15 determines that the pupil is reflected in the detected area when the highest coincidence value is higher than a predetermined coincidence threshold value. A plurality of templates may be prepared according to the size of the pupil. In this case, the Purkinje image detection unit 15 executes template matching between each template and the eye region, and obtains the highest value of the matching degree. If the highest coincidence value is higher than the coincidence degree threshold, the Purkinje image detection unit 15 determines that the pupil appears in an area overlapping the template corresponding to the highest coincidence value. Note that the degree of coincidence is calculated as, for example, a normalized cross-correlation value between a template and a region overlapping the template. The coincidence threshold is set to 0.7 or 0.8, for example.

The luminance of the region where the pupil is reflected is lower than the luminance of the surrounding region, and the pupil is substantially circular. Therefore, the Purkinje image detector 15 sets two rings having different radii concentrically within the eye region. When the difference value obtained by subtracting the average luminance value of the inner pixels from the average luminance value of the pixels corresponding to the outer ring is larger than a predetermined threshold, the Purkinje image detection unit 15 is surrounded by the inner ring. The region may be the pupil region. The Purkinje image detection unit 15 may add to the condition for detecting the pupil region that the average luminance value of the region surrounded by the inner ring is equal to or less than a predetermined threshold value. In this case, for example, the predetermined threshold is set to a value obtained by adding 10% to 20% of the difference between the maximum luminance value and the minimum luminance value in the eye region to the minimum luminance value.
When the Purkinje image detection unit 15 succeeds in detecting the pupil region, the Purkinje image detection unit 15 calculates the average value of the horizontal coordinate values and the average value of the vertical coordinate values of each pixel included in the pupil region as the coordinates of the center of gravity of the pupil region. . On the other hand, if the Purkinje image detection unit 15 fails to detect the pupil region, the Purkinje image detection unit 15 notifies the image generation unit 13 to increase the second integration number.

  The Purkinje image detection unit 15 detects the Purkinje image of the light source 2 in the eye region. The luminance of the region where the Purkinje image of the light source 2 is reflected is higher than the luminance of the surrounding region. In addition, the shape of the region where the Purkinje image of the light source 2 is reflected substantially matches the shape of the light emitting surface of each light source. Therefore, the Purkinje image detection unit 15 sets two rings that have a shape that substantially matches the contour shape of the light emitting surface of the light source in the eye region, and that have different sizes and centers. Then, the Purkinje image detection unit 15 has a difference value obtained by subtracting the average value of the luminance of the outer pixels from the average value of the luminance of the pixels corresponding to the inner ring, greater than a predetermined difference threshold value, and A region where the inner luminance average value is higher than a predetermined luminance threshold is detected. Then, the Purkinje image detection unit 15 sets the area surrounded by the inner ring in the detected area as the Purkinje image of the light source 2. Note that the difference threshold value can be, for example, an average value of difference values between neighboring pixels in the eye region. The predetermined luminance threshold can be set to 80% of the maximum luminance value in the eye region, for example.

  Note that the Purkinje image detection unit 15 may detect the region in which the pupil is shown by using any of various other methods for detecting the region in which the pupil is shown on the image. Similarly, the Purkinje image detection unit 15 detects the region where the Purkinje image of the light source is captured using any of various other methods for detecting the region where the Purkinje image of the light source is captured on the image. Also good.

When the Purkinje image detection unit 15 succeeds in detecting the Purkinje image of the light source 2, the average value of the horizontal coordinate value and the average value of the vertical coordinate value of each pixel included in the light source are used as the coordinates of the center of gravity of the Purkinje image. calculate. On the other hand, if the Purkinje image detection unit 15 fails to detect the Purkinje image of the light source, the Purkinje image detection unit 15 notifies the image generation unit 13 to increase the second integration number.
The Purkinje image detection unit 15 notifies the calibration unit 17 of the center of gravity of the Purkinje image and the center of the pupil when performing the calibration process. On the other hand, the Purkinje image detection unit 15 notifies the gaze detection unit 16 of the center of gravity of the Purkinje image and the center of the pupil when performing the gaze detection process.

  The line-of-sight detection unit 16 detects the user's line-of-sight direction based on the center of gravity of the Purkinje image and the center of the pupil during execution of the line-of-sight detection process.

  Since the surface of the cornea is substantially spherical, the position of the Purkinje image of the light source is substantially constant regardless of the line-of-sight direction. On the other hand, the center of gravity of the pupil moves according to the user's line-of-sight direction. Therefore, the line-of-sight detection unit 16 can detect the user's line-of-sight direction by obtaining the relative position of the pupil center of gravity with respect to the center of gravity of the Purkinje image.

  In the present embodiment, the line-of-sight detection unit 16 determines the relative position of the pupil centroid relative to the centroid of the Purkinje image of the light source, for example, the horizontal direction of the centroid of the Purkinje image from the horizontal coordinate and the vertical coordinate of the pupil centroid. It is obtained by subtracting the coordinate and the vertical coordinate. The line-of-sight detection unit 16 determines the user's line-of-sight direction by referring to a line-of-sight direction reference table that represents the relationship between the relative position of the pupil center of gravity and the user's line-of-sight direction. The line-of-sight detection unit 16 outputs the user's line-of-sight direction via the interface unit 4.

  FIG. 5 is a diagram illustrating an example of a line-of-sight direction reference table. In each column in the left column of the line-of-sight direction reference table 500, coordinates of the relative position of the pupil centroid with respect to the centroid of the Purkinje image of the light source are represented. In each column in the right column of the line-of-sight direction reference table 500, the line-of-sight direction corresponding to the coordinates of the relative position of the pupil center of gravity in the same row is represented. Note that the coordinates of the relative position of the pupil center of gravity are expressed in units of pixels on the second composite image. On the other hand, the line-of-sight direction is a set of a predetermined direction, for example, a direction in which the user's line-of-sight direction is parallel to the optical axis of the camera 3 and a horizontal direction angle and a vertical direction angle formed by the reference direction and the line-of-sight direction. expressed. For example, row 501 indicates that the line-of-sight direction with respect to the coordinate (0,0) of the relative position of the pupil center of gravity is (0 °, 0 °). Also, row 502 indicates that the line-of-sight direction with respect to the coordinates (−2, 0) of the relative position of the pupil center of gravity is (−10 °, 0 °).

The calibration unit 17 updates the line-of-sight direction reference table by executing calibration processing.
For example, when the calibration unit 17 receives a signal indicating that the line-of-sight detection device 1 has been activated from the control unit 6, the calibration unit 17 performs a calibration process. When the calibration process is started, the calibration unit 17 instructs the user to view a predetermined reference position via a user interface such as a display of the device on which the line-of-sight detection device 1 is mounted, for example. Is notified. The reference position can be, for example, the center of the display of the device on which the line-of-sight detection device 1 is mounted, or the installation position of the camera 3. It is assumed that the user gazes at the reference position during calibration.

The calibration unit 17 uses the Purkinje image to determine the coordinates of the center of gravity of the Purkinje image and the center of the pupil detected based on the second composite image created from the original image taken when the user is gazing at the reference position. Received from the detector 15. Then, the calibration unit 17 updates the line-of-sight direction reference table so that the relative position of the pupil center of gravity relative to the center of gravity of the Purkinje image corresponds to the line-of-sight direction corresponding to the reference position. Similarly, the calibration unit 17 uses the center of gravity of the Purkinje image obtained from the second composite image created based on the original image taken when the user is gazing at another reference position as a reference. The line-of-sight direction reference table is updated so that the relative position of the center of gravity of the pupil is associated with the line-of-sight direction corresponding to the reference position.
Further, the calibration unit 17 uses the relative position of the pupil centroid relative to the centroid of the Purkinje image with respect to other sight line directions, using the relative position of the pupil centroid corresponding to the already determined sight line direction. For example, it is determined by linear interpolation.
The calibration unit 17 stores the updated line-of-sight direction reference table in the memory 5.

  6 and 7 are operation flowcharts of the line-of-sight detection process controlled by the control unit 6. The control unit 6 repeatedly executes the line-of-sight detection process shown in this operation flowchart at a predetermined cycle, for example, at intervals of 100 milliseconds to 1 second.

  The camera control unit 12 of the control unit 6 acquires, from the camera 3, a plurality of original images obtained by photographing the user's face with a predetermined cycle and a predetermined exposure condition (step S101). Then, the camera control unit 12 temporarily stores each original image in the memory 5. Further, the image generation unit 13 of the control unit 6 determines the first integration number (step S102). The image generation unit 13 creates a first synthesized image by synthesizing the first accumulated number of original images (step S103). Then, the image generation unit 13 passes the first composite image to the face detection unit 14 of the control unit 6.

  The face detection unit 14 performs a face area and face direction detection process on the first composite image (step S104). Then, the face detection unit 14 determines whether or not the face area and the face direction have been detected (step S105). If either the face area or the face orientation cannot be detected (step S105—No), the face detection unit 14 notifies the image generation unit 13 to increase the first integration number. Then, the image generation unit 13 increases the first integration number by a predetermined number of 1 or more (step S106). And the control part 6 performs the process after step S103 again.

  On the other hand, the face detection unit 14 notifies the Purkinje image detection unit 15 of the face area when both the face area and the face direction can be detected (Yes in step S105). Further, the face detection unit 14 outputs information representing the face orientation to the device on which the line-of-sight detection device 1 is mounted via the interface unit 4 (step S107).

  As illustrated in FIG. 7, the image generation unit 13 determines a second integration number that is greater than the first integration number (step S108). Then, the image generation unit 13 generates a second combined image by combining the second accumulated number of original images (step S109). The image generation unit 13 passes the second composite image to the Purkinje image detection unit 15. The Purkinje image detection unit 15 performs pupil and Purkinje image detection processing on the face region on the second composite image (step S110). Then, the Purkinje image detection unit 15 determines whether or not the pupil and the Purkinje image have been detected (step S111). If either the pupil or the Purkinje image cannot be detected (step S111-No), the Purkinje image detection unit 15 notifies the image generation unit 13 to increase the second integration number. Then, the image generation unit 13 increases the second integration number by a predetermined number of 1 or more (step S112). And the control part 6 performs the process after step S109 again.

  On the other hand, if both the pupil and the Purkinje image can be detected (step S111-Yes), the Purkinje image detection unit 15 notifies the line-of-sight detection unit 16 of the center of gravity of the Purkinje image and the center of the pupil. The line-of-sight detection unit 16 determines the user's line-of-sight direction based on the relative positional relationship between the center of gravity of the Purkinje image and the center of gravity of the pupil (step S113). Then, the line-of-sight detection unit 16 outputs information representing the line-of-sight direction to the device on which the line-of-sight detection device 1 is mounted via the interface unit 4 (step S114).

  Note that the face detection unit 14 may execute only the face area detection process in step S104. In this case, also in step S105, the face detection unit 14 determines whether a face area is detected.

  As described above, this line-of-sight detection device combines a composite image suitable for detection of a face region, a face orientation, and the like by combining different numbers of original images captured under a predetermined exposure condition. A composite image suitable for detection such as a Purkinje image is created separately. Therefore, this gaze detection apparatus can detect the user's gaze direction with reference to the face area even when the surrounding environment is bright. Moreover, even if the detection of the Purkinje image or the like fails, this line-of-sight detection device can create a composite image suitable for the detection of the Purkinje image or the like only by changing the cumulative number of original images. Therefore, this line-of-sight detection device does not need to re-photograph the user's eyes after failure to detect the Purkinje image, and can shorten the time lag until the Purkinje image can be detected, so the time required to detect the line-of-sight direction Can be suppressed.

In addition, this invention is not limited to said embodiment. For example, the camera may generate a dark image that is captured in a state where the illumination light source is turned off at a certain period during the execution of the line-of-sight detection process. The image generation unit of the control unit corrects the difference image obtained by subtracting the luminance value of the corresponding pixel of the dark image from the luminance value of each pixel of the second combined image, and corrects the second combined image. It may be an image.
In this case, only the Purkinje image of the light source is different between the dark image and the original image, and the Purkinje image is not reflected in the dark image. Therefore, as described above, the Purkinje image is enhanced in the difference image between the second composite image and the dark image. On the other hand, even if an image of another light source such as the sun or room light is shown on the cornea, the image of the other light source on the original image and the image of the other light source on the dark image are almost the same. Then, the image of the other light source is canceled and its contrast is lowered. Therefore, the Purkinje image detection unit detects the Purkinje image based on the corrected second composite image, so that it becomes difficult to erroneously detect the image of the other light source as the Purkinje image of the light source of the line-of-sight detection device. The Purkinje image can be detected accurately.
Note that the exposure condition for capturing a dark image may be the same as the exposure condition for capturing an original image. Alternatively, the exposure condition for photographing the dark image is smaller than the exposure amount for obtaining the second composite image by one photographing, and the exposure amount is larger than the exposure condition for photographing the original image. It may be set to be.

In another modification, the face detection unit refers to the position and face orientation of the face area detected in the past in order to determine the position or face orientation of the face area on the latest first composite image. Also good. For example, the face detection unit uses a motion detection process such as block matching to obtain a region on the latest first composite image that most closely matches the face region detected in the past composite image, and determines the region. It may be a face area. Alternatively, the face detection unit uses any of various tracking techniques, and the most probable face from the position of the face area detected in the past among the face candidate areas extracted on the latest first composite image. A candidate area may be obtained and the face candidate area may be used as the face area. Further, when the device on which the line-of-sight detection device is mounted is equipped with a sensor for detecting an angular velocity such as a gyro, the face detection unit acquires information on the angular velocity from the sensor and uses it to determine the face orientation. May be. For example, the face detection unit obtains the rotation angle of the device from the time when the face orientation is detected in the past to the time when the latest first composite image is obtained based on the information about the angular velocity, and the face orientation is determined by the rotation angle. It may be determined that has changed.
In still another modification, the Purkinje image detection unit may perform pupil detection processing on the entire second composite image.

  In addition, the computer program that realizes the function of each unit of the control unit according to the above-described embodiments and modifications is recorded in a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. May be provided.

  All examples and specific terms listed herein are intended for instructional purposes to help the reader understand the concepts contributed by the inventor to the present invention and the promotion of the technology. It should be construed that it is not limited to the construction of any example herein, such specific examples and conditions, with respect to showing the superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

The following supplementary notes are further disclosed regarding the embodiment described above and its modifications.
(Appendix 1)
A light source that illuminates the user's face;
An imaging unit that generates a plurality of original images by photographing a user's face a plurality of times under a predetermined exposure condition;
A first synthesized image is generated by synthesizing a first number of original images among the plurality of original images, and a second number greater than the first number among the plurality of original images. An image generation unit that generates a second combined image by combining a number of original images;
A face detection unit for detecting a user's face from the first composite image;
A Purkinje image detector for detecting a user's pupil and Purkinje image of the light source from the second composite image;
A line-of-sight detection unit that detects a user's line-of-sight direction based on the positional relationship between the user's pupil and the Purkinje image of the light source;
A line-of-sight detection apparatus comprising:
(Appendix 2)
The predetermined exposure condition is a condition in which a luminance value of a pixel corresponding to the user's face is not saturated on the original image when an average illuminance within a photographing target range of the imaging unit is a maximum value assumed. The line-of-sight detection device according to attachment 1.
(Appendix 3)
When the face detection unit fails to detect the user's face, the image generation unit corrects the first number to be increased by a predetermined number, and the corrected first of the plurality of original images is corrected. A first composite image modified by synthesizing a number of original images is generated, and the face detection unit detects a user's face from the modified first composite image. The line-of-sight detection device described.
(Appendix 4)
When the Purkinje image detection unit fails to detect the user's pupil or Purkinje image of the light source, the image generation unit corrects the second number to be increased by a predetermined number, and out of the plurality of original images Generating a modified second composite image by compositing the modified second number of original images, and the Purkinje image detection unit generates the user's pupil and the second image from the modified second composite image. The line-of-sight detection device according to any one of appendices 1 to 3, which detects a Purkinje image of the light source.
(Appendix 5)
The Purkinje image detection unit detects the user's pupil and the Purkinje image of the light source within a region on the second composite image corresponding to the user's face detected by the face detection unit. The visual line detection device according to any one of the above.
(Appendix 6)
The imaging unit further obtains a dark image obtained by photographing the user's face with the light source turned off,
The image generation unit generates, as the second combined image, a difference image between an image obtained by combining the second number of original images and the dark image among the plurality of original images.
The line-of-sight detection device according to any one of appendices 1 to 5.
(Appendix 7)
A plurality of original images are generated by photographing a user's face illuminated by a light source a plurality of times under a predetermined exposure condition,
Generating a first synthesized image by synthesizing a first number of original images among the plurality of original images;
Generating a second synthesized image by synthesizing a second number of original images larger than the first number among the plurality of original images;
Detecting a user's face from the first composite image;
Detecting the pupil of the user and the Purkinje image of the light source from the second composite image;
Detecting the user's line-of-sight direction based on the positional relationship between the user's pupil and the Purkinje image of the light source;
A gaze detection method including the above.
(Appendix 8)
Obtaining a plurality of original images generated by photographing a user's face illuminated by a light source a plurality of times under a predetermined exposure condition;
Generating a first synthesized image by synthesizing a first number of original images among the plurality of original images;
Generating a second synthesized image by synthesizing a second number of original images larger than the first number among the plurality of original images;
Detecting a user's face from the first composite image;
Detecting the pupil of the user and the Purkinje image of the light source from the second composite image;
Detecting the user's line-of-sight direction based on the positional relationship between the user's pupil and the Purkinje image of the light source;
A computer program for eye-gaze detection that causes a computer to execute the above.

DESCRIPTION OF SYMBOLS 1 Line-of-sight detection apparatus 2 Light source 3 Camera 4 Interface part 5 Memory 6 Control part 11 Light source control part 12 Camera control part 13 Image generation part 14 Face detection part 15 Purkinje image detection part 16 Line-of-sight detection part 17 Calibration part

Claims (5)

A light source that illuminates the user's face;
An imaging unit that generates a plurality of original images by photographing a user's face a plurality of times under a predetermined exposure condition;
A first synthesized image is generated by synthesizing a first number of original images among the plurality of original images, and a second number greater than the first number among the plurality of original images. An image generation unit that generates a second combined image by combining a number of original images;
A face detection unit for detecting a user's face from the first composite image;
A Purkinje image detection unit for detecting the user's pupil and the Purkinje image of the light source in an area on the second composite image corresponding to the user's face detected by the face detection unit;
A line-of-sight detection unit that detects a user's line-of-sight direction based on the positional relationship between the user's pupil and the Purkinje image of the light source;
A line-of-sight detection apparatus comprising:   When the face detection unit fails to detect the user's face, the image generation unit corrects the first number to be increased by a predetermined number, and the corrected first of the plurality of original images is corrected. 2. The corrected first composite image is generated by combining the number of original images, and the face detection unit detects a user's face from the corrected first composite image. Gaze detection device.   When the Purkinje image detection unit fails to detect the user's pupil or Purkinje image of the light source, the image generation unit corrects the second number to be increased by a predetermined number, and out of the plurality of original images Generating a modified second composite image by compositing the modified second number of original images, and the Purkinje image detection unit generates the user's pupil and the second image from the modified second composite image. The line-of-sight detection device according to claim 1, wherein a Purkinje image of the light source is detected. The imaging unit further obtains a dark image obtained by photographing the user's face with the light source turned off,
The image generation unit generates, as the second combined image, a difference image between an image obtained by combining the second number of original images and the dark image among the plurality of original images.
The gaze detection device according to any one of claims 1 to 3 . A plurality of original images are generated by photographing a user's face illuminated by a light source a plurality of times under a predetermined exposure condition,
Generating a first synthesized image by synthesizing a first number of original images among the plurality of original images;
Generating a second synthesized image by synthesizing a second number of original images larger than the first number among the plurality of original images;
Detecting a user's face from the first composite image;
Detecting the user's pupil and Purkinje image of the light source in an area on the second composite image corresponding to the detected user's face ;
Detecting the user's line-of-sight direction based on the positional relationship between the user's pupil and the Purkinje image of the light source;
A gaze detection method including the above.

Images