JP2014052813A - Pupil detection device and pupil detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a pupil detection device capable of precisely detecting a pupil with a small computational quantity.SOLUTION: The pupil detection device includes: an image acquisition part which acquires a first image including a pupil of a person being irradiated with infrared light from a light source arranged at a first position and a second image including the pupil of the person irradiated with infrared light from a light source arranged at a second position; a difference value calculation part which calculates difference values between luminances of a plurality of image areas included in the first image and luminances of a plurality of image areas included in the second image and corresponding to the plurality of image areas of the first image; a candidate area specification part which specifies a plurality of successive image areas whose calculated difference values are equal to or larger than a first threshold as a candidate area for the pupil; a variance value calculation part which calculates a variance value indicative of how the respective image areas in the candidate area vary in luminance; and a pupil detection part which detects the candidate area as the position of the pupil of the person when the variance value is equal to or less than a second threshold.

Description

  The present invention relates to a pupil detection device and a pupil detection method.

  2. Description of the Related Art Conventionally, there has been known an apparatus that detects a gaze point of a person to be inspected by causing a person to be inspected to gaze an observation surface such as a monitor screen and analyzing a face image of the person to be inspected at this time. In such an apparatus, the position of the pupil is specified based on the reflection of light by the pupil. However, when the subject to be inspected wears spectacles, the light is also reflected on the lens or frame of the spectacles. In some cases, it is erroneously detected as a pupil.

  As a technique for preventing such erroneous detection, for example, Patent Document 1 acquires an infrared image and a visible image of a person to be inspected, and in a high-luminance region that shows brightness of a predetermined value or more in the infrared image, A technique for calculating a pupil position based on a region where the luminance of a corresponding region in a visible image is less than a predetermined value is disclosed. Patent Document 2 discloses a technique for detecting a cornea reflection image based on two images taken when two light sources are turned on from different directions and each light source is turned on.

Japanese Patent No. 3450801 JP 2010-244156 A

  However, the technique disclosed in Patent Document 1 has a problem in that many regions other than the pupil that show brightness of a predetermined value or more in the infrared image are detected, and the possibility of erroneous detection of the pupil is high. Moreover, in the technique of patent document 2, there existed a problem that the corneal reflection used as a detection target was small and the calculation load for detection became large.

  The present invention has been made in view of the above, and an object of the present invention is to provide a pupil detection device and a pupil detection method capable of accurately detecting a pupil with a small amount of calculation.

  In order to solve the above-described problems and achieve the object, the present invention is a pupil detection device, which is a first image picked up by an image pickup unit, which is infrared light from a light source arranged at a first position. The first image including the pupil of the person in the state of being irradiated and the second image captured by the imaging unit, where the distance between the imaging unit is longer than the first position An image acquisition unit that acquires a second image including the pupil of the person irradiated with infrared light from a light source arranged at a second position, and a plurality of image regions included in the first image A difference value calculating unit for calculating a difference value between each of the plurality of image areas of the second image corresponding to each of the plurality of image areas of the first image; Compared to the first threshold value, a differential value greater than or equal to the first threshold value is calculated. A candidate region specifying unit that specifies the image region as a candidate region of the pupil, a variation value calculating unit that calculates a variation value indicating a degree of luminance variation of each image region in the candidate region, and the variation value A pupil detection unit that compares with a second threshold value and detects the candidate region as the position of the pupil of the person when the variation value is equal to or less than the second threshold value.

  In addition, the present invention is a pupil detection method, including a first image captured by an imaging unit, which includes a human pupil that is irradiated with infrared light from a light source disposed at a first position. A light source disposed at a second position, which is the first image and a second image captured by the imaging unit, the distance between the first image and the imaging unit being longer than the first position An image acquisition step of acquiring a second image including the pupil of the person irradiated with infrared light, brightness of each of a plurality of image regions included in the first image, and a plurality of the first images A difference value calculating step of calculating a difference value of brightness of each of the plurality of image areas of the second image corresponding to each of the image areas, and comparing the difference value with a first threshold value, and a difference greater than or equal to the first threshold value A plurality of consecutive image areas whose values are calculated are candidates for the pupil A candidate area specifying step for specifying as a region, a variation value calculating step for calculating a variation value indicating a degree of luminance variation of each image region in the candidate region, and comparing the variation value with a second threshold value, A pupil detection step of detecting the candidate region as the position of the pupil of the person when a variation value is equal to or less than the second threshold value.

  According to the present invention, it is possible to accurately detect the pupil with a small amount of calculation.

FIG. 1 is a diagram illustrating a schematic configuration of a gaze direction detection device. FIG. 2 is a diagram for explaining a coordinate system used for processing of the line-of-sight direction device. FIG. 3 is a block diagram illustrating an example of detailed functions of each unit illustrated in FIG. 1. FIG. 4 is a flowchart showing the line-of-sight detection process. FIG. 5 is a flowchart showing detailed processing in pupil detection processing. FIG. 6A is a diagram illustrating a bright camera image. FIG. 6B is a diagram illustrating a camera dark image. FIG. 6C is a diagram illustrating the difference image. FIG. 7A is a diagram illustrating the luminance with respect to the reflected light from the pupil and the difference value. FIG. 7B is a diagram illustrating the luminance with respect to the reflected light from the glasses and the difference value. FIG. 8 is a flowchart illustrating processing of the gaze direction detecting device according to the second embodiment.

  Hereinafter, embodiments of a pupil detection device and a pupil detection method will be described in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration of a gaze direction detection device 100 according to the first embodiment. The line-of-sight detection device 100 is a device that detects the pupil of the detection target who is the detection target of the line-of-sight direction and detects the direction of the line of sight based on the position of the pupil, and includes a pupil detection device.

  The line-of-sight detection device 100 includes a first display unit 101, a second display unit 102, a speaker 103, two cameras 110R and 110L functioning as a stereo camera, bright light sources 112R and 112L, and dark light sources 114R and 114L. A drive / IF unit 130, a control unit 140, and a storage unit 160.

  The 1st display part 101 displays the image etc. which a to-be-detected person gazes at. The second display unit 102 displays the operation content of the line-of-sight direction detection device 100 and the like. The first display unit 101 and the second display unit 102 are, for example, monitor screens. The speaker 103 outputs a sound or the like for alerting the person to be detected when the line-of-sight direction is detected. Note that the second display unit 102 may not be provided, and in this case, the operation content and the like may be displayed by the first display unit 101.

  The right camera 110 </ b> R and the left camera 110 </ b> L are respectively installed on the right side and the left side toward the first display unit 101 below the first display unit 101. The right camera 110R and the left camera 110L are imaging units capable of stereo shooting. Hereinafter, the right camera 110R and the left camera 110L are appropriately referred to as a stereo camera 110.

  The stereo camera 110 images the face of the person to be detected. The stereo camera 110 captures an image at a timing when infrared light is irradiated from the bright light sources 112R and 112L and the dark light sources 114R and 114L. That is, the stereo camera 110 captures an image of infrared reflected light.

  The stereo camera 110 only needs to be installed at a position where the pupil of the detection subject can be imaged, and the installation position is not limited to the embodiment. Further, the image to be captured (hereinafter referred to as a captured image) may be an image including the pupil of the detection subject, may be an image including only a part of the face including the pupil, and may be added to the face. It may be an image including other than the face.

  The bright light sources 112R and 112L and the dark light sources 114R and 114L are all LED (Light Emitting Diode) light sources. The bright light sources 112R and 112L are arranged around the stereo camera 110. The dark light sources 114R and 114L are disposed at a position between the stereo cameras 110. That is, the distance between the dark light sources 114R and 114L and the stereo camera 110 is longer than the distance between the bright light sources 112R and 112L and the stereo camera 110. More specifically, the right bright light source 112R and the left bright light source 112L are arranged along the circumferential direction around the lenses of the right camera 110R and the left camera 110L, respectively. Thereby, the reflected light from the pupil is uniformly incident on each of the cameras 110R and 110L. On the other hand, the right dark light source 114R and the left dark light source 114L are arranged on the right side and the left side toward the first display unit 101 between the right camera 110R and the left camera 110L, respectively.

  As described above, since the bright light sources 112R and 112L are arranged closer to the stereo camera 110 than the dark light sources 114R and 114L, the stereo camera 110 is irradiated with light from the bright light sources 112R and 112L. When a pupil image of the person to be detected is captured, a brighter (brighter) pupil image is obtained than when captured in a state where light is emitted from the dark light sources 114R and 114L.

  As another example, the arrangement of the bright light sources 112R and 112L and the dark light sources 114R and 114L may be reversed. That is, the right dark light source 114R and the left dark light source 114L are arranged around the right camera 110R and the left camera 110L, respectively, and the right bright light source 112R and the left bright light source 112L are respectively between the right camera 110R and the left camera 110L. You may arrange | position at the right side and the left side toward the display part 101. FIG.

  The drive / IF unit 130 drives each unit included in the stereo camera 110. The drive / IF unit 130 functions as an interface between each unit included in the stereo camera 110 and the control unit 140.

  The control unit 140 is connected to the first display unit 101, the second display unit 102, and the drive / IF unit 130. The control unit 140 controls images to be displayed on the first display unit 101 and the second display unit 102.

  The control unit 140 also detects the pupil of the person to be detected based on the captured image obtained by the stereo camera 110, and detects the line-of-sight direction based on the pupil position.

  The storage unit 160 stores various information. For example, the storage unit 160 stores an image to be displayed on the first display unit 101. The storage unit 160 also stores a threshold value to be described later.

  FIG. 2 is a diagram for explaining a coordinate system used for processing of the line-of-sight direction detection apparatus 100. The first display unit 101 according to the present embodiment is a rectangular screen having a long side in the horizontal direction. As shown in FIG. 2, the line-of-sight direction detection apparatus 100 uses the center position of the screen of the first display unit 101 as the origin, the top and bottom are the Y coordinates (up is the + direction), and the side is the X coordinates (the first display unit). Processing is performed in an XYZ coordinate system in which the right direction is 101 and the depth is the Z coordinate (the front side of the first display unit 101 is the + direction).

  FIG. 3 is a block diagram illustrating an example of detailed functions of each unit illustrated in FIG. 1. The drive / IF unit 130 includes camera IFs 131R and 131L, an LED drive control unit 133, and a speaker drive unit 134. The right camera IF 131R and the left camera IF 131L are connected to the right camera 110R and the left camera 110L, respectively, and drive the right camera 110R and the left camera 110L. The right camera 110R transmits a frame synchronization signal to the left camera 110L and the LED drive control unit 133.

  The LED drive control unit 133 controls light emission of the bright light sources 112R and 112L and the dark light sources 114R and 114L. Specifically, the LED drive control unit 133 causes the right bright light source 112R and the left bright light source 112L to emit light in the first frame at different timings based on the frame synchronization signal. Correspondingly, the right camera 110R captures a face image during irradiation of the right bright light source 112R, and the left camera 110L captures a face image during irradiation of the left bright light source 112L.

  Further, the LED drive control unit 133 causes the right dark light source 114R and the left dark light source 114L to emit light in the second frame with the timing similarly shifted. Correspondingly, the right camera 110R captures a face image during irradiation of the right dark light source 114R, and the left camera 110L captures a face image during irradiation of the left dark light source 114L. Note that images captured by the left and right cameras 110R and 110L are input to the control unit 140 via the camera IFs 131R and 131L.

  Note that, hereinafter, an image captured by the right camera 110R during irradiation of the right bright light source 112R is a right camera bright image, an image captured by the left camera 110L during irradiation of the left bright light source 112L is a left camera bright image, and a right dark light source 114R. An image captured by the right camera 110R during irradiation is referred to as a right camera dark image, and an image captured by the left camera 110L during irradiation of the left dark light source 114L is referred to as a left camera dark image.

  The speaker driving unit 134 drives the speaker 103. The controller 140 controls the visual line direction detection device 100 as a whole. Specifically, the control unit 140 includes a display control unit 141, an image acquisition unit 142, a difference image generation unit 143, a candidate area specification unit 144, an area calculation unit 145, an aspect ratio calculation unit 146, an average A luminance calculation unit 147, a pupil detection unit 148, and a gaze direction detection unit 149 are provided. The display control unit 141 controls display of various types of information on the first display unit 101 and the second display unit 102.

  The image acquisition unit 142 acquires captured images from the camera IFs 131R and 131L. The difference image generation unit 143 calculates a difference value of the luminance of each pixel corresponding to the right camera bright image and the right camera dark image, and generates a right difference image using the difference value as the value of each pixel. Similarly, the difference image generation unit 143 generates a left difference image based on the left camera bright image and the left camera dark image. That is, the difference image generation unit 143 functions as a difference calculation unit. In the line-of-sight direction detection apparatus 100 according to the present embodiment, processing is performed in units of pixels when processing a region included in an image, but the processing unit is not limited to the embodiment. What is necessary is just to divide an image into image areas of an arbitrary size and process values such as luminance in units of image areas.

  The candidate area specifying unit 144 binarizes each pixel value of the right difference image and the left difference image using a preset binarization threshold. The candidate area specifying unit 144 specifies an area where pixels that are equal to or higher than the binarization threshold are continuous as a candidate area. The candidate area specifying unit 144 further assigns consecutive numbers to the specified candidate areas, that is, performs a labeling process. The candidate region is a region that is a candidate for a pupil.

  The area calculation unit 145 calculates the area of the candidate region. The aspect ratio calculation unit 146 calculates the aspect ratio of the candidate area. Specifically, the aspect ratio calculation unit 146 specifies the rightmost pixel position and the leftmost pixel position in the horizontal direction (X-axis direction) of the candidate area, and determines the distance in the X-axis direction between the two pixel positions. Specified as horizontal length. The aspect ratio calculation unit 146 further specifies the uppermost pixel position and the lowermost pixel position in the vertical direction (Y-axis direction) of the candidate region, and sets the distance in the Y-axis direction between the two pixel positions to the vertical direction. Specify as length. The aspect ratio calculation unit 146 calculates a ratio between the specified vertical length and horizontal length as an aspect ratio.

  The average luminance calculation unit 147 calculates the average luminance value of each pixel included in the candidate area, that is, the average luminance. The pupil detection unit 148 detects the pupil of the detected person based on the area, aspect ratio, and average luminance of each candidate region.

  The line-of-sight direction detection unit 149 detects the line-of-sight direction of the detection subject based on the detection result of the pupil detection unit 148. Specifically, the line-of-sight direction detection unit 149 first detects the pupils detected from the right difference image and the left difference image, respectively, using the right camera bright image and the left camera bright image by the stereo vision technique. The position in the three-dimensional coordinate system shown, that is, the pupil position is calculated. The line-of-sight direction detection unit 149 further calculates the position of the corneal reflection of the detected person based on the right camera bright image and the left camera bright image. The line-of-sight direction detection unit 149 further calculates the distance between the pupil cameras, which is the distance between the pupil and the left and right cameras 110R and 110L. The line-of-sight direction detection unit 149 detects the line-of-sight direction of the detected person based on the pupil position, the corneal reflection position, and the inter-pupil camera distance, and further, the display surface of the first display unit 101 and the line-of-sight direction The point of intersection with is detected as the point of interest. For details of the gaze direction detection processing, for example, Japanese Patent No. 4517049 can be referred to.

  By using a bright image, the pupil can be clearly identified as compared to a dark image. Therefore, in this embodiment, the gaze direction detection unit 149 detects the gaze direction using the left and right camera bright images, but as another example, the gaze direction detection unit 149 The line-of-sight direction may be detected using an image.

  FIG. 4 is a flowchart showing the processing of the visual line detection device 100. As shown in FIG. 4, in the line-of-sight detection process, first, the LED drive control unit 133 turns on the left bright light source 112L (step S101). Next, the left camera 110L captures the face image of the detection subject in a state where the left bright light source 112L is irradiated, and the image acquisition unit 142 acquires the left camera bright image obtained by the left camera 110L ( Step S102).

  Next, the LED drive control unit 133 turns on the right bright light source 112R (step S103). Next, the right camera 110R captures the face image of the detection subject in a state where the right bright light source 112R is irradiated, and the image acquisition unit 142 acquires the right camera bright image obtained by the right camera 110R ( Step S104).

  Next, the LED drive control unit 133 turns on the left dark light source 114L (step S105). Next, the left camera 110L captures a face image of the detected person in a state where the left dark light source 114L is irradiated, and the image acquisition unit 142 acquires a left camera dark image obtained by the left camera 110L ( Step S106).

  Next, the LED drive control unit 133 turns on the right dark light source 114R (step S107). Next, the right camera 110R captures the face image of the detection subject in a state where the right dark light source 114R is irradiated, and the image acquisition unit 142 acquires the right camera dark image obtained by the right camera 110R ( Step S108).

  Next, the difference image generation unit 143 calculates a luminance difference between corresponding pixels of the left camera bright image and the left camera dark image obtained by the left camera 110L, and uses the calculated difference value as the value of each pixel. A left difference image is generated. Further, the difference image generation unit 143 calculates the luminance difference between the corresponding pixels of the right camera bright image and the right camera dark image obtained by the right camera 110R, and uses the calculated difference value as the value of each pixel. A difference image is generated (step S109). Next, pupil detection processing using the left difference image and the right difference image is performed by the pupil detection unit 148 and the like (step S110). The pupil detection process will be described later.

  When the pupil is detected, the line-of-sight direction detection unit 149 further uses the left camera bright image and the right camera bright image to calculate the position of corneal reflection from the image near the pupil (step S111). Next, the line-of-sight direction detection unit 149 calculates the inter-pupil camera distance (step S112). The gaze direction detection unit 149 further calculates the pupil position, and detects the gaze direction based on the pupil position, the corneal reflection position, and the inter-pupil camera distance (step S113). If there is no instruction to end the process (step S114, No), the process returns to step S101 and the process is repeated. When there is an instruction to end the process (step S114, Yes), the process is completed.

  FIG. 5 is a flowchart showing detailed processing in the pupil detection processing (step S110). In the pupil detection process (step S110), a process for pupil detection is performed using each of the left difference image and the right difference image. Note that the process is the same regardless of whether the left difference image or the right difference image is used. Hereinafter, in the description of FIG. 5, the left difference image and the right difference image are simply referred to as difference images.

  First, the candidate area specifying unit 144 binarizes the value of each pixel of the difference image using the binarization threshold (step S201). Next, the candidate area specifying unit 144 specifies a continuous pixel area as a candidate area among the pixels determined to be equal to or higher than the binarization threshold by the binarization process. Further, the candidate area specifying unit 144 sequentially assigns numbers (labels) to the specified candidate areas. That is, the candidate area specifying unit 144 labels candidate areas (step S202).

  Next, the candidate area specifying unit 144 selects a labeled candidate area (step S203). In step S203, a candidate area with a smaller number is selected from a plurality of unprocessed candidate areas. Next, the pupil detection unit 148 sequentially performs the following processing on the labeled candidate regions. That is, the pupil detection unit 148 first instructs the area calculation unit 145 to calculate the area of the selected candidate region. In accordance with this instruction, the area calculation unit 145 calculates the area of the selected candidate region (step S204).

  Next, the pupil detection unit 148 compares the calculated area value with an area range that is a preset area value range. When the calculated area value is within the area range (step S205, Yes), the pupil detection unit 148 instructs the aspect ratio calculation unit 146 to calculate the aspect ratio of the selected candidate region. . In accordance with this instruction, the aspect ratio calculation unit 146 calculates the aspect ratio of the selected candidate area (step S206).

  Next, the pupil detection unit 148 compares the calculated aspect ratio value with an aspect ratio range that is a range of preset aspect ratio values. The pupil detection unit 148 calculates the average luminance of the selected candidate region to the average luminance calculation unit 147 when the calculated aspect ratio is a value within the aspect ratio range (Yes in step S207). Instruct. In accordance with this instruction, the average luminance calculation unit 147 calculates the average luminance of the selected candidate area (step S208).

  Next, the pupil detection unit 148 calculates the variation value of the candidate area (step S209). Specifically, the pupil detection unit 148 calculates the standard deviation of the value of each pixel in the candidate area as the variation value. In the present embodiment, the pupil detection unit 148 calculates the standard deviation as the variation value, but the variation value may be a value indicating the degree of variation in the pixel values in the candidate region. The standard deviation is not limited.

  Next, the pupil detection unit 148 compares the calculated variation value with a variation threshold that is a threshold for a preset variation value. Then, when the variation value is equal to or less than the variation threshold value (step S210, Yes), the pupil detection unit 148 determines that the selected candidate region is a pupil (step S211).

  On the other hand, if the area is not within the area range in step S205 (step S205, No), or if the aspect ratio is not within the aspect ratio range in step S207 (step S207, No), the variation value in step S210. Is larger than the variation threshold (No in step S210), the pupil detection unit 148 determines that the selected candidate region is not a pupil (step S212).

  Next, when there is an unprocessed label candidate area (step S213, Yes), the process returns to step S203, and an unprocessed selection candidate is selected (step S203). When there is no unprocessed candidate area (step S213, No), the pupil detection process (step S110) is completed.

  Here, the relationship between the variation value in the difference image and the pupil will be described with reference to FIGS. FIG. 6A is a diagram illustrating a bright camera image captured by either of the left and right cameras 110R and 110L. FIG. 6B is a diagram illustrating a camera dark image captured by either the left or right camera 110R or 110L. FIG. 6C is a diagram illustrating a difference image generated from the camera bright image illustrated in FIG. 6A and the camera dark image illustrated in FIG.

  When the bright light sources 112R and 112L and the dark light sources 114R and 114L are irradiated, the light is reflected not only by the pupil but also by the lens and frame of the glasses. For this reason, as shown in FIGS. 6A and 6B, not only the pupil regions 501 and 503 but also the glasses lenses and the frame regions 502 and 504 have images with higher brightness than the other regions. can get.

  Reflection by the pupil with respect to the bright light sources 112R and 112L is relatively large, and as illustrated in FIG. 6A, the pupil region 501 is a region having a higher luminance than other regions. In addition, the luminance of the pupil region 501 has a substantially constant value. The reflection by the pupil with respect to the dark light sources 114R and 114L is very small compared to the reflection by the pupil with respect to the bright light sources 112R and 112L. For this reason, in the camera dark image shown in FIG. 6B, the luminance of the pupil region 503 is very small compared to the luminance of the region 501. Note that the luminance in the region 503 is a substantially constant value. For this reason, as shown in FIG. 6C, in the difference image, the pupil region 505 is an image having a substantially constant luminance.

  On the other hand, due to the positions of the bright light sources 112R and 112L and the dark light sources 114R and 114L being different, the positions of the high brightness regions 502 and 504 obtained as the reflection by the glasses in the camera bright image and the camera dark image are in the positions. It does not match. Also, the luminance value obtained corresponding to the reflection by the glasses is not a constant value. For this reason, in the differential image as shown in FIG. 6C, the area 502 and the area 504 do not coincide with each other as in the area 506, and the area is smaller than that of the area 502 and the area 504, and the luminance is high. In some cases, a region where the difference in luminance of a part of a region obtained as one continuous region is 0, such as a region indicating, or a region 507 may occur.

  FIGS. 7A and 7B illustrate the luminance corresponding to the reflected light from the pupil and the glasses and the luminance difference value, respectively. As shown in FIG. 7A, in the pupil, the luminance obtained corresponding to the light from the bright light source is uniformly high. The luminance obtained with respect to the light from the dark light source is a uniform value although the luminance is much lower than the luminance obtained corresponding to the luminance from the bright light source. For this reason, a difference image shows a uniform value.

  On the other hand, as shown in FIG. 7-2, the reflection by the glasses is not uniform, and therefore the difference is also a variable value. As shown in FIG. 7-2, in the reflection by the glasses, the central portion may cancel each other, and the difference image may be divided into two regions. Further, in the reflection by the glasses, there is a case where the uniformity at the reflection edge portion is lost and a change with respect to the luminance average is partially increased.

  Thus, the degree of variation in the difference in luminance obtained by the reflection by the pupil and the reflection by the glasses are different. That is, the degree of variation in the difference in the region corresponding to the pupil is smaller than that in the region corresponding to the glasses. Therefore, in the gaze direction detecting apparatus 100 according to the present embodiment, as described above, when the degree of variation of the candidate region is equal to or less than the variation threshold, the candidate region is determined to be a pupil.

  As described above, the line-of-sight direction detection apparatus 100 according to the present embodiment generates a difference image indicating a difference between a camera bright image including a bright pupil and a camera dark image including a dark pupil, and in a high-luminance region in the difference image. Whether or not the pupil is a pupil can be determined based on whether or not the luminance variation value is equal to or less than the variation threshold. That is, the line-of-sight direction detection apparatus 100 can accurately detect the pupil with a small amount of calculation, and can detect the line-of-sight direction based on the pupil.

(Second Embodiment)
The line-of-sight detection device 100 according to the second embodiment uses two candidates from a plurality of candidate areas determined to be pupils in the pupil detection process (step S110) shown in FIG. 5 in order to detect the pupils with higher accuracy. A region is extracted, and it is determined whether or not the combination of the two candidate regions is the left and right pupils of the detection subject.

  FIG. 8 is a flowchart illustrating specific processing of the gaze direction detecting device 100 according to the second embodiment. When the process is completed for all labels in step S213 shown in FIG. 5, the process proceeds to step S231 shown in FIG. 8, and the pupil detection unit 148 extracts a combination of two candidate areas from the candidate areas determined to be pupils in step S211. (Step S231).

  If it is determined in step S211 that only two candidate areas are pupils, the processes in and after step S231 shown in FIG. 8 need only be performed for these two candidate areas. On the other hand, if it is determined in step S211 that three or more candidate regions are pupils, combinations of all two candidate regions that can be taken from the three candidate regions are extracted, and for each combination, the steps after step S231 are performed. Process.

  Next, the pupil detection unit 148 calculates an area ratio (area ratio) between the two candidate regions (step S232). Specifically, the pupil detection unit 148 calculates a ratio of the smaller area with respect to the larger area as the area ratio.

  Next, the pupil detection unit 148 compares the calculated area ratio with a preset area ratio threshold. If the area ratio is less than or equal to the area ratio threshold (step S233, Yes), the pupil detection unit 148 further calculates the ratio of the average luminance (average luminance ratio) between the two candidate regions (step S234). Specifically, the pupil detection unit 148 calculates, as an average luminance ratio, a ratio of the smaller average luminance based on the larger average luminance.

  Next, the pupil detection unit 148 compares the calculated average luminance ratio with a preset average luminance ratio threshold value. When the average luminance ratio is equal to or less than the average luminance ratio threshold (step S235, Yes), the pupil detection unit 148 determines that the two candidate regions are pupils (step S236). In step S233, when the area ratio is larger than the area ratio threshold (step S233, No), and in step S235, when the average luminance ratio is larger than the average luminance ratio threshold (step S235, No), pupil detection is performed. The unit 148 determines that the two candidate regions are not pupils (step S237).

  If there is a combination of two unprocessed candidate areas (step S238, Yes), the process proceeds to step S231. If there is no combination of two unprocessed candidate areas (No at step S238), the process is completed.

  In the present embodiment, the pupil detection unit 148 evaluates the areas of the two candidate areas by comparing the area ratio and the area ratio threshold, but the difference between the areas of the two candidate areas is within a predetermined range. It is only necessary to be able to determine whether or not, and specific processing for that purpose is not limited to the embodiment. For example, when an allowable range is defined based on one of the two areas, and the other area is a value within the allowable range, it may be determined that the area condition for determining a pupil is satisfied. . As another example, it may be determined whether or not a difference value between two areas is equal to or less than a predetermined threshold. The same applies to the average luminance ratio. That is, when an allowable range is defined based on one of the two average luminances and the other average luminance is a value within the allowable range, it is determined that the average luminance condition for determining a pupil is satisfied. May be.

  The remaining configuration and processing of the visual line detection device 100 according to the second embodiment are the same as the configuration and processing of the visual line detection device 100 according to the first embodiment.

  As described above, the present invention has been described using the embodiment, but various changes or improvements can be added to the above embodiment. For example, the present invention may be used for detecting a driver's line of sight in an in-vehicle device or the like, and detecting a line of sight in a device that performs an operation using the line of sight.

DESCRIPTION OF SYMBOLS 100 Gaze direction detection apparatus 110R, 110L Camera 112R, 112L Bright light source 114R, 114L Dark light source 130 Drive / IF part 133 LED drive control part 140 Control part 141 Display control part 142 Image acquisition part 143 Difference image generation part 144 Candidate area | region identification part 145 Image calculation unit 146 Aspect ratio calculation unit 147 Average luminance calculation unit 148 Pupil detection unit 149 Gaze direction detection unit 160 Storage unit

Claims (5)

A first image captured by the imaging unit, the first image including a human pupil irradiated with infrared light from a light source arranged at a first position, and the first image captured by the imaging unit A second image of the person in a state in which infrared light is irradiated from a light source disposed at a second position where the distance from the imaging unit is longer than the first position. An image acquisition unit for acquiring a second image including a pupil;
A difference value for calculating a difference value between each of a plurality of image areas included in the first image and each of a plurality of image areas of the second image corresponding to each of the plurality of image areas of the first image. A calculation unit;
A candidate area specifying unit that compares the difference value with a first threshold value and specifies a plurality of continuous image areas in which a difference value equal to or greater than the first threshold value is calculated as a candidate area of the pupil;
A variation value calculation unit that calculates a variation value indicating a degree of variation in luminance of each image region in the candidate region;
A pupil detection unit that compares the variation value with a second threshold value and detects the candidate region as the position of the pupil of the person when the variation value is equal to or less than the second threshold value. Pupil detection device.   The pupil detection device according to claim 1, wherein the calculation unit calculates a standard deviation of luminance of each image region in the candidate region as the variation value. An average luminance calculating unit for calculating an average value of the luminance of each image area in the candidate area;
The pupil detection unit extracts two different candidate areas in which the variation value is equal to or less than the second threshold, and the average value of one of the two extracted candidate areas is based on the other average value. The pupil detection device according to claim 1, wherein the two candidate regions are detected as positions of two pupils of the person when the value is within a first range. An area calculation unit for calculating the area of the candidate region;
The pupil detection unit is a case where the area of one of the two extracted candidate regions is a value within a second range based on the other area, and one of the two candidate regions extracted The two candidate regions are detected as positions of two pupils of the person when the average value is a value within the first range with the other average value as a reference. 4. The pupil detection device according to 3. A first image captured by the imaging unit, the first image including a human pupil irradiated with infrared light from a light source arranged at a first position, and the first image captured by the imaging unit A second image of the person in a state in which infrared light is irradiated from a light source disposed at a second position where the distance from the imaging unit is longer than the first position. An image acquisition step of acquiring a second image including the pupil;
A difference value for calculating a difference value between each of a plurality of image areas included in the first image and each of a plurality of image areas of the second image corresponding to each of the plurality of image areas of the first image. A calculation process;
A candidate area specifying step of comparing the difference value with a first threshold value and specifying a plurality of consecutive image areas in which a difference value equal to or greater than the first threshold value is calculated as a candidate area of the pupil;
A variation value calculating step of calculating a variation value indicating a degree of variation in luminance of each image region in the candidate region;
A pupil detection step of comparing the variation value with a second threshold value and detecting the candidate region as the position of the pupil of the person when the variation value is equal to or less than the second threshold value. Pupil detection method.