WO2011158463A1 - 外光映り込み判定装置、視線検出装置及び外光映り込み判定方法 - Google Patents
外光映り込み判定装置、視線検出装置及び外光映り込み判定方法 Download PDFInfo
- Publication number
- WO2011158463A1 WO2011158463A1 PCT/JP2011/003195 JP2011003195W WO2011158463A1 WO 2011158463 A1 WO2011158463 A1 WO 2011158463A1 JP 2011003195 W JP2011003195 W JP 2011003195W WO 2011158463 A1 WO2011158463 A1 WO 2011158463A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- luminance
- evaluation value
- reflection
- external light
- histogram
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/193—Preprocessing; Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30268—Vehicle interior
Definitions
- the present invention relates to an external light reflection determination device, a line-of-sight detection device, and an external light reflection determination method.
- pupil detection is performed. If this pupil detection is performed while wearing glasses, the pupil may not be detected. This is due to a reflection phenomenon in which external light is reflected on the spectacle lens.
- the degree of external light reflection on the spectacle lens differs depending on the lens material, coating, face posture, external light intensity, wavelength, and the like. Therefore, it is difficult to estimate the reflection state with high accuracy from indirect information other than the reflection itself.
- first external light reflection detection method For such a problem, conventionally, a method for detecting external light reflection on a spectacle lens (first external light reflection detection method) has been proposed (see, for example, Patent Document 1).
- first external light reflection detection method external light reflection on the spectacle lens is detected based on the moving direction of the edge of the eye region.
- the reflection on the glasses of the driver that occurs when the car is moving forward is detected, when the edge moves from the bottom to the top of the image Reflection is detected.
- a method for detecting a high-luminance region of the eye region has been proposed (see, for example, Patent Document 2).
- shooting is performed while switching the projector that irradiates the face, and a high-luminance region that moves in a plurality of obtained images is detected as reflection on the spectacle lens.
- the reflection on the glasses worn by the driver which occurs when the vehicle is moving forward, is assumed.
- the detection accuracy decreases. This is because the edge moving direction cannot be uniquely determined.
- the present invention has been made in view of the above points, and it is an external light projection that can determine the reflection even when the edge is not used and the luminance at the reflection occurrence point in the glasses is low.
- An object of the present invention is to provide an image determination device, a line-of-sight detection device, and an external light reflection determination method.
- the reflection determination apparatus is a reflection determination apparatus that determines external light reflection on spectacles, an image acquisition unit that acquires an eye area image of a user wearing the glasses, A difference histogram is calculated by calculating a difference between the luminance histogram calculating means for calculating a luminance histogram representing the luminance distribution of the eye area image and the two luminance histograms calculated from the two eye area images having different imaging timings. Based on the difference histogram and the weight according to the luminance, the evaluation value calculating means for calculating an evaluation value related to the reflection of external light, and the external value based on the calculated evaluation value. Determining means for determining light reflection.
- the reflection determination method is a reflection determination method for determining external light reflection on glasses, acquiring an eye area image of a user wearing the glasses, A luminance histogram representing a luminance distribution is calculated, and a difference histogram is calculated by calculating a difference between the two luminance histograms calculated from the two eye area images having different imaging timings, and according to the difference histogram and the luminance.
- An evaluation value related to the external light reflection is calculated based on the weight and the external light reflection is determined based on the calculated evaluation value.
- An eye gaze detection apparatus includes an image acquisition unit that acquires an eye region image of a user wearing glasses, a luminance histogram calculation unit that calculates a luminance histogram representing a luminance distribution of the eye region image, and imaging Based on difference histogram calculation means for calculating a difference histogram by taking a difference between the two luminance histograms calculated from the two eye area images having different timings, and the weight based on the difference histogram and the luminance
- the evaluation value calculating means for calculating the evaluation value related to the reflection of external light on the glasses, and the normalized evaluation value obtained by normalizing the calculated evaluation value is subtracted from the predetermined maximum reliability value.
- Reliability calculation means for calculating the reliability of the pupil detection result in consideration of the influence on the pupil detection caused by the external light reflection
- Executes line-of-sight detection processing of the user comprises a gaze detection processing means for outputting the reliability calculated in the reliability calculation unit with line-of-sight detection result
- an external light reflection determination device a line-of-sight detection device, and an external device that can determine reflection even when the edge is not used and the luminance of the reflection occurrence point in the glasses is low.
- a method for determining reflection of light can be provided.
- Block diagram showing the configuration of the eye region detection unit Flow diagram for explaining the operation of the reflection determination device
- the figure which shows the face image which is the target image The figure which serves for processing explanation of the luminance histogram calculation section
- the figure which serves for processing explanation of difference calculation section A figure used to explain weight variations
- FIG. 1 is a block diagram showing a configuration of a reflection determination apparatus 100 according to Embodiment 1 of the present invention.
- the reflection determination device 100 determines whether or not the degree of the reflection phenomenon that occurs when external light is reflected on the glasses exceeds a predetermined level.
- the reflection determination device 100 is provided, for example, in a passenger compartment of an automobile and is used by being connected to a line-of-sight detection device. This line-of-sight detection apparatus executes the process of detecting the line-of-sight direction of the driver only when the reflection determination apparatus 100 determines that reflection is weak. In the following, a case where the reflection determination device 100 is applied to a visual line detection device will be described.
- the reflection determination device 100 includes an eye region image acquisition unit 101, a luminance histogram calculation unit 102, a luminance histogram storage unit 103, a difference calculation unit 104, an evaluation value calculation unit 105, and an evaluation value storage unit. 106 and a reflection determination unit 107.
- the eye area image acquisition unit 101 acquires the eye area image and outputs it to the luminance histogram calculation unit 102.
- the eye area image acquisition unit 101 includes an image input unit 111 and an eye area detection unit 112.
- the image input unit 111 images an imaging target (that is, a person here).
- the target image data is output to the eye area detection unit 112.
- the image input unit 111 is installed in front of the driver's seat, for example, on a car handle or on a dashboard. As a result, the face of the driving driver is photographed by the image input unit 111.
- the eye area detection unit 112 detects an eye area image from the target image received from the image input unit 111.
- the eye region detection unit 112 includes a face detection unit 121, a face part detection unit 122, and an eye region determination unit 123, as shown in FIG.
- the face detection unit 121 detects a face image from the target image received from the image input unit 111, and outputs the face image data to the face part detection unit 122.
- the face part detection unit 122 detects a face part group (that is, the corner of the eye, the eye, etc.) from the face image received from the face detection part 121, and outputs the position coordinates of each face part to the eye region determination part 123.
- a face part group that is, the corner of the eye, the eye, etc.
- the eye area determination unit 123 determines the position and size of the eye area image based on the position coordinates of each face part received from the face part detection unit 122.
- the position and size of the eye area image are output to the luminance histogram calculation section 102 together with the target image output from the image input section 111 as the eye area image detection result. Note that the position and size of the eye region image are calculated for each of the right eye and the left eye.
- the luminance histogram calculation unit 102 calculates the luminance histogram of the eye region from the target image data received from the eye region image acquisition unit 101, and the calculated luminance histogram is stored in the luminance histogram storage unit 103 and the difference calculation unit 104. Output to.
- the luminance histogram storage unit 103 stores the luminance histogram received from the luminance histogram calculation unit 102 in association with the imaging time of the target image used for calculating the luminance histogram.
- the difference calculation unit 104 calculates the difference between the luminance histogram received from the luminance histogram calculation unit 102 and the past luminance histogram stored in the luminance histogram storage unit 103 and outputs the difference to the evaluation value calculation unit 105 as a “difference histogram”. To do.
- the difference calculation unit 104 receives the luminance histogram received this time from the luminance histogram calculation unit 102. And a difference histogram is calculated based on the history of the luminance histogram stored in the luminance histogram storage unit 103. More specifically, the difference histogram is calculated by taking a difference between the current luminance histogram and the past luminance histogram for each bin. This difference histogram is output to the evaluation value calculation unit 105. For example, the difference histogram is calculated by taking a difference between a luminance histogram of a certain frame and a luminance histogram of a frame 10 frames before that frame. The previous 10 frames are examples, and the present invention is not limited to this.
- Evaluation value calculation unit 105 calculates an evaluation value from the difference histogram and weight received from difference calculation unit 104. Specifically, the evaluation value calculation unit 105 calculates the evaluation value by calculating the product of the difference histogram and the weight for each bin and calculating the sum of the calculation results. The weight described above takes a value corresponding to the luminance.
- the evaluation value calculation unit 105 has, for example, a correlation table between luminance and weight, and is associated with the value of each bin of the difference histogram and the luminance of each bin in the correlation table. Multiply by the weight value. Then, the evaluation value calculation unit 105 obtains an evaluation value by adding the multiplication results obtained for each bin for all bins.
- the calculated evaluation value is output to the evaluation value storage unit 106 and the reflection determination unit 107.
- the evaluation value storage unit 106 stores the evaluation value received from the evaluation value calculation unit 105 in association with the imaging time of the target image used for calculating the evaluation value.
- the reflection determination unit 107 determines external light reflection based on the evaluation value calculated by the evaluation value calculation unit 105. This determination is performed based on the evaluation value received this time from the evaluation value calculation unit 105 and the history of evaluation values stored in the evaluation value storage unit 106.
- the reflection determination unit 107 determines that the evaluation value calculated by the evaluation value calculation unit 105 is continuously equal to or greater than a predetermined threshold value a predetermined number of times (that is, always within a predetermined time or more). When the evaluation value is equal to or greater than a predetermined threshold value), it is determined that the reflection that affects the accuracy of the subsequent line-of-sight detection has occurred. When the reflection determination unit 107 determines that reflection has occurred, the line-of-sight detection processing is not performed in the function unit that performs line-of-sight detection in the subsequent stage.
- FIG. 3 is a flowchart for explaining the operation of the reflection determination apparatus 100.
- the flow diagram of FIG. 3 also includes a processing flow in the above-described line-of-sight detection apparatus.
- the processing flow shown in FIG. 3 starts with the start of the shooting operation.
- the shooting operation may be started by a user operation, or may be started with some external signal as a trigger.
- step S201 the image input unit 111 images an imaging target (that is, a person here). Thereby, a target image is acquired.
- the image input unit 111 for example, a digital camera including a CMOS image sensor and a lens is assumed. Accordingly, the PPM (Portable Pix Map File format) image captured by the image input unit 111 is temporarily stored in an image storage unit (not shown) (for example, a memory space of a PC) included in the image input unit 111. Then, the data is output to the eye area detection unit 112 in the PPM format.
- an image storage unit for example, a memory space of a PC
- step S202 the face detection unit 121 detects a face image from the target image received from the image input unit 111.
- FIG. 4 is a diagram illustrating a face image that is a target image.
- the horizontal direction of the image is the X axis and the vertical direction of the image is the Y axis, and one pixel is one coordinate point.
- an image candidate to be a feature (that is, a feature image candidate) is extracted from the input image, and the extracted feature image candidate is compared with a feature image representing a face area prepared in advance.
- a feature image candidate having a high degree of similarity is detected.
- the similarity is obtained, for example, as a reciprocal of the absolute value of the difference between the average face Gabor feature amount acquired in advance and the Gabor feature amount extracted by scanning the input image.
- the face detection unit 121 identifies, as the face image 401, a region having the highest correlation in the image 400 of FIG.
- the face area detection processing may be performed by detecting a skin color area from the image (that is, skin color area detection) or by detecting an ellipse part (that is, ellipse detection). However, it may be performed by using a statistical pattern identification method. In addition, any method may be employed as long as the technique can perform the face detection.
- step S ⁇ b> 203 the face part detection unit 122 detects a face part group (that is, a mouth corner, a corner of the eye, an eye, etc.) from the face image received from the face detection part 121, and outputs the position coordinates of each face part to the eye region determination unit 123. To do.
- the search area for the face part group is the face area 401 specified in step S202.
- FIG. 4 shows a face component group 402.
- a two-dimensional coordinate such as an end point of a face part such as a mouth corner, an eye corner, or an eye or a nostril is detected using a separability filter.
- the learning part learns the correspondence between a plurality of face images and the positions of the facial parts corresponding to the facial images in advance, and the facial part detection unit 122 receives the correspondences when the facial image 401 is input.
- a location with the highest likelihood of the detection may be detected as a face part.
- the face part detection unit 122 may search for a face part from the face image 401 using a standard face part template.
- step S204 the eye region determination unit 123 determines an eye region from the face image received from the face detection unit 121 and the face component group received from the face component detection unit 122.
- a rectangular area 403 including the corners of the eyes and the eyes is determined as the eye area, and the upper left corner coordinates and the lower right corner coordinates of the rectangle are acquired as eye area information.
- step S205 the luminance histogram calculation unit 102 calculates a luminance histogram in the eye region 403 from the face image received from the face detection unit 121 and the eye region information received from the eye region determination unit 123 (see FIG. 5).
- FIG. 5A shows an eye area 403
- FIG. 5B shows a luminance histogram of the eye area 403.
- the brightness histogram calculated here has 16 bins. That is, when the gradation of the face image is 256 gradations, 16 gradations are associated with one bin. Therefore, the luminance histogram calculation unit 102 counts the number of pixels having luminance belonging to each bin in the eye area 403. Note that the number of bins and the number of gradations associated with one bin are examples, and are not limited to this number.
- step S206 the luminance histogram storage unit 103 stores the luminance histogram received from the luminance histogram calculation unit 102 as a past histogram in association with the photographing time of the face image used for the calculation. It should be noted that past information for a time required for the reflection determination described later may be overwritten or deleted.
- step S207 the difference calculation unit 104 calculates the difference between the two luminance histograms as shown in FIG. 6A from the luminance histogram received from the luminance histogram calculation unit 102 and the past histogram received from the luminance histogram storage unit 103.
- the difference histogram is calculated. Specifically, a difference is calculated with an arbitrary bin between the first luminance histogram and the second luminance histogram, and the absolute value of the calculation result becomes the value of the arbitrary bin in the difference histogram. That is, when the luminance histogram shown in FIG. 6B is the first luminance histogram and the luminance histogram shown in FIG. 6C is the second luminance histogram, the difference histogram is as shown in FIG. 6D.
- step S208 the evaluation value calculation unit 105 calculates the product of the difference histogram and the weight for each bin, and calculates the evaluation value by calculating the sum of the calculation results.
- the average luminance of the arbitrary bin is used as the weight used for the arbitrary bin. That is, the center luminance of the rectangle corresponding to the bin shown in FIG. 5 is the weight used for that bin. In addition, the variation of the used weight is demonstrated in detail later.
- the evaluation value is calculated by the following equation (1).
- V means an evaluation value
- B means the average luminance of each bin
- S means the value of each bin in the difference histogram.
- the evaluation value is calculated by calculating the product of the difference histogram and the weight for each bin and calculating the total sum of the calculation results. Since the difference histogram is used for the calculation of the evaluation value, the evaluation level reflects the fluctuation level of the luminance histogram. Further, a weight corresponding to each bin is used to calculate the evaluation value, and the average luminance of each bin is used as the weight. That is, here, the weight is proportional to the luminance (weight variation 1). Therefore, the evaluation value is sensitive to fluctuations in the high-luminance bins, but is less sensitive to fluctuations in the low-luminance bins than the high-luminance bins, but reflects the fluctuations.
- the accuracy of gaze detection is easily affected by the fluctuation level of the luminance histogram. That is, the greater the fluctuation of the luminance histogram, the lower the accuracy of line-of-sight detection. Therefore, by using the evaluation value reflecting the fluctuation level of the luminance histogram as described above, it is possible to accurately determine whether or not the reflection that affects the accuracy of the line-of-sight detection occurs.
- the brightness of an image area in which reflection occurs tends to be higher than the brightness of an image area in which reflection does not occur, but the absolute value of the brightness is not necessarily high.
- the evaluation value that reflects the fluctuation of the low-luminance bin as described above even if the low-luminance reflection occurs, the reflection that affects the accuracy of the line-of-sight detection is generated. Whether or not it has occurred can be determined with high accuracy.
- step S209 the evaluation value storage unit 106 stores the evaluation value received from the evaluation value calculation unit 105 as a past evaluation value in association with the photographing time of the face image used for the calculation. At this time, the past evaluation values over the time necessary for the reflection determination described later may be overwritten or deleted.
- step S ⁇ b> 210 the reflection determination unit 107 determines external light reflection based on the evaluation value calculated by the evaluation value calculation unit 105. Based on the evaluation value calculated by the evaluation value calculation unit 105, the reflection determination unit 107 determines whether or not there is a reflection that affects the accuracy of subsequent line-of-sight detection. This determination is performed based on the evaluation value received this time from the evaluation value calculation unit 105 and the history of evaluation values stored in the evaluation value storage unit 106.
- the reflection determination unit 107 determines that the evaluation value calculated by the evaluation value calculation unit 105 is continuously equal to or greater than a predetermined threshold value a predetermined number of times (that is, always within a predetermined time or more). When the evaluation value is equal to or greater than a predetermined threshold value), it is determined that the reflection that affects the accuracy of the subsequent line-of-sight detection has occurred.
- step S211 the line-of-sight detection unit (not shown) detects the line of sight when the reflection determination unit 107 determines that no reflection has occurred.
- the line-of-sight detection is performed, for example, from a face direction vector that represents the direction of the front direction of the face calculated from the coordinates of the face component group 402, and a line-of-sight direction vector with respect to the front direction of the face calculated from the coordinates of the corners of the eyes, eyes, and pupil. Calculated.
- the face orientation vector is calculated by the following procedure, for example. First, the three-dimensional coordinates of the driver's facial parts group acquired in advance are converted by rotating and translating. Then, the converted three-dimensional coordinates are projected onto the target image used for pupil detection. Then, the rotation / translation parameter that most closely matches the face component group detected in step S203 is calculated. At this time, when the three-dimensional coordinates of the driver's face parts group are acquired in advance, the set of the vector representing the direction in which the driver's face is facing and the vector rotated by the determined rotation parameter is the face orientation. Is a vector.
- the line-of-sight direction vector is calculated, for example, by the following procedure. First, when the face is facing in a predetermined direction, the driver's facial parts group and the three-dimensional coordinates of the pupil center when looking in the same direction as the face direction are stored in advance. The detection of the center of the pupil is performed, for example, by taking the center of gravity of a pixel having a predetermined luminance or less in the eye region. Next, from the detected three-dimensional coordinates of the pupil, a position moved by a predetermined distance in the direction opposite to the line-of-sight direction is calculated as the eyeball center position.
- the predetermined distance is suitably about 12 mm, which is the radius of a general adult eyeball, but is not limited to the above value, and an arbitrary value may be used.
- the three-dimensional coordinates of the eyeball center at the time of detection are obtained using the face rotation / translation parameters acquired at the time of calculating the face orientation vector.
- a search is made as to where the detected pupil center is on the sphere.
- a vector connecting the center of the eyeball and the searched point on the sphere is calculated as the line-of-sight direction.
- step S212 an end determination is performed.
- the end determination may be performed by manually inputting an end command, or may be performed by the determination apparatus 100 by reflecting an external signal as a trigger.
- step S212 If it is determined in step S212 that the process is to end, the process in FIG. 3 ends.
- FIG. 7 is a diagram for explaining weight variation.
- FIG. 7A shows the correlation between the luminance and weight of variation 1.
- FIG. 7B shows the correlation between the luminance and weight of variation 2.
- FIG. 7C shows the correlation between the luminance and weight of variation 3.
- FIG. 7D shows the correlation between the luminance and weight of variation 4.
- the weight value is zero in the low luminance region, and the weight value increases in proportion to the luminance in the high luminance region excluding the low luminance region.
- the weight of variation 2 is suitable when the entire eye area is very bright and low-luminance reflection does not clearly occur. By using this weight, it is possible to prevent the evaluation value from being affected by a low-luminance portion (such as eyelashes).
- the weighting curve of variation 3 is an S-shaped curve.
- the weight of variation 3 is suitable for the case where misjudgment frequently occurs because the whole eye area is bright and the difference in low luminance is large. By using this weight, it is possible to increase the weight when the luminance is high, and to decrease the weight when the luminance is low, thereby reducing erroneous determination.
- the luminance histogram calculation unit 102 calculates the luminance histogram representing the luminance distribution of the eye area image
- the difference calculation unit 104 has different imaging timings.
- a difference histogram is calculated by taking a difference between the two luminance histograms calculated from the two eye area images, and the evaluation value calculation unit 105 calculates the external light based on the difference histogram and the weight according to the luminance.
- An evaluation value related to the reflection is calculated, and the reflection determination unit 107 determines external light reflection based on the calculated evaluation value.
- the second embodiment relates to a reflection determination apparatus that calculates an evaluation value in the same manner as the reflection determination apparatus 100 according to the first embodiment, and calculates the reliability of a pupil detection result or the like based on the calculated evaluation value.
- the first embodiment it has been described that pupil detection is not performed when the evaluation value continuously exceeds a predetermined value. However, even when pupil detection is unstable, there are cases where a pupil detection result is desired along with information that the reliability of the detection result is low. Thus, Embodiment 2 provides reliability information such as pupil detection results together with pupil detection results.
- FIG. 8 shows a configuration of the reflection determination apparatus 800 according to Embodiment 2 of the present invention.
- components having the same functions as the components of the reflection determination apparatus 100 according to the first embodiment are denoted by the same reference numerals, and description thereof is omitted.
- the reflection determination apparatus 800 includes a reliability calculation unit 801.
- the reliability calculation unit 801 normalizes the evaluation value input from the evaluation value calculation unit 105, and subtracts the normalized evaluation value obtained thereby from the maximum value of reliability, thereby detecting pupil detection. Calculate reliability.
- the calculated pupil detection reliability is output to a gaze detection unit (not shown) that performs gaze detection in the gaze detection device. Then, the line-of-sight detection unit (not shown) outputs the pupil detection reliability together with the line-of-sight detection result.
- the pupil detection reliability deteriorates when the degree of reflection is high because pupil detection becomes difficult, and when the degree of reflection is low, pupil detection becomes easy.
- V n is a value that takes a value between 0 and 1.
- V n is, for example, a value obtained by dividing V by the theoretical maximum value of V or the empirical maximum value. When the value obtained by dividing V by the empirical maximum value of V is 1 or more, V n is 1.
- the pupil detection reliability in the present embodiment is the reliability related to the pupil detection result in consideration of the influence on the pupil detection caused by the external light reflection of the glasses. That is, the reliability of the pupil detection result in consideration of the influence on the pupil detection due to other causes, such as instability of pupil detection due to insufficient illumination, is not included in the pupil detection reliability in the present embodiment.
- the reliability is calculated for each cause or phenomenon, and the reliability of the final pupil detection result is calculated by summing or summing them. It may be calculated.
- the reliability calculation unit 801 normalizes the evaluation value calculated by the evaluation value calculation unit 105, and The reliability of pupil detection is calculated by subtracting the normalized evaluation value thus obtained from the maximum value of reliability.
- a line-of-sight detection unit (not shown) detects the user's line of sight and outputs pupil detection reliability together with the line-of-sight detection result.
- each functional block used in each of the above embodiments is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them. Although referred to as LSI here, it may be referred to as IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.
- the method of circuit integration is not limited to LSI, and implementation with a dedicated circuit or a general-purpose processor is also possible.
- An FPGA Field Programmable Gate Array
- a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.
- the external light reflection determination device, the line-of-sight detection device, and the external light reflection determination method of the present invention determine the reflection even when the edge is not used and the luminance of the reflection occurrence point in the glasses is low. Useful as something that can be done.
Abstract
Description
[映り込み判定装置100の構成]
図1は、本発明の実施の形態1に係る映り込み判定装置100の構成を示すブロック図である。
以上の構成を有する映り込み判定装置100の動作について説明する。図3は、映り込み判定装置100の動作説明に供するフロー図である。図3のフロー図には、上記した視線検出装置における処理フローも含まれている。
重みのバリエーションとしては、上記したバリエーション1の他に、次のようなバリエーションが考えられる。図7は、重みのバリエーションの説明に供する図である。図7Aには、バリエーション1の輝度と重みの相関関係が示されている。図7Bには、バリエーション2の輝度と重みの相関関係が示されている。図7Cには、バリエーション3の輝度と重みの相関関係が示されている。図7Dには、バリエーション4の輝度と重みの相関関係が示されている。
バリエーション2の重み曲線は、低輝度領域で重み値がゼロであり、低輝度領域を除く高輝度領域で重み値が輝度に比例して増加する。このバリエーション2の重みは、目領域全体がかなり明るく、低輝度な映りこみが明らかに発生しない場合に適している。この重みを用いることにより、評価値が低輝度部分(まつげなど)の影響を受けないようにすることができる。
バリエーション3の重み曲線は、S字曲線となる。このバリエーション3の重みは、目領域全体がなり明るいにもかかわらず、低輝度での差分が大きいため誤判断が頻発する場合に適している。この重みを用いることにより、輝度が高い場合の重みをより大きく、輝度が低い場合をより小さくすることができるので、誤判断を軽減することができる。
バリエーション4の重み曲線は、低輝度領域で重み値が一定であり、中輝度領域で重み値が輝度に比例して増加し、高輝度領域で重み値が一定となる。この重みを用いることにより、目領域全体が暗く低輝度な映り込みが発生しやすい場合に、低輝度の重みが一定以下にならないようにすることができる。これと同時に、高輝度な部分は映り込みである可能性が高いので、一定以上輝度が高い場合には重みを最大にして評価値が高くなりやすくすることができる。
実施の形態2は、実施の形態1に係る映り込み判定装置100と同様に評価値を算出し、算出された評価値に基づいて、瞳孔検出結果等の信頼度を算出する映り込み判定装置に関する。実施の形態1では、評価値が連続して所定の値を超えた場合には瞳孔検出を行わないとして説明した。しかしながら、瞳孔検出が不安定な場合でも、検出結果の信頼性が低いという情報と共に、瞳孔検出結果が欲しい場合もある。そこで、実施の形態2は、瞳孔検出結果と共に、瞳孔検出結果等の信頼度情報を提供するものである。
上記各実施の形態では、本発明をハードウェアで構成する場合を例にとって説明したが、本発明はソフトウェアで実現することも可能である。
101 目領域画像取得部
102 輝度ヒストグラム算出部
103 輝度ヒストグラム記憶部
104 差分算出部
105 評価値算出部
106 評価値記憶部
107 映り込み判定部
111 画像入力部
112 目領域検出部
121 顔検出部
122 顔部品検出部
123 目領域決定部
801 信頼度算出部
Claims (6)
- 眼鏡への外光映り込みを判定する映り込み判定装置であって、
前記眼鏡を掛けているユーザの目領域画像を取得する画像取得手段と、
前記目領域画像の輝度分布を表す輝度ヒストグラムを算出する輝度ヒストグラム算出手段と、
撮像タイミングの異なる2つの前記目領域画像から算出された2つの前記輝度ヒストグラム間で差を取ることにより、差分ヒストグラムを算出する差分ヒストグラム算出手段と、
前記差分ヒストグラムと輝度に応じた重みとに基づいて、前記外光映り込みに関する評価値を算出する評価値算出手段と、
前記算出された評価値に基づいて、前記外光映り込みを判定する判定手段と、
を具備する映り込み判定装置。 - 前記評価値算出手段は、前記差分ヒストグラムの各ビンの値と前記各ビンの輝度に応じた重みとを乗算し、乗算結果を全てのビンで加算することにより、前記評価値を算出する、
請求項1に記載の映り込み判定装置。 - 前記判定手段は、前記評価値算出手段にて算出された評価値が連続して所定回数だけ所定の閾値以上の値となる場合に、前記外光映り込みが発生していると判定する、
請求項1に記載の映り込み判定装置。 - 請求項3に記載の映り込み判定装置と、
前記判定手段で前記外光映り込みが発生していると判定されない場合に、前記ユーザの視線検出処理を実行する視線検出処理手段と、
を具備する視線検出装置。 - 眼鏡を掛けているユーザの目領域画像を取得する画像取得手段と、
前記目領域画像の輝度分布を表す輝度ヒストグラムを算出する輝度ヒストグラム算出手段と、
撮像タイミングの異なる2つの前記目領域画像から算出された2つの前記輝度ヒストグラム間で差を取ることにより、差分ヒストグラムを算出する差分ヒストグラム算出手段と、
前記差分ヒストグラムと輝度に応じた重みとに基づいて、前記眼鏡への外光映り込みに関する評価値を算出する評価値算出手段と、
前記算出された評価値を正規化することにより得られる正規化評価値を、所定の信頼度最高値から減算することにより、前記外光映り込みに起因する瞳孔検出への影響を考慮した瞳孔検出結果に関する信頼度を算出する信頼度算出手段と、
前記ユーザの視線検出処理を実行し、視線検出結果と共に前記信頼度算出手段において算出された前記信頼度を出力する視線検出処理手段と、
を具備する視線検出装置。 - 眼鏡への外光映り込みを判定する映り込み判定方法であって、
前記眼鏡を掛けているユーザの目領域画像を取得し、
前記目領域画像の輝度分布を表す輝度ヒストグラムを算出し、
撮像タイミングの異なる2つの前記目領域画像から算出された2つの前記輝度ヒストグラム間で差を取ることにより、差分ヒストグラムを算出し、
前記差分ヒストグラムと輝度に応じた重みとに基づいて、前記外光映り込みに関する評価値を算出し、
前記算出された評価値に基づいて、前記外光映り込みを判定する、
映り込み判定方法。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011535804A JP5661043B2 (ja) | 2010-06-17 | 2011-06-07 | 外光映り込み判定装置、視線検出装置及び外光映り込み判定方法 |
CN201180003330.0A CN102473282B (zh) | 2010-06-17 | 2011-06-07 | 外部光映入判定装置、视线检测装置及外部光映入判定方法 |
EP11795371.1A EP2584525B1 (en) | 2010-06-17 | 2011-06-07 | Determining ambient light reflection |
US13/390,169 US8659751B2 (en) | 2010-06-17 | 2011-06-07 | External light glare assessment device, line of sight detection device and external light glare assessment method |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2010-138354 | 2010-06-17 | ||
JP2010138354 | 2010-06-17 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2011158463A1 true WO2011158463A1 (ja) | 2011-12-22 |
Family
ID=45347879
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2011/003195 WO2011158463A1 (ja) | 2010-06-17 | 2011-06-07 | 外光映り込み判定装置、視線検出装置及び外光映り込み判定方法 |
Country Status (5)
Country | Link |
---|---|
US (1) | US8659751B2 (ja) |
EP (1) | EP2584525B1 (ja) |
JP (1) | JP5661043B2 (ja) |
CN (1) | CN102473282B (ja) |
WO (1) | WO2011158463A1 (ja) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9524446B2 (en) | 2013-12-09 | 2016-12-20 | Fujitsu Limited | Image processing device and image processing method |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5529660B2 (ja) * | 2010-07-20 | 2014-06-25 | パナソニック株式会社 | 瞳孔検出装置及び瞳孔検出方法 |
GB2495324B (en) | 2011-10-07 | 2018-05-30 | Irisguard Inc | Security improvements for Iris recognition systems |
GB2495323B (en) * | 2011-10-07 | 2018-05-30 | Irisguard Inc | Improvements for iris recognition systems |
KR101874494B1 (ko) * | 2011-11-25 | 2018-07-06 | 삼성전자주식회사 | 특징점의 삼차원 위치 계산 장치 및 방법 |
EP4201304A1 (en) * | 2012-10-24 | 2023-06-28 | Nidek Co., Ltd. | Ophthalmic analysis apparatus |
JP6157165B2 (ja) * | 2013-03-22 | 2017-07-05 | キヤノン株式会社 | 視線検出装置及び撮像装置 |
US10789693B2 (en) * | 2017-01-05 | 2020-09-29 | Perfect Corp. | System and method for performing pre-processing for blending images |
CN109670389B (zh) * | 2017-10-16 | 2023-04-07 | 富士通株式会社 | 评价人脸图像中的光照条件的方法和设备 |
CN108198180B (zh) * | 2018-01-10 | 2019-11-26 | 南通大学 | 一种图像亮度值变化原因的判定方法 |
KR20210073135A (ko) * | 2019-12-10 | 2021-06-18 | 삼성전자주식회사 | 눈 복원 기반의 눈 추적 방법 및 장치 |
JP2021114111A (ja) * | 2020-01-17 | 2021-08-05 | Necソリューションイノベータ株式会社 | 撮像支援装置、撮像支援方法、及びプログラム |
JP6956985B1 (ja) * | 2020-12-22 | 2021-11-02 | 株式会社スワローインキュベート | 目検出方法、目検出装置及び目検出プログラム |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002352229A (ja) | 2001-05-30 | 2002-12-06 | Mitsubishi Electric Corp | 顔部位検出装置 |
JP2006318374A (ja) * | 2005-05-16 | 2006-11-24 | Matsushita Electric Ind Co Ltd | 眼鏡判別装置および認証装置ならびに眼鏡判別方法 |
JP2010138354A (ja) | 2008-12-15 | 2010-06-24 | Asahi Kasei Chemicals Corp | 樹脂組成物及びこれを用いた成形体 |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4917480A (en) * | 1985-01-18 | 1990-04-17 | Kabushiki Kaisha Topcon | Eye refractive power measuring apparatus |
EP0611286B1 (en) * | 1990-12-14 | 1998-07-08 | Humphrey Engineering, Inc. | Method and apparatus for controlling perceived brightness using a time varying shutter |
US5305012A (en) * | 1992-04-15 | 1994-04-19 | Reveo, Inc. | Intelligent electro-optical system and method for automatic glare reduction |
EP1947608A3 (en) * | 1999-02-05 | 2012-04-11 | Samsung Electronics Co., Ltd. | Digital video processing method and apparatus thereof |
AUPR872301A0 (en) * | 2001-11-08 | 2001-11-29 | Sleep Diagnostics Pty Ltd | Alertness monitor |
CN100391232C (zh) * | 2005-01-07 | 2008-05-28 | 智辉研发股份有限公司 | 数字影像闪光场景检测与消除方法 |
US7430365B2 (en) * | 2005-03-31 | 2008-09-30 | Avago Technologies Ecbu (Singapore) Pte Ltd. | Safe eye detection |
US7578593B2 (en) * | 2006-06-01 | 2009-08-25 | Delphi Technologies, Inc. | Eye monitoring method with glare spot shifting |
JP5045212B2 (ja) * | 2007-04-25 | 2012-10-10 | 株式会社デンソー | 顔画像撮像装置 |
JP4883013B2 (ja) | 2008-01-17 | 2012-02-22 | トヨタ自動車株式会社 | 顔画像処理装置 |
CN101281730A (zh) * | 2008-03-20 | 2008-10-08 | 青岛海信电器股份有限公司 | 液晶显示方法 |
-
2011
- 2011-06-07 WO PCT/JP2011/003195 patent/WO2011158463A1/ja active Application Filing
- 2011-06-07 CN CN201180003330.0A patent/CN102473282B/zh active Active
- 2011-06-07 JP JP2011535804A patent/JP5661043B2/ja active Active
- 2011-06-07 EP EP11795371.1A patent/EP2584525B1/en active Active
- 2011-06-07 US US13/390,169 patent/US8659751B2/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002352229A (ja) | 2001-05-30 | 2002-12-06 | Mitsubishi Electric Corp | 顔部位検出装置 |
JP2006318374A (ja) * | 2005-05-16 | 2006-11-24 | Matsushita Electric Ind Co Ltd | 眼鏡判別装置および認証装置ならびに眼鏡判別方法 |
JP2010138354A (ja) | 2008-12-15 | 2010-06-24 | Asahi Kasei Chemicals Corp | 樹脂組成物及びこれを用いた成形体 |
Non-Patent Citations (1)
Title |
---|
See also references of EP2584525A4 |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9524446B2 (en) | 2013-12-09 | 2016-12-20 | Fujitsu Limited | Image processing device and image processing method |
Also Published As
Publication number | Publication date |
---|---|
JPWO2011158463A1 (ja) | 2013-08-19 |
EP2584525B1 (en) | 2015-04-01 |
JP5661043B2 (ja) | 2015-01-28 |
CN102473282B (zh) | 2015-01-14 |
US8659751B2 (en) | 2014-02-25 |
US20120170027A1 (en) | 2012-07-05 |
EP2584525A1 (en) | 2013-04-24 |
CN102473282A (zh) | 2012-05-23 |
EP2584525A4 (en) | 2013-04-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5661043B2 (ja) | 外光映り込み判定装置、視線検出装置及び外光映り込み判定方法 | |
US10304164B2 (en) | Image processing apparatus, image processing method, and storage medium for performing lighting processing for image data | |
US9733703B2 (en) | System and method for on-axis eye gaze tracking | |
JP5694161B2 (ja) | 瞳孔検出装置及び瞳孔検出方法 | |
JP4307496B2 (ja) | 顔部位検出装置及びプログラム | |
JP5529660B2 (ja) | 瞳孔検出装置及び瞳孔検出方法 | |
JP5538160B2 (ja) | 瞳孔検出装置及び瞳孔検出方法 | |
WO2018000020A1 (en) | Systems and methods for performing eye gaze tracking | |
US8154591B2 (en) | Eyelid opening level determination device and computer readable medium storing computer program thereof | |
US10311583B2 (en) | Eye motion detection method, program, program storage medium, and eye motion detection device | |
JP2003150942A (ja) | 目位置追跡方法 | |
US11308321B2 (en) | Method and system for 3D cornea position estimation | |
JP2012068948A (ja) | 顔属性推定装置およびその方法 | |
JP2005296382A (ja) | 視線検出装置 | |
JP5128454B2 (ja) | 瞼検出装置、瞼検出方法及びプログラム | |
CN112041783A (zh) | 曝光时间控制 | |
US11156831B2 (en) | Eye-tracking system and method for pupil detection, associated systems and computer programs | |
KR20190097494A (ko) | 전자 장치 및 그의 제어 방법 | |
CN115731258A (zh) | 运动目标识别的方法以及拍摄设备 | |
KR20130123316A (ko) | 사용자의 얼굴 분석 결과에 따라 이동 단말을 제어하는 장치 및 방법 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201180003330.0 Country of ref document: CN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2011535804 Country of ref document: JP |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11795371 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13390169 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2011795371 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |