JP2019101974A - Image processing device - Google Patents

Abstract

To provide an image processing device capable of estimating a line of sight of an object person who is photographed by image processing even if the person wears sunglasses.SOLUTION: An image processing device comprises a near infrared LED 12 radiating a near infrared ray with waves remarkably reduced in solar spectrum on the ground surface, an optical band-pass filter 13 which makes the waves that the near infrared LED 12 radiates penetrate therethrough, a camera 14 photographing a face of a person lit by the near infrared LED 12 through the optical band-pass filter 13, and a processing unit 15 performing data processing on a face image photographed by the camera 14 to detect a pupil from the face image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, and more particularly, to a technology that can be used to estimate a gaze that is a feature of an eye based on an image obtained by capturing a face of a person or the like.

  2. Description of the Related Art Conventionally, there is known an image processing apparatus that processes data of an image obtained by photographing a face of a person with a camera to specify an eye feature (for example, Patent Document 1). The technique of Patent Document 1 aims to accurately detect the area of the pupil.

  Further, in the image processing apparatus disclosed in Patent Document 1, the second differentiation unit 41 for obtaining the luminance gradient by differentiating at least the eye area where the eyes of the face image are located in the vertical direction of the eyes and the eye area It is a curve represented by a binarization unit 42 that binarizes a gradient and extracts an edge point, an eye inside point and an eye outside point as end points, and that is represented by the end point and a control point, and that matches the edge point And an eyelid contour specifying unit 44 which specifies the curve to be drawn as a curve representing the outline of the upper eyelid or the lower eyelid.

JP, 2012-190351, A

  In a device that estimates the line of sight of a person by image processing from a moving image of a person's face, it is desirable to be able to detect center coordinates of the iris region or pupil region from the image with high accuracy.

  However, in the method described in Patent Document 1, it is difficult to detect an iris or a pupil from an image when a subject being photographed wears sunglasses. Therefore, it has been difficult to estimate the line of sight of the subject by image processing.

  The present invention has been made in view of the above-described circumstances, and an object thereof is to estimate the line of sight of a person by image processing even when the subject being photographed wears sunglasses. Providing an image processing apparatus capable of

In order to achieve the object described above, an image processing apparatus according to the present invention is characterized by the following (1) to (9).
(1)
A light source that emits near-infrared light of a wavelength at which the solar spectrum at the ground surface significantly decreases;
An optical filter that transmits the wavelength irradiated by the light source;
A camera for capturing a human face illuminated by the light source through the optical filter;
A processing unit that performs data processing on a face image captured by the camera and detects a pupil from the face image;
An image processing apparatus comprising:
(2)
The image processing apparatus according to the above (1), wherein
The processing unit is
Applying a filtering process to the face image to detect an edge in a certain scanning direction to generate an edge image in which the magnitude of the gradient is held as information;
In a state where a circular sampling circle virtually positioned on the surface of a three-dimensional model is projected onto the edge image, the gradient is generated at each position in the edge image corresponding to each of a plurality of points constituting the sampling circle. We extract the size of as a sampling value and use it as the target of likelihood evaluation,
Among the plurality of sampling circles, the sampling circle with the largest likelihood is detected as a pupil,
An image processing apparatus characterized by
(3)
The image processing apparatus according to the above (2), wherein
The image processing apparatus, wherein the processing unit detects a pupil from the face image by using the sampling circle whose radius changes dynamically.
(4)
The image processing apparatus according to the above (3), wherein
The processing unit changes the radius of the sampling circle according to the passage of time.
An image processing apparatus characterized by
(5)
The image processing apparatus according to the above (3), wherein
The processing unit changes the radius of the sampling circle when the likelihood is lower than a certain threshold.
An image processing apparatus characterized by
(6)
The image processing apparatus according to the above (3), wherein
It further comprises an optical sensor that detects visible light,
The processing unit changes the radius of the sampling circle based on the intensity detected by the light sensor.
An image processing apparatus characterized by
(7)
An image processing apparatus according to any one of the above (2) to (6), wherein
The processing unit is
Using a sampling circle in which a plurality of circles with different radii are arranged with the same position as the center point,
In the state where the sampling circle virtually positioned on the surface of the three-dimensional model is projected onto the edge image, the gradient of the gradient is generated at each position in the edge image corresponding to each of a plurality of points forming the sampling circle. The magnitude is extracted as a sampling value, which is the target of likelihood evaluation,
Among the plurality of sampling circles, the sampling circle with the largest likelihood is detected as a pupil,
An image processing apparatus characterized by
(8)
An image processing apparatus according to any one of the above (2) to (7), wherein
The processing unit matches the black circle image simulating the pupil with the binarized image obtained by binarizing the face image as a front stage of projecting the sampling circle onto the edge image, and is the black circle with the largest likelihood. The coordinates on the binarized image where the center of the image is located are extracted as the center position of the pupil, and the radius of the black circle image is extracted as the radius of the pupil.
Projecting the sampling circle in which the radius of the pupil is set in the vicinity of the center position of the pupil on the edge image;
An image processing apparatus characterized by
(9)
The image processing apparatus of the above configuration (8), wherein
The processing unit matches the black circle image whose radius dynamically changes with the binarized image.
An image processing apparatus characterized by

According to the image processing apparatus of the above configuration (1), even if visible light reflected by an object in front of the driver receives natural light (sunlight) is reflected on the surface of the sunglasses, it is blocked by the optical filter and the camera It does not enter. For this reason, it can suppress that an object is reflected in the sunglasses surface in the image image | photographed by the camera. Also, in the image taken by the camera, the difference in density between the pupil and the iris surrounding the pupil appears more prominently than the difference in density between the iris and the sclera (eye) or iris surrounding the iris (eyelid) . Therefore, by setting the detection target as the pupil by image processing, the line of sight of a person can be accurately estimated.
According to the image processing device having the configuration of (2), the pupil area can be identified with high accuracy by detecting the pupil using a particle filter.
According to the image processing device having the configuration of (3), by dynamically changing the diameter of the particle filter, it is possible to set the diameter of the particle filter following the pupil to be reduced or enlarged. Thereby, the central coordinates of the pupil can be detected with high accuracy.
According to the image processing apparatus having the configuration of (4), the diameter of the particle filter can be updated periodically, so that the diameter of the particle filter can be set following the pupil to be reduced or enlarged. Thereby, the central coordinates of the pupil can be detected with high accuracy.
According to the image processing apparatus having the configuration of (5), when the likelihood calculated by the current particle filter is decreased, the diameter of the particle filter is updated to follow the pupil which is reduced or enlarged. The diameter of the filter can be set. Thereby, the central coordinates of the pupil can be detected with high accuracy.
According to the image processing apparatus having the configuration of (6), the diameter of the particle filter is dynamically changed by selecting the diameter of the particle filter according to the magnitude of the signal transmitted from the optical sensor. As described above, by dynamically changing the diameter of the particle filter from the brightness of the surrounding environment, the central coordinates of the pupil can be detected with high accuracy.
According to the image processing device having the configuration of (7), before and after the pupil diameter changes, a large (small) sampling circle of the diameter captures the pupil, and after the pupil diameter changes, the diameter decreases ( A large) sampling circle can capture the pupil. Thus, the central coordinates of the pupil can be detected with high accuracy in response to changes in the pupil diameter.
According to the image processing apparatus having the configuration of (8), by performing a rough search of the pupil using a template matching method, it is possible to narrow down the range in which the particle filter is dispersed in the subsequent processing. Thereby, the area of the pupil can be identified with high accuracy.
According to the image processing apparatus of the configuration of (9), the diameter of the black circle image is dynamically changed, whereby the accuracy of the rough search of the pupil is improved. As a result, when a plurality of particle filters are dispersed, the appearance rate of the particle filter that matches the shape of the pupil is increased, so that the center coordinates of the pupil can be detected with high accuracy.

  According to the image processing apparatus of the present invention, even when the subject being photographed wears sunglasses, the line of sight of the person can be estimated by image processing.

  The present invention has been briefly described above. Furthermore, the details of the present invention will be further clarified by reading the modes for carrying out the invention described below (hereinafter referred to as "embodiments") with reference to the attached drawings. .

FIGS. 1A and 1B are views showing a specific example of an arrangement state in which the image processing apparatus of the present invention is mounted on a vehicle. FIG. 2 is a diagram of the sunlight spectrum on the ground. FIG. 3 is a flowchart showing an outline of a gaze detection algorithm implemented by the image processing apparatus according to the first embodiment of the present invention. FIG. 4 is a diagram showing the relationship between a 3D (three-dimensional) model of an eye and a 2D (two-dimensional) image. FIG. 5 is a specific example of a flow of processing for searching a pupil from an image and data in a pupil detection algorithm implemented by the image processing apparatus according to the first embodiment of the present invention. FIG. 6 is a flowchart for explaining processing for detecting a pupil position using a particle filter. FIG. 7 is a specific example of process data when the process shown in FIG. 6 is performed. FIG. 8 shows a flowchart for dynamically changing the diameter of the particle filter by the image processing apparatus of the first embodiment. FIGS. 9A to 9C are schematic diagrams showing the positional relationship between the particle filter having the highest likelihood and the pupil in the process of the flowchart shown in FIG. FIG. 10 is a view for explaining the shape of a particle filter used in the image processing apparatus of the third embodiment. FIG. 11 is a schematic view in which the particle filter shown in FIG. 10 is dispersed. FIG. 12 is a flowchart of dynamically changing the diameter of the black circle image by the image processing apparatus according to the fourth embodiment.

  Specific embodiments of the image processing apparatus of the present invention will be described below with reference to the drawings.

<Overview of Image Processing Device of the Present Invention>
The specific example of the arrangement | positioning state by which the image processing apparatus of this invention was mounted in the vehicle is shown to Fig.1 (a), (b).
In the example shown in FIGS. 1A and 1B, a near infrared LED (Light Emitting Diode) 12, an optical band pass filter (Band-Pass Filter) 13, a camera 14 and a processing unit 15 are provided in a vehicle interior 10 of a vehicle. Is installed. Specifically, in the meter unit in front of the driver's seat or at the column cover so that the person 14 sitting in the driver's seat, that is, the face including the driver's eyes area 11a can be photographed by the camera 14 A near infrared LED 12, an optical band pass filter 13, and a camera 14 are provided.

  The near-infrared LED 12 is a light source that emits near-infrared light, and is disposed in a meter unit in front of a driver's seat, a column cover, or the like at a position or angle at which the person 11 can be irradiated with near-infrared light. The wavelengths of near infrared light emitted by the near infrared LED 12 are preferably around 940 nm, around 1100 nm, around 1400 nm, and around 1800 nm. The reason why this numerical value is preferable will be described later.

  The optical band pass filter 13 is an optical element that transmits near infrared light emitted by the near infrared LED 12 and blocks light of other wavelengths. The optical band pass filter 13 is disposed in front of the photographing direction of the camera 14 as shown in FIG. 1 (b). The optical band pass filter 13 is disposed in the meter unit in front of the driver's seat or in a column cover or the like at a position or angle where only near infrared light transmitted through the optical band pass filter 13 enters the camera 14.

  The processing unit 15 processes two-dimensional image data obtained by photographing the face of the person 11 by the camera 14, and estimates information indicating the direction of the line of sight 11b from the data of the eye area 11a. The information of the line of sight 11b estimated by the processing unit 15 can be used, as a representative example, for the on-vehicle apparatus to recognize whether the driver driving the vehicle is moving the line of sight for safety confirmation. Of course, such gaze information can be used for various applications. Therefore, it is assumed that the image processing apparatus of the present invention is configured as, for example, the processing unit 15 in FIG. 1 or a part thereof.

  Now, the image processing apparatus of the present invention includes the near infrared LED 12 and the optical band pass filter 13. Specifically, the near infrared LED 12 emits near infrared light, and the optical band pass filter 13 has the same wavelength. Is characterized in that it transmits only near infrared light. In order to explain the feature of the present invention, first, when the subject being photographed wears sunglasses, problems that are potentially present in the conventional method as described in Patent Document 1 will be described.

  From a close look at the image of the subject wearing sunglasses, taken by the conventional method, it was confirmed that an object (landscape) in front of the driver was reflected in the sunglasses. This is because an object in front of the driver receives natural light (sunlight) and reflected visible light is reflected on the surface of the sunglasses and is incident on the camera. If the position of the object reflected in such sunglasses and the position of the driver's eye located at the back of the sunglasses overlap on the image, the accuracy in detecting the driver's iris or pupil is significantly reduced. The inventors rushed.

  Therefore, the present inventors focused on the sunlight spectrum on the ground surface. The sunlight spectrum on the ground is shown in FIG. The sunlight spectrum outside the atmosphere is shown in FIG. 2 for reference. As can be read from FIG. 2, the sunlight spectrum at the ground surface has an irradiance peak at visible light, and the irradiance decreases as the wavelength goes to near infrared light (750 [nm] to 2500 [nm]) I understand that. In particular, it can be seen that, in the near infrared light, the irradiance decreases significantly at wavelengths near 940 nm, 1100 nm, 1400 nm, and 1800 nm. Based on this, if we let only the near-infrared light of the wavelength where the sunlight spectrum on the ground surface falls remarkably be incident on the camera 14, the natural light at this wavelength is low in intensity and the object is a natural light Since the intensity of light received and reflected is also weak, it is possible to suppress the reflection of an object on the surface of the sunglasses in the photographed image. On the other hand, near-infrared light illuminated by the near-infrared LED 12 and reflected on the face of the subject can have sufficient intensity. It is believed that this improves the accuracy of detecting the iris or pupil of the driver. The configuration invented according to this idea is the configuration shown in FIGS. 1 (a) and 1 (b) described above. With the configuration shown in FIGS. 1 (a) and 1 (b), the optical band pass filter 13 is obtained even if visible light reflected by an object in front of the driver receives natural light (sunlight) is reflected on the surface of the sunglasses. And is not incident on the camera 14 in the first place. In addition, at the wavelength that passes through the optical band pass filter 13, the intensity of light that the object receives and reflects natural light is weak. In this way, it is possible to suppress the reflection of an object on the surface of the sunglasses in the image taken by the camera 14. In the embodiment of the present invention, although the object into which the object is reflected in the photographed image is sunglasses, the same phenomenon is also confirmed in glasses. For this reason, according to the present invention, it is possible to suppress the reflection of an object on the surface of the glasses in the photographed image. Therefore, the effects of the present invention can be generalized as follows. That is, when there is a shield that receives and reflects natural light (sunlight) between the camera and the human eye, it is possible to suppress an object from being reflected on the surface of the shield in the captured image.

  The present inventors proceeded to verify the gaze estimation method by image processing using the configurations shown in FIGS. 1 (a) and 1 (b). As a result, the configurations shown in FIGS. 1 (a) and 1 (b) We found that it was suitable. This is because, in an image captured by the camera 14 in which near-infrared light is incident, the difference in density between the pupil and the iris surrounding the pupil is the sclera (half eye) or iris (eyelid) surrounding the iris and the iris. This is because it appears more prominently than the difference in light and shade of. The fact that the difference in density is remarkable leads to the fact that the boundary between the pupil and the iris can be accurately extracted when the edge extraction processing is performed on the image captured by the camera 14. Thus, according to the image processing apparatus of the present invention shown in the configurations shown in FIGS. 1A and 1B, the line of sight of a person can be accurately estimated by setting the detection target as the pupil by image processing. . The outline of the image processing apparatus of the present invention has been described above. Hereinafter, the image processing apparatus according to the first embodiment of the present invention will be described in detail.

<Details of Image Processing Apparatus According to First Embodiment of the Present Invention>
<Overview of gaze detection algorithm>
An outline of a gaze detection algorithm implemented by the image processing apparatus according to the first embodiment of the present invention will be described. FIG. 3 is a flowchart showing an outline of a gaze detection algorithm implemented by the image processing apparatus according to the first embodiment of the present invention. When the computer of the image processing apparatus executes a predetermined program, operations in accordance with the gaze detection algorithm shown in FIG. 3 are sequentially performed.

  The camera 14 constantly captures an image of an area including the face of the person 11 at a constant cycle, and outputs a signal of the image. In step S12, the computer of the image processing apparatus captures one frame of video signal from the camera 14.

  Then, in the next step S13, conversion of the data format of the image, that is, “down process” including gray scale conversion is performed. For example, at each pixel position in one frame, two 8-bit data representing luminance with gradation in the range of “0 to 255” are arranged in the vertical direction and Generate image data of a two-dimensional (2D) array.

In the next step S14, face detection is performed using, for example, the "Viola-Jones method", and an area including a face is extracted as a rectangular area from one frame of two-dimensional image data. That is, a face region is extracted using a detector which is characterized by the face shadow difference and created by learning using "Boosting". The technology of the "Viola-Jones method" is shown, for example, in the following document.
"P. viola and MJJones,""Rapid Object Detection using a Boosted Cascade of Simple Features," IEEE CVPR (2001). "

  In the next step S15, the eye area is detected from the data of the face rectangular area extracted in S14 using, for example, the "Viola-Jones method".

  In the first embodiment, it is assumed that the line of sight is detected by a model-based method using an eyeball 3D model described later, so it is necessary to specify the eyeball center. The eyeball center is estimated in step S16. Here, it is assumed that the center coordinates of the rectangular area of the eye detected in step S15 is the eyeball center. For example, it is also conceivable to use a method of determining the eyeball center based on the positions of the eyes and the corners, and a method of estimating the skeleton from feature points of the face and calculating the eyeball center position simultaneously.

  In step S17, a method of template matching is applied to the data of the rectangular area of the eye detected in S15 to perform a rough pupil search. Specifically, the coordinates of the image center (the center of the black circle) of the black circle image having the largest likelihood are matched with the image obtained by matching the black circle image as a template to the image obtained by binarizing the eye image in which the eye periphery is cut out The center position of the pupil in the image is used, and the radius of the black circle image with the largest likelihood is the radius of the pupil in the eye image. The process in step S17 is performed to approximate the center position and radius of the pupil in the eye image.

  In step S18, using the center position and radius of the pupil searched in S17, the particle filter method is applied to detect the center position and radius of the pupil with higher accuracy. Since the characteristic process of the present invention is included in step S18, the contents will be described in detail later.

  Since the data of the eyeball center coordinates and the coordinates of the center position of the pupil can be obtained for the image of one frame by the processes of steps S13 to S18, the numerical data is output in step S19. The direction of the line of sight can be specified by the eye center coordinates and the coordinates of the center position of the pupil. Also, by repeating the loop processing of steps S11 to S20 until the video output from the camera 14 is interrupted, the gaze detection in real time is realized.

<Description of eyeball 3D model>
The relationship between a 3D (three dimensional) model of the eye and a 2D (two dimensional) image is shown in FIG.
In the eyeball 3D model 30 shown in FIG. 4, the eyeball is a sphere represented by the eyeball center 31 and the radius R. In addition, a circular shaped iris 32 is on the surface of the eyeball, and a circular shaped pupil 33 is in the center of the iris 32.

  Therefore, the direction of the line of sight can be identified as the direction from the eyeball center 31 toward the center of the iris 32 or pupil 33, and the rotation angle yaw (yaw) with respect to the reference direction in the horizontal plane and the rotation angle pitch (pitch) with respect to the vertical reference direction Can be represented by Further, center coordinates of the iris 32 or the pupil 33 can be represented by an eyeball radius R, a yaw, and a pitch based on the eyeball center 31.

  On the other hand, the video captured by the camera 14 represents a specific two-dimensional plane. Therefore, in the case of applying the two-dimensional image 34 obtained by photographing by the camera 14 to the eyeball 3D model 30, it is necessary to perform two-dimensional / three-dimensional mutual conversion. Therefore, conversion is performed using, for example, the following formula.

X =-R x cos (pitch) x sin (yaw) (1)
Y = R × sin (pitch) (2)
X: distance in the x direction from the eyeball center 31 on the two-dimensional image plane Y: distance in the y direction from the eyeball center 31 on the two-dimensional image plane

<Details of Process for Searching for Pupil from Image>
<Flow of processing>
Details of a pupil detection algorithm implemented by the image processing apparatus according to the first embodiment of the present invention will be described. FIG. 5 is a specific example of a flow of processing for searching a pupil from an image and data in a pupil detection algorithm implemented by the image processing apparatus according to the first embodiment of the present invention.
In step S41 of FIG. 5, using the result of step S15 in FIG. 4, the rectangular region of the eye and its periphery is cut out from the two-dimensional image data of the entire one frame, and data D41 is acquired.

  In step S42 of FIG. 5, after the binarized image D42 obtained by binarizing the data D41 acquired in step S41 so that the gray scale for each pixel is only the black / white binary value is generated, this binary value is generated. The template matching in step S17 of FIG. 3 is performed on the digitized image D42. That is, while using a black circle-shaped image D4t similar to the shape of the pupil as a template, while scanning this template on the image of the binarized image D42, an approximate pupil position P41 with similar features is searched for , Identify its position and the radius or diameter of the pupil.

  Further, in step S43 of FIG. 5, the Sobel filter process is performed on the eye data D41 acquired in step S41. Specifically, the eye data D41 is sequentially scanned in the horizontal direction from left to right, and black (gradation value: 0) is output in a portion where there is no luminance change, and the larger the gradient of the luminance change, the more white The edge is detected as it approaches (gradation value: 255). Thus, an edge image D43 of the eye image from which the edge has been extracted is obtained.

  In step S44 in FIG. 5, a particle filter process is performed on the edge image D43 of the eye image obtained as a result of step S43 in the vicinity of the approximate pupil position obtained as a result of step S42.

<Processing of pupil detection by particle filter>
Details of step S44 shown in FIG. 5 are shown in FIG. That is, FIG. 6 shows a flowchart for explaining a process for detecting a pupil position using a particle filter. Further, FIG. 7 shows a specific example of process data when the process shown in FIG. 6 is executed.

By executing the rough search in step S42 on the binarized binarized image data D91 in FIG. 7, a circular pupil area 91 is detected. In step S18a of FIG. 6, the center position of the pupil obtained by the rough search in step S17 or S42 is set to the coordinate system (R, Pp, Py of the eyeball 3D model 30 based on the equations (1) and (2). To obtain an eye rotation angle (Pp, Py) representing the central position of the pupil.
R: eyeball radius Pp: variable representing rotation angle pitch Py: variable representing rotation angle yaw

  Therefore, the position of the circular pupil area 92 obtained by the rough search can be reflected on the eye surface position on the eye 3D model 30A as shown in FIG.

Subsequently, in step S18b of FIG. 6, a plurality of particle filters necessary for finding actual pupil position candidates are dispersed in the vicinity of the pupil center position (Pp, Py) obtained by the rough search. Specifically, for example, 100 sets of sampling circles 93 are created using random numbers. These sampling circles 93 can be reflected on the eye surface position on the eye 3D model 30B as shown in FIG. Each sampling circle 93 is a circular area centered at a position obtained by adding different random number components (Ppd, Pyd) to the central position (Pp, Py) of the pupil.
Ppd: Random number component to be added to rotation angle pitch: Random number component to be added to rotation angle yaw As the radius of each sampling circle 93, the radius of the black circle image used by template matching is applied.

  When creating 100 sets of sampling circles 93, each random component (Ppd, Pyd) is generated so as to add random noise according to the normal distribution. Therefore, sampling circles 93 are randomly formed at various positions near the region 92 of the pupil, as in the eyeball 3D model 30B shown in FIG.

  In step S18c of FIG. 6, each of the 100 sets of sampling circles 93 is projected onto the plane of the edge image D92 (corresponding to the edge image D43 in FIG. 5) shown in FIG. Although the shape of the pupil area 92 and each sampling circle 93 is a circle, each circle on the eyeball 3D model 30 is a plane of a two-dimensional image unless the position of each circle faces the front of the camera 14 When projected onto, it is projected in an inclined state, so it becomes elliptical on a two-dimensional plane. Therefore, for convenience, the sampling circle 93 projected on the two-dimensional plane is referred to as a sampling ellipse 94. The sampling ellipse 94 is composed of a plurality of points equally allocated to each position on the arc.

  In step S18d of FIG. 6, "sampling processing" to be described next is performed. That is, for each sampling ellipse 94, the edge intensity (gradient magnitude) of the edge image of each point (the total number is 120 in this example) constituting the ellipse is added, and the result is the likelihood of one sampling ellipse 94 I assume.

  The processes of steps S18c and S18d are repeated for all 100 sets of sampling ellipses 94. When all the processes are completed, the process proceeds from step S18e to step S18f. In step S18f, one sampling ellipse 94 having the largest likelihood detected in step S18d is identified from the 100 sets of sampling ellipses 94, and the position and shape thereof are detected as the actual position and shape of the pupil. The details of pupil detection using the particle filter in the image processing apparatus according to the first embodiment of the present invention have been described above.

<Details of Image Processing Apparatus of Second Embodiment According to the Present Invention>
<Overview of pupil detection algorithm>
Subsequently, a pupil detection algorithm using a particle filter in an image processing apparatus according to a second embodiment of the present invention will be described. Although the pupil detection algorithm described in the first embodiment does not touch the point at which the radius of the pupil changes in order to help the understanding of the invention, in fact, if the surrounding environment becomes brighter, the pupil shrinks and becomes darker Will expand. In the second embodiment of the present invention to be described later, a pupil detection algorithm capable of coping with changes in pupil radius will be described.

<Example 1: Method of dynamically changing the diameter of particle filter>
In the first embodiment, a method of dynamically changing the diameter of the particle filter following the pupil to be reduced or enlarged focusing on the particle filter dispersed in the eyeball 3D model 30 will be described. In the pupil detection algorithm using the particle filter in the first embodiment, a flow for dynamically changing the diameter of the particle filter is added to the pupil detection algorithm by the image processing apparatus according to the first embodiment. For this reason, the detailed description about the process which overlaps with the pupil detection algorithm by the image processing apparatus of 1st Embodiment is abbreviate | omitted. A flowchart for dynamically changing the diameter of the particle filter by the image processing apparatus of the first embodiment is shown in FIG. Further, FIGS. 9A to 9C are schematic diagrams showing the positional relationship between the particle filter having the highest likelihood and the pupil in the execution of the flowchart shown in FIG.

The particle diameter change processing for dynamically changing the diameter of the particle filter shown in FIG. 8 is performed in parallel with the pupil detection processing shown in FIG. Therefore, in the particle diameter change process shown in FIG. 8, data referred to in the pupil detection process shown in FIG. 6 is used. Further, the particle diameter change process shown in FIG. 8 is executed when the following condition (1) or (2) is satisfied.
(1) When a fixed time has elapsed since the previous particle diameter change processing was performed (2) The likelihoods of the plurality of sampling ellipses calculated in step S18e of the pupil detection processing shown in FIG. 6 are all predetermined threshold values In the following, the particle diameter changing process shown in FIG. 8 will be described in detail.

  If the above condition (1) or (2) is satisfied, a particle filter with a diameter φ of 2 mm is created (step S28a), and the center position of the pupil obtained by rough search of the created particle filter A plurality of particle filters are scattered in the vicinity of Pp, Py) to calculate the likelihood of each of the particle filters (step S 28 b). The process of step S28b is the same as the process of steps S18a to S18e described with reference to FIG. 6, and thus the description thereof will not be repeated. Then, among the plurality of particle filters, the one with the highest likelihood is identified, and the likelihood is stored (step S28c). Thereafter, a particle filter with the diameter φ increased by 2 [mm] is created (step S28 d).

  Subsequently, a plurality of particle filters having a diameter φ of 4 mm are dispersed in the vicinity of the center position (Pp, Py) of the pupil obtained by the rough search to calculate the likelihood of each of the particle filters (step S28b). Then, among the plurality of particle filters, the one with the highest likelihood is identified, and the likelihood is stored (step S28c). Thereafter, a particle filter with the diameter φ further increased by 2 [mm] is created (step S28 d).

  Similarly, a plurality of particle filters having a diameter φ of 6 [mm] are scattered in the vicinity of the center position (Pp, Py) of the pupil obtained by the rough search to calculate the likelihood of each of the particle filters (step S28b). Then, among the plurality of particle filters, the one with the highest likelihood is identified, and the likelihood is stored (step S28c). Thereafter, when the diameter φ reaches 6 [mm] (step S28 e, Yes), the particle filter diameter φ stored in step S 28 c refers to the likelihood in the respective cases of 2, 4 and 6 [mm]. The diameter φ of the particle filter with the highest degree is set as the diameter of the sampling circle when the pupil detection process shown in FIG. 6 is performed next (step S28 f). Thus, the particle diameter change process shown in FIG. 8 is completed. Incidentally, by storing the time when step S28f is processed, the elapsed time for determination of the above-mentioned condition (1) is measured.

  Here, with reference to FIG. 9, the positional relationship between the particle filter having the highest likelihood and the pupil in the case where the diameter φ is 2, 4 and 6 [mm] will be described. Note that FIG. 9 shows the case where the area 92 of the pupil is close to a circle with a diameter φ of 2 [mm]. In this case, as shown in FIG. 9A, when a sampling circle 93 with a diameter φ of 2 mm is dispersed in the vicinity of the pupil area 92, most of the circumference follows the contour of the pupil area 92. The sampling circle 93 located as such is chosen as the one with the highest likelihood. Further, as shown in FIGS. 9B and 9C, when a sampling circle 93 with a diameter φ of 4 mm or 6 mm is dispersed near the pupil area 92, the contour of the pupil area 92 is obtained. A sampling circle 93, which lies along a portion of the circumference, is chosen as the one with the highest likelihood. At this time, when the likelihoods of the sampling circles 93 each having a diameter φ of 2, 4 and 6 [mm] are compared, the sampling circle 93 having a long diameter φ 2 [mm] along the pupil area 92 is the most The likelihood increases. Thus, the diameter of the particle filter is dynamically changed by selecting the diameter φ of the particle filter with the highest likelihood. By setting the diameter of the particle filter following the pupil to be reduced or enlarged in this manner, the central coordinates of the pupil can be detected with high accuracy.

  In the process of step S28f, the diameter φ of the particle filter with the highest likelihood is set as the diameter of the sampling circle when the pupil detection process shown in FIG. 6 is performed next time. At this time, it is determined whether the highest likelihood among particle filters different in diameter φ exceeds a certain threshold, and if it exceeds the diameter φ of the particle filter having the highest likelihood, next time The diameter of the sampling circle may be set when the pupil detection process shown in FIG. 6 is performed. By doing this, it is possible to prevent the particle filter diameter φ from being selected when none of the particle filters having different diameters conform to the shape of the pupil.

  Further, in the first embodiment, the diameter φ is increased by 2 [mm] at step S 28 d in order to help the understanding of the invention, but particles more along the shape of the pupil can be obtained by setting the width to be increased more finely. You can set the filter. Further, instead of increasing the diameter φ in step S28d, the diameter φ may be decreased. In this case, in step S28a, an upper limit value that can be taken by the diameter φ of the particle filter is set.

<Example 2: Method of changing the diameter of the particle filter according to the surrounding environment>
In the first embodiment, the method of setting the diameter of the particle filter by image processing has been described. However, in the second embodiment, attention is focused on the brightness of light inserted into the pupil. In the second embodiment, an optical sensor is connected to the image processing apparatus to input a signal from the optical sensor, and the diameter of the particle filter is reduced when the intensity of the signal is large, that is, when the surrounding environment is bright. If the intensity of the signal is small, that is, if the surrounding environment is dark, the diameter of the particle filter is enlarged. Thus, the diameter of the particle filter is dynamically changed by selecting the diameter φ of the particle filter in accordance with the magnitude of the signal transmitted from the optical sensor. The central coordinates of the pupil can also be detected with high accuracy by dynamically changing the diameter of the particle filter from the brightness of the surrounding environment.

Example 3 Method of Using Multi-Circle Particle Filter
In the third embodiment, paying attention to the particle filter dispersed in the eyeball 3D model 30, the shape of the particle filter that can correspond to the reduced or enlarged pupil will be described. The pupil detection algorithm using the particle filter in the third embodiment is the same as the pupil detection algorithm by the image processing apparatus of the first embodiment except that the shape of the particle filter is different. For this reason, the detailed description about the process which overlaps with the pupil detection algorithm by the image processing apparatus of 1st Embodiment is abbreviate | omitted. The shape of the particle filter used in the image processing apparatus of the third embodiment is shown in FIG. Further, a schematic view in which the particle filter shown in FIG. 10 is dispersed is shown in FIG.

  The particle filter PF illustrated in FIG. 10 is a circular pattern in which three sampling circles C1, C2, and C3 having different radii have the same position as a central point. As shown in FIG. 11, in the particle filter PF, sampling circles C1, C2, and C3 are dispersed in the eyeball 3D model 30, and sampling ellipses D1, D2, and D3 are projected on an edge image D92 on a two-dimensional plane. Note that FIG. 11 is the same as FIG. 7 except for the shapes of the sampling circle and the sampling ellipse, and thus detailed description thereof will be omitted.

  This particle filter PF has a large number of sampling points 201, 202, 203 arranged at equal intervals on the circumference for each sampling circle C1, C2, C3. Further, the sampling points 201, 202, and 203 for each sampling circle C1, C2, and C3 have the same number of sampling points set, and are arranged radially so as to be aligned in the radial direction. In calculating the likelihood of the particle filter PF, the edge intensities (gradient magnitudes) of the edge image at the sampling points 201, 202 and 203 for each sampling circle C1, C2 and C3 are added, and the result is one The likelihood in the particle filter PF.

  When scattering a plurality of particles in the vicinity of the pupil position obtained by the coarse search, three different positions are obtained by adding different random number components (Ppd, Pyd) to the center position (Pp, Py) of the pupil region 92. Each sampling circle C1, C2, C3 is arranged to be a central point common to the circles.

  The particle filter PF composed of these three circles is projected as sampling ellipses D1, D2, and D3 on an edge image D92 on a two-dimensional plane as shown in FIG.

  In the particle filter PF of the third embodiment, it is possible to cope with a change in pupil diameter by means of each circle formed in triple. That is, even if the pupil captured at an image frame at a certain time t1 changes the pupil diameter at an image frame at the next time t2, the sampling circle (C1, C2, or C3) which has captured the pupil at time t1. A sampling circle (either C1, C2, C3) larger or smaller than) can capture the pupil at time t2. As described above, according to the particle filter of the third embodiment, the central coordinates of the pupil can be detected with high accuracy in response to the change of the pupil diameter.

<Example 4: Method of dynamically changing the diameter of the black circle image used for template matching>
In the fourth embodiment, a method of dynamically changing the diameter of a black circle image following a pupil to be reduced or enlarged by focusing on a black circle image which performs matching on an image obtained by binarizing an eye image in which the periphery of the eye is cut out Will be explained. In the pupil detection algorithm using the black circle image in the fourth embodiment, a flow for dynamically changing the diameter of the black circle image is added to the pupil detection algorithm in the first embodiment. For this reason, the detailed description about the process which overlaps with the pupil detection algorithm by the image processing apparatus of Example 1 is omitted. A flowchart for dynamically changing the diameter of the black circle image by the image processing apparatus of the fourth embodiment is shown in FIG.

  The black circle diameter changing process for dynamically changing the diameter of the black circle image shown in FIG. 12 is performed in parallel with the pupil detection process shown in FIG. Therefore, in the black circle diameter changing process shown in FIG. 12, data referred to in the pupil detection process shown in FIG. 6 is used. Further, the black circle diameter changing process shown in FIG. 8 is executed in step S28g after step S28f described in the first embodiment with reference to FIG. Hereinafter, the process of step S28g will be described.

  When the diameter φ of the particle filter with the highest likelihood is set as the diameter of the sampling circle when the pupil detection process shown in FIG. 6 is performed next in step S28 f, then the particle filter with the highest likelihood is the next. The diameter .phi. Is set as the diameter of the black circle image at the next execution of the pupil rough search shown in step S17 of FIG. 3 (step S28 g).

  Thus, the diameter of the black circle image is dynamically changed by selecting the diameter of the black circle image in accordance with the diameter φ of the particle filter with the highest likelihood. By setting the diameter of the black circle image following the pupil to be reduced or enlarged in this manner, the accuracy of the pupil rough search shown in step S17 of FIG. 3 is improved. This means that the center position of the pupil at the time of scattering the particle filter in step S18 of FIG. 3 can be obtained more accurately. As a result, the appearance rate of the particle filter (that is, the particle filter having a higher likelihood) conforming to the shape of the pupil when scattering a plurality of particle filters is increased, so that the center coordinates of the pupil can be detected with high accuracy.

  From another point of view, it can be said that the accuracy of detecting the center coordinates of the pupil can be maintained even if the number of scattering particle filters is reduced. In the first embodiment, the case of scattering 100 sets of sampling circles has been described. However, as the accuracy of the pupil rough search shown in step S17 of FIG. It can be reduced. Since the reduction of the number of sampling circles to be dispersed contributes greatly to the reduction of the calculation amount of the pupil detection processing, the pupil detection processing can be speeded up.

Here, the features of the embodiments of the image processing apparatus according to the present invention described above will be briefly summarized and listed in the following [1] to [9].
[1]
A light source (near infrared LED 12) that emits near infrared light of a wavelength at which the solar spectrum at the ground surface significantly decreases;
An optical filter (optical band pass filter 13) that transmits the wavelength irradiated by the light source (12);
A camera (14) for photographing a human face illuminated by the light source (12) through the optical filter (13);
A processing unit (15) that performs data processing on a face image captured by the camera (14) and detects a pupil from the face image;
An image processing apparatus comprising:
[2]
The image processing apparatus according to the above [1], wherein
The processing unit (15)
The face image is subjected to filtering processing for detecting an edge in a certain scanning direction to generate an edge image (D43, D92) in which the magnitude of the gradient is held as information;
In a state in which a circular sampling circle (93) virtually positioned on the surface of a three-dimensional model is projected onto the edge image (D43, D92), it corresponds to each of a plurality of points constituting the sampling circle (93) The magnitude of the gradient is extracted as a sampling value for each position in the edge image (D43, D92) to be a target of likelihood evaluation,
Among the plurality of sampling circles (93), the sampling circle with the largest likelihood is detected as a pupil
An image processing apparatus characterized by
[3]
An image processing apparatus having the configuration of [2] above,
The image processing apparatus characterized in that the processing unit (15) detects a pupil from the face image using the sampling circle (93) whose radius changes dynamically.
[4]
The image processing apparatus having the configuration of [3] above,
The processing unit changes the radius of the sampling circle (93) according to the passage of time.
An image processing apparatus characterized by
[5]
The image processing apparatus having the configuration of [3] above,
The processing unit changes the radius of the sampling circle (93) when the likelihood is lower than a certain threshold.
An image processing apparatus characterized by
[6]
The image processing apparatus having the configuration of [3] above,
It further comprises an optical sensor that detects visible light,
The processing unit changes the radius of the sampling circle (93) based on the intensity detected by the light sensor.
An image processing apparatus characterized by
[7]
An image processing apparatus according to any one of the above [2] to [6],
The processing unit (15)
Using sampling circles (C1, C2, C3) in which a plurality of circles having different radii are arranged at the same position as a central point,
A plurality of sampling circles (C1, C2, C3) in a state where the sampling circles (C1, C2, C3) virtually positioned on the surface of the three-dimensional model are projected onto the edge image (D92) For each position in the edge image corresponding to each point, the magnitude of the gradient is extracted as a sampling value to be a target of likelihood evaluation,
Among the plurality of sampling circles (C1, C2, C3), the sampling circle with the largest likelihood is detected as a pupil.
An image processing apparatus characterized by
[8]
An image processing apparatus according to any one of the above [2] to [7],
The processing unit (15) simulates a pupil of a binarized image (D42) obtained by binarizing the face image as a previous stage of projecting the sampling circle (93) onto the edge image (D43, D92). The black circle image is matched, and the coordinates on the binarized image at which the center of the black circle image with the highest likelihood is located are extracted as the pupil center position, and the radius of the black circle image is extracted as the pupil radius.
Projecting the sampling circle in which the radius of the pupil is set in the vicinity of the center position of the pupil on the edge image;
An image processing apparatus characterized by
[9]
An image processing apparatus having the configuration of [8] above,
The processing unit matches the black circle image whose radius dynamically changes with the binarized image (D42).
An image processing apparatus characterized by

11 person 12 near infrared LED
13 optical band pass filter 14 camera 15 processing unit 30 eye 3D model 31 eye center 32 iris 33 pupil 93 sampling circle 94 sampling ellipse 201, 202, 203 sampling point C1, C2, C3 sampling circle D1, D2, D3 sampling ellipse

Claims (9)

A light source that emits near-infrared light of a wavelength at which the solar spectrum at the ground surface significantly decreases;
An optical filter that transmits the wavelength irradiated by the light source;
A camera for capturing a human face illuminated by the light source through the optical filter;
A processing unit that performs data processing on a face image captured by the camera and detects a pupil from the face image;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein
The processing unit is
Applying a filtering process to the face image to detect an edge in a certain scanning direction to generate an edge image in which the magnitude of the gradient is held as information;
In a state where a circular sampling circle virtually positioned on the surface of a three-dimensional model is projected onto the edge image, the gradient is generated at each position in the edge image corresponding to each of a plurality of points constituting the sampling circle. We extract the size of as a sampling value and use it as the target of likelihood evaluation,
Among the plurality of sampling circles, the sampling circle with the largest likelihood is detected as a pupil,
An image processing apparatus characterized by The image processing apparatus according to claim 2,
The image processing apparatus, wherein the processing unit detects a pupil from the face image by using the sampling circle whose radius changes dynamically. The image processing apparatus according to claim 3,
The processing unit changes the radius of the sampling circle according to the passage of time.
An image processing apparatus characterized by The image processing apparatus according to claim 3,
The processing unit changes the radius of the sampling circle when the likelihood is lower than a certain threshold.
An image processing apparatus characterized by The image processing apparatus according to claim 3,
It further comprises an optical sensor that detects visible light,
The processing unit changes the radius of the sampling circle based on the intensity detected by the light sensor.
An image processing apparatus characterized by The image processing apparatus according to any one of claims 2 to 6, wherein
The processing unit is
Using a sampling circle in which a plurality of circles with different radii are arranged with the same position as the center point,
In the state where the sampling circle virtually positioned on the surface of the three-dimensional model is projected onto the edge image, the gradient of the gradient is generated at each position in the edge image corresponding to each of a plurality of points forming the sampling circle. The magnitude is extracted as a sampling value, which is the target of likelihood evaluation,
Among the plurality of sampling circles, the sampling circle with the largest likelihood is detected as a pupil,
An image processing apparatus characterized by The image processing apparatus according to any one of claims 2 to 7, wherein
The processing unit matches the black circle image simulating the pupil with the binarized image obtained by binarizing the face image as a front stage of projecting the sampling circle onto the edge image, and is the black circle with the largest likelihood. The coordinates on the binarized image where the center of the image is located are extracted as the center position of the pupil, and the radius of the black circle image is extracted as the radius of the pupil.
Projecting the sampling circle in which the radius of the pupil is set in the vicinity of the center position of the pupil on the edge image;
An image processing apparatus characterized by The image processing apparatus according to claim 8,
The processing unit matches the black circle image whose radius dynamically changes with the binarized image.
An image processing apparatus characterized by