JP6510451B2 - Device, method and program for specifying the pupil area of a person in an image - Google Patents

Description

  The present invention relates to pupil identification technology for identifying a pupil area of a person in an image.

  For example, there is a method of using a change in pupil diameter of a person as one of methods of estimating a person's emotion to a visual stimulus. Patent Document 1 discloses an apparatus for measuring a pupil diameter with high accuracy. However, the configuration described in Patent Document 1 is strongly restricted to the subject, and the apparatus for measuring is complicated and expensive. On the other hand, Non-Patent Document 1 discloses a configuration for specifying a pupil area of a person in an image from image data. Specifically, the pupil region is specified by binarizing the brightness in the vicinity of the pupil.

JP, 2014-79374, A

"Pupil detection from an eye image", [online], [search on March 9, 2016], Internet <URL: http://opencv-code.com/tutorials/pupil-detection-from-an-eye- image />

  According to the configuration of Non-Patent Document 1, the pupil region can be identified with high accuracy when the face is photographed from substantially the front. However, in an image captured from an oblique direction or the like, the pupil region can not be detected with high accuracy due to the reflection of light in the eye or the like.

  The present invention provides a specifying device, a specifying method, and a program that can specify a pupil area in an image with high accuracy regardless of the face orientation.

  According to one aspect of the present invention, the identification device compares the luminance value of each pixel of the image including the eye with the first threshold to determine the region including the pupil and the iris, and identifies the reference position within the determined region. First identifying means for correcting the luminance value of each pixel of the image according to the distance from the reference position, and comparing the second threshold value with the luminance value of each pixel corrected by the correcting means; And second specifying means for specifying the region of the pupil.

  According to the present invention, the pupil region in the image can be specified with high accuracy regardless of the orientation of the face.

The block diagram of the specific apparatus by one Embodiment. The block diagram of the specific apparatus by one Embodiment. The figure which shows the identification result of the image containing an eye, and a pupil area | region. FIG. 6 is a diagram illustrating the relationship between the angle and the aspect ratio according to one embodiment.

  Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. The following embodiment is an exemplification, and the present invention is not limited to the contents of the embodiment. Further, in each of the following drawings, components that are not necessary for the description of the embodiment will be omitted from the drawings.

First Embodiment
FIG. 1 is a block diagram of a specifying device according to the present embodiment. The holding unit 5 holds various information used to specify the pupil area in advance, and other functional blocks of the specific device use the information. Image data corresponding to an image including human eyes as shown in FIG. 3A is input to the center-of-gravity determination unit 1 of the specific device. Note that image data including the entire face of a person is input to a specific device, and a rectangular area including human eyes is determined by using known face recognition technology, and image data of the determined rectangular area is determined by the gravity center determination unit 1 May be input to the

  The center-of-gravity determination unit 1 compares the luminance value of each pixel of the input image data with the first threshold held by the holding unit 5 to determine the pixel corresponding to the pupil and the iris and the other pixels. . Normally, the pupil and the iris part have lower luminance than the other parts of the eye, so that the area corresponding to the pupil and the iris can be detected by appropriately setting the first threshold. The center-of-gravity determination unit 1 obtains the center-of-gravity position of the area corresponding to the pupil and the iris determined by the first threshold, and outputs the center-of-gravity position to the luminance value correction unit 2.

The luminance value correction unit 2 corrects, for each pixel of the image data, the luminance value p in accordance with the distance d from the gravity center position determined by the gravity center determination unit 1. Specifically, the luminance value correction unit 2 obtains the weighting coefficient w based on the distance d. In this example, the weighting factor w is determined by the probability density function f (d) of normal distribution. The weighting factor w = f (d) is scaled to 1 when d is 0. The average μ and the standard deviation σ of the normal distribution are determined in advance and stored in the holding unit 5. The luminance value correction unit 2 obtains the luminance value p ′ after the correction of the luminance value p based on the weight coefficient w according to the following equation (1).
p '= pmax x (1-w) + p x w
= Pmax-(pmax-p) x w (1)
Here, pmax is the maximum value of the luminance value. In equation (1), when the luminance value is the same, the corrected luminance value becomes higher as the weight coefficient w becomes smaller. Here, since the weighting factor w = f (d), the weighting factor w becomes smaller as the distance d becomes larger. Therefore, according to the equation (1), when the pixel of the same luminance value is at the distance d2 larger than the distance d1 and the distance d1, the corrected luminance value of the pixel at the distance d2 is after the correction of the pixel at the distance d1. Higher than the luminance value of. In addition, the corrected luminance value of the pixel having the same distance d (that is, the same weighting factor w) becomes higher in proportion to the luminance value of the original pixel. The luminance value correction unit 2 outputs the pixel value thus corrected to the binarization unit 3.

  The binarization unit 3 binarizes the corrected luminance value of each pixel indicated by the corrected image data, and outputs binarized data. Specifically, 1 is output when the corrected luminance value of the pixel is less than or equal to the second threshold held by the holding unit 5, and 0 is output when the luminance value is greater than the second threshold. The identifying unit 4 identifies a region where the pixel value of the binarized data is 1 as a pupil region.

  FIG. 3 (B) shows a pupil region detected by directly thresholding each pixel value of the image data corresponding to the image of FIG. 3 (A) as described in Non-Patent Document 1, and FIG. As described above, (C) shows the pupil area detected by correcting the luminance value of the pixel according to the distance from the center of gravity of the area including the iris and the pupil, and thresholding the corrected pixel value. In FIGS. 3B and 3C, the white area indicates the detected pupil area. In FIG. 3 (B), the iris area is also detected as a pupil area. On the other hand, in the present embodiment, since the pixel value is corrected by the distance from the center of gravity of the area including the iris and the pupil, the possibility of misdetecting the iris area as the pupil area is reduced, and the pupil area is accurately detected. Can.

  For example, in an image taken from the front of a person, there is a difference between the color value or the luminance value of the pupil and the iris, but in an image taken from an oblique direction of the person, the colors of the pupil and the iris The difference between the values or the luminance values may be small. Furthermore, in an image obtained by photographing a person from an oblique direction, the brightness value near the eyes may be lowered due to the collision of light and a region near the eyes may be erroneously detected as a pupil region. Furthermore, false detection of a shadow due to eyelashes or eyebrows may also occur as a pupil region. In the present embodiment, as described above, by correcting the luminance value to be higher as the distance from the center of gravity of the area including the iris and the pupil becomes larger, the pupil area is accurately adjusted regardless of the photographing direction of the person. It can be detected.

  In the above embodiment, the probability density function of the normal distribution is used to calculate the weighting factor w. However, if the distance d is increased, the weighting factor w may be decreased, and the relationship between the distance d and the weighting factor w Is not limited to the probability density function of normal distribution. In addition, for pixels with the same luminance value, the luminance value after correction may be increased as the distance d increases, and the method of calculating the luminance value after correction is not limited to Formula (1).

  In addition, equation (1) was to multiply pmax by (1-w), multiply p by the weighting coefficient w, and use the sum as the corrected luminance value. However, the configuration is such that the first weighting factor and the second weighting factor are obtained, pmax is multiplied by the second weighting factor, p is multiplied by the first weighting factor, and the sum is the corrected luminance value. It is good. At this time, the first weighting factor becomes smaller as the distance d becomes larger, and the second weighting factor becomes a value larger as the distance d becomes larger.

Second Embodiment
Subsequently, a second embodiment of the present invention will be described focusing on differences from the first embodiment. The configuration of the identification device of this embodiment is the same as that of FIG. However, in the present embodiment, the processing in the luminance value correction unit 2 is different from that in the first embodiment. First, in the present embodiment, the angle D of the direction of the face of the person with respect to the optical axis direction of the camera when the image data to be input to the specific device is acquired is set in the holding unit 5. Note that the angle D is an angle in the horizontal plane, the angle D when the person is facing the front with respect to the camera is 0 degrees, and the angle D when the person is facing right is 90 degrees. Further, the angle D is a positive value regardless of which direction the face of the person is facing. Also, the angle D may be approximately that angle, and the acquisition method is arbitrary. For example, the angle D may be determined from an image corresponding to image data.

  Further, in the present embodiment, information indicating the relationship between the angle and the flattening of the pupil is set in the holding unit 5 in advance. Note that the flattening of the pupil is defined as the ratio to the long side of the difference between the long side and the short side of the smallest square containing the pupil. For example, FIG. 4 is information indicating the relationship between the angle and the flattening of the pupil. In FIG. 4, the reason why the flatness is not 0 when the angle is 0 is that the human eye is facing at a certain angle with respect to the front direction of the face. Therefore, the relationship between the angle and the flattening is prepared for each of the right eye and the left eye. Then, when identifying the pupil region from the right eye, the relationship between the angle to the right eye and the aspect ratio is used, and when identifying the pupil region from the left eye, the relationship between the angle to the left eye and the aspect ratio is used.

From the angle D and the information shown in FIG. 4, the luminance value correction unit 2 obtains the flattening R of the pupil included in the image. Then, the weight coefficient w of the pixel is obtained by the following equation (2). The function fx is a probability density function of a normal distribution of the mean μ and the standard deviation (1-R) σ, and the function fy is a probability density function of a normal distribution of the mean μ and the standard deviation σ.
w = fx (dx) x fy (dy) (2)
Here, dx and dy are the horizontal distance and the vertical distance from the barycentric position of the pixel, respectively. The luminance value correction unit 2 corrects the luminance value according to Expression (1) based on the weighting coefficient w obtained by Expression (2).

  Also in the present embodiment, as in the first embodiment, the luminance value of the pixel having the same luminance value increases as the distance from the center of gravity increases, as in the first embodiment. However, in the present embodiment, the distance is decomposed in the horizontal direction and the vertical direction, and in the horizontal direction, the weighting factor w is adjusted according to the flatness ratio R. Specifically, as the angle D increases, the pupil on the image becomes vertically long, and the aspect ratio R increases. The decrease rate of fx (dx) with respect to the increase of the horizontal distance increases as the aspect ratio R increases. Therefore, for pixels having the same horizontal distance from the position of the center of gravity, the luminance value is corrected to be higher as the aspect ratio R is larger. By correcting the luminance value in this manner, it is possible to detect the pupil region accurately regardless of the direction of the face of the person.

Third Embodiment
Subsequently, a third embodiment of the present invention will be described focusing on differences from the first embodiment. FIG. 2 is a block diagram of the identification device according to the present embodiment. A difference from the first embodiment shown in FIG. 1 is that a gravity center correction unit 6 is provided between the gravity center determination unit 1 and the luminance value correction unit 2. In the present embodiment, the relationship between the angle D and the correction amount of the center of gravity described in the second embodiment is experimentally obtained in advance, and information indicating the relationship is stored in the holding unit 5 in advance. For example, the coefficients a, b, c, d are held in the holding unit 5, and the correction amount .DELTA.x in the horizontal direction and the correction amount .DELTA.y in the vertical direction are calculated by the following equations (3) and (4), respectively. .
Δx = a × D + b (3)
Δy = c × D + d (4)
If the coordinates of the center-of-gravity position determined by the center-of-gravity determination unit 1 are (G1, G2), the center-of-gravity correction unit 6 outputs the corrected center-of-gravity position as (G1 + Δx, G2 + Δy) to the luminance value correction unit 2. Then, the luminance value correction unit 2 corrects the luminance value of each pixel as in the first embodiment, based on the distance d from the corrected gravity center position (G1 + Δx, G2 + Δy).

  The center of gravity determination unit 1 calculates the center of gravity of the entire area including the iris and the pupil, but when the direction of the face changes, the deviation between the center of gravity of the entire area including the iris and the pupil and the center of gravity of the pupil changes Do. In the present embodiment, the center of gravity of the area including the iris and the pupil determined by the center of gravity determining unit 1 is corrected to be closer to the center of gravity of the pupil based on the angle of the face direction. Thus, the pupil region can be detected with high accuracy regardless of the direction of the face of the person. In this embodiment, the position is corrected not only in the horizontal direction (x-axis direction) but also in the vertical direction (y-axis direction), but even if the barycentric position is corrected only in the horizontal direction good.

<Others>
In the second embodiment and the third embodiment, the flat rate R of the pupil is determined from the angle D of the face under the premise that the person is looking at the front of the face direction, but the person is The direction of the eye can also be an angle D, as only the eye is directed to the side with respect to the direction. Specifically, the angle between the direction of the pupil of the eye and the direction of the optical axis of the camera may be an angle D.

  In the first to third embodiments, the identifying unit 4 identifies the pupil region based on the binarized data. Here, the specifying unit 4 may be configured to determine whether or not the specified pupil region is correct. For example, the specifying unit 4 obtains the flatness R based on the information indicating the relationship between the angle D and the flatness described in the third embodiment. Then, the smallest square including the detected pupil area is determined, and the ratio K of the long side to the difference between the long side and the short side is determined. That is, the flattening of the detected pupil region is determined. Then, for example, it is determined whether K is a value within a predetermined range including R. For example, with a predetermined value being ΔR, it is determined whether K is between R-ΔR and R + ΔR. If K is a value within a predetermined range including R, the identifying unit 4 determines that the pupil region has been correctly identified. On the other hand, when K is not within the predetermined range including R, the identifying unit 4 determines that the pupil region has not been correctly identified. Then, when it is determined that at least the pupil region can not be specified correctly, the specifying unit 4 notifies and displays that effect to the user.

  Furthermore, in the above embodiment, the gravity center determination unit 1 determines the gravity center position of the area including the pupil and the iris. However, it suffices to determine pixels near the center of the area including the pupil and the iris, and the present invention is not limited to determining the barycentric position. That is, the luminance value may be corrected based on the distance from the reference position, with the position of the pixel whose difference in distance from each pixel of the region including the pupil and the iris is less than a predetermined value. .

  The specific device according to the present invention can be realized by a program that causes a computer to operate as the above-mentioned specific device. These computer programs are stored in a computer readable storage medium or can be distributed via a network.

  1: center of gravity determination unit, 2: brightness value correction unit, 3: binarization unit, 4: identification unit, 5: holding unit

Claims (11)

First specifying means for determining a region including a pupil and an iris by comparing a luminance value of each pixel of an image including eyes with a first threshold, and specifying a reference position within the determined region;
A correction unit that corrects the luminance value of each pixel of the image in accordance with the distance from the reference position;
Second specifying means for specifying the area of the pupil by comparing the luminance value of each pixel corrected by the correcting means with a second threshold value;
A specific device characterized by comprising:   The identification apparatus according to claim 1, wherein the first identification unit identifies, as the reference position, a barycentric position of a region including the pupil and the iris.   The first specifying means determines the barycentric position of the area including the pupil and the iris, and the position obtained by correcting the barycentric position based on information indicating the relationship between the direction of the eye and the correction amount of the barycentric position is the reference The identification device according to claim 1, wherein the identification device is identified as a position.   The correction means corrects the luminance value such that the luminance value after correction becomes higher as the distance from the reference position becomes larger if the luminance value is the same for pixels other than the pixel having the largest luminance value. The identification device according to any one of claims 1 to 3, characterized in that:   The correction method according to any one of claims 1 to 4, wherein, when the distance from the reference position is the same, the luminance value is corrected such that the luminance value after correction becomes higher as the luminance value is higher. The specific device described in item 1. The correction means obtains a first weighting factor and a second weighting factor from the distance between the pixel and the reference position, and a first value which is a product of the first weighting factor and the luminance value of the pixel. A second value which is the product of the second weighting factor and the maximum value of the luminance value is determined, and the corrected luminance value of the pixel is calculated based on the sum of the first value and the second value. Ask for
The first weighting factor decreases as the distance between the pixel and the reference position increases.
The identification device according to any one of claims 1 to 3, wherein the second weighting factor increases as the distance between a pixel and the reference position increases.   7. The apparatus according to claim 6, wherein the first weighting factor is a value between 0 and 1, and the second weighting factor is a value obtained by subtracting 1 from the first weighting factor. Specific device. The correction means obtains a third weight coefficient from the horizontal distance between the pixel and the reference position and the direction of the eye, and obtains a fourth weight coefficient from the vertical distance between the pixel and the reference position. Determining the first weighting factor by the product of the fourth weighting factor and the weighting factor of
The third weighting factor decreases as the horizontal distance between the pixel and the reference position increases.
The fourth weighting factor decreases as the vertical distance between the pixel and the reference position increases.
The specification according to claim 6 or 7, characterized in that the rate at which the third weighting factor decreases as the horizontal distance increases is such that the direction of the eye is oriented to the side with respect to the image. apparatus.   The second specification means determines the flatness of the specified area of the pupil, and determines whether the calculated flatness is within a predetermined range including a value based on the direction of the eye. The specific device according to any one of claims 1 to 8. An identification method in an apparatus for identifying a pupil area from an image including an eye, comprising:
A first identification step of comparing a luminance value of each pixel of the image with a first threshold to determine a region including a pupil and an iris, and specifying a reference position within the determined region;
Correcting the luminance value of each pixel of the image according to the distance from the reference position;
A second identification step of identifying the area of the pupil by comparing the luminance value of each pixel corrected in the correction step with a second threshold value;
A specific method characterized by including.   A program that causes a computer to function as the specific device according to any one of claims 1 to 9.