KR102466084B1 - Image-based pupil detection method - Google Patents

Abstract

The present invention relates to an image-based pupil detection method, and more particularly, can improve the reliability of pupil measurement by accurately detecting pupils even when pupils are distorted through reflected light, eyelid occlusion, etc. in an eyeball image input through a camera. It relates to an image-based pupil detection method.

Description

Image-based pupil detection method {Image-based pupil detection method}

The present invention relates to an image-based pupil detection method, and more particularly, can improve the reliability of pupil measurement by accurately detecting pupils even when pupils are distorted through reflected light, eyelid occlusion, etc. in an eyeball image input through a camera. It relates to an image-based pupil detection method.

The pupil is a circular empty space located at the center of the iris, and controls the amount of light reaching the retina to maximize visual perception and reduces chromatic aberration and spherical aberration by increasing the depth of focus.

Recently, pupils have been used for gaze tracking, pupil inspection, user authentication, etc. to track a user's gaze, and pupil measurement technology capable of automatically detecting and measuring pupils is required to be used in these fields.

Conventionally, in order to detect the pupil, a method of detecting the boundary between the iris and the pupil and then searching for the pupil has been used, but this method has a problem of low accuracy, and high-resolution images have been used to detect the pupil, but high-resolution When using an image of , processing time increases exponentially, resulting in poor utilization.

In addition, the current pupil detection technology has a problem in that measurement accuracy is lowered due to reflected light reflected in the eye, eyelid occlusion, and rapid pupil size change.

KR10-2013275 B1 "Pupil detection device and pupil detection method" KR10-2015-0081680 A "Pupil detection method, computer readable storage medium recording the method and pupil detection device KR10-2014-0106926 A "Pupil detection device and pupil detection method"

The present invention has been made to solve the above problems, and an object of the present invention is to improve the accuracy of pupil measurement by accurately detecting the pupil in the eyeball image even in the case of reflected light reflected in the eye, eyelid occlusion, and rapid pupil size change, An object of the present invention is to provide an image-based pupil detection method that can be used for tracking, pupil inspection, and user authentication, and can also reduce processing time for pupil detection of high-resolution images.

In order to achieve the above object, the present invention comprises the steps of generating a filter bank including binarized images in which circles of different sizes are generated; Receiving an eye image captured by a camera; extracting an expected pupil position (hereinafter referred to as "expected pupil area") by calculating the eyeball image with the binary images of the filter bank; and determining a final pupil position (hereinafter referred to as "actual pupil area") by extracting boundary regions having an elliptical shape from the expected pupil region and comparing them with a preset condition among the boundary regions. It provides an image-based pupil detection method characterized in that.

In a preferred embodiment, the step of generating the filter bank: generating a plurality of image backgrounds having various image sizes; projecting circles of various sizes based on the center coordinates of the image backgrounds; and a first filter bank including images obtained by binarizing a pixel value of an area corresponding to the inside of the circle projected onto the image backgrounds to “0” and a pixel value of other areas to “255” and projection of the image backgrounds. generating a second filter bank including binarized images in which a pixel value inside the circle is set to "255" and a pixel value of the other area is set to "0".

In a preferred embodiment, the step of extracting the expected pupil region: generates first convolutional images by performing a convolutional operation on the eyeball image with binarized images of the first filter bank, respectively, and converting the eyeball image into the first filter bank. Generating second convolutional images by performing a convolution operation with binarized images of two filter banks, respectively; inverting the second convolutional images; extracting multiplication images by multiplying the convolution images by a first convolution image using a binarized image of the same size and an inverted second convolution image in the image generating step; The pixel coordinates (hereinafter referred to as "effective coordinates") of the multiplication image (hereinafter referred to as "effective multiplication image") having the highest brightness value among the multiplication images and the binarized image used to extract the effective multiplication image extracting the size of (hereinafter referred to as “effective binarized image”); and setting pixel coordinates corresponding to the effective coordinates in the eyeball image as center coordinates of the effective binarized image, and extracting an image size of the effective binarized image as an expected pupil area from the eyeball image.

In a preferred embodiment, the step of extracting the actual pupil area includes: extracting a contour image and a gradient image of the extracted expected pupil area; detecting boundary regions having an elliptical shape from the contour image; counting the number of pixels having a gradient value equal to or greater than a predetermined value at pixel coordinates of the gradient image corresponding to the boundary areas; extracting a boundary area having the largest number of counted pixels; and determining whether the extracted boundary region corresponds to a preset condition, wherein the boundary region is extracted as an actual pupil region when the boundary region meets the preset condition, and the boundary regions are detected when the boundary region is not suitable. Repeat from step 1.

In a preferred embodiment, the gradient image includes a vertical gradient image differentiated in a vertical direction and a horizontal gradient image differentiated in a horizontal direction.

In a preferred embodiment, in the suitability determination step, the ratio of the major axis and the minor axis of the border region is less than or equal to a set ratio, the size is within a predetermined range, and the change in center coordinates and size of the border region within a certain frame is within an error range. If it falls within the range, it is judged to be suitable.

In addition, the present invention further provides a computer program stored in a medium for performing the image-based pupil detection method in combination with a computer.

The present invention has the following excellent effects.

According to the image-based pupil detection method of the present invention, a predicted pupil region is primarily detected from an input eyeball image using a filter bank including binarized images onto which circles of various sizes are projected, and then a contour image of the expected pupil region is obtained. Since the final pupil position is detected through the gradient image and the pupil, the pupil can be accurately detected regardless of not only the reflected light but also the irregular shape of the pupil, thereby improving the reliability of the pupil measurement.

In addition, according to the image-based pupil detection method of the present invention, since an expected pupil region is primarily detected and then an actual pupil region is extracted, unnecessary calculation work and erroneous detection can be minimized. .

1 is a flowchart of an image-based pupil detection method according to an embodiment of the present invention;
2 is a flowchart for explaining a process of extracting an expected pupil area according to an embodiment of the present invention;
3 is a schematic diagram for explaining a process of extracting an expected pupil area according to an embodiment of the present invention;
4 is a flowchart for explaining a process of determining an actual pupil area according to an embodiment of the present invention;
5 is a schematic diagram illustrating a process of determining an actual pupil area according to an embodiment of the present invention.

The terms used in the present invention have been selected from general terms that are currently widely used as much as possible, but in certain cases, there are terms arbitrarily selected by the person who came out. In this case, the meaning described or used in the detailed description of the invention is not a simple name of the term. should be taken into account to understand its meaning.

Hereinafter, the technical configuration of the present invention will be described in detail with reference to preferred embodiments shown in the accompanying drawings.

However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Like reference numbers indicate like elements throughout the specification.

1 is a flowchart of an image-based pupil detection method according to an embodiment of the present invention, FIG. 2 is a flowchart for explaining a process of extracting an expected pupil area according to an embodiment of the present invention, and FIG. 3 is an embodiment of the present invention. 4 is a flowchart for explaining a process of determining an actual pupil area according to an embodiment of the present invention, and FIG. 5 is a schematic diagram for explaining a process of extracting an expected pupil area according to an embodiment of the present invention It is a schematic diagram for explaining the judgment process.

Referring to FIGS. 1 to 5 , the image-based pupil detection method according to an embodiment of the present invention is a method capable of accurately detecting a pupil position by receiving an eyeball image captured through a camera.

In addition, the image-based pupil detection method according to an embodiment of the present invention is performed by a computer, and a computer program for performing the image-based pupil detection method by operating the computer is stored in the computer.

In addition, the computer is a computer in a broad sense that includes not only a general personal computer, but also a server computer accessible through a communication network, a cloud system, a smart device such as a smart phone and a tablet PC, and an embedded system embedded in a security camera.

In addition, the computer program may be stored and provided in a separate recording medium, and the recording medium may be specially designed and configured for the present invention or may be known and usable to those skilled in the field of computer software. .

For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CDs and DVDs, magneto-optical recording media capable of both magnetic and optical recording, ROM, RAM, and flash memory. etc., alone or in combination, may be a hardware device specially configured to store and execute program instructions.

In addition, the computer program may be a program composed of program instructions, local data files, local data structures, etc. alone or in combination, and may be executed by a computer using an interpreter or the like as well as machine code generated by a compiler. It can be a program written in high-level language code

In addition, the image-based pupil detection method according to an embodiment of the present invention largely includes generating a filter bank (S1000), receiving an eyeball image (S2000), and predicting a pupil position from the eyeball image (hereinafter, "expected It includes extracting a pupil area" (referred to as "pupil area") (S3000) and determining a final pupil position (hereinafter referred to as "actual pupil area") in the predicted pupil area (S4000).

In the image-based pupil detection method, first, a filter bank is generated (S1000).

The filter bank of the present invention means a set of binarized images necessary to extract the expected pupil area, which will be described below. To generate the filter bank, first, a plurality of image backgrounds having various sizes are generated (S1100 ).

Next, one circle is projected and generated for each image background based on the center coordinates of the plurality of image backgrounds (S1200).

At this time, circles of different sizes may be formed for each image background, and the sizes are not limited.

Next, a filter bank is generated by extracting and collecting binarized images of image backgrounds onto which circles are projected (S1300).

In detail, contrary to the first filter bank including images in which the pixel value of the area corresponding to the inside of the circle projected on the image background is binarized to "0" and the pixel value of the other area to "255", the A second filter bank including binarized images with a pixel value of an area corresponding to the inside of a circle projected on the image background to “255” and a pixel value of the other area to “0” is generated.

That is, the filter bank has two filter banks, and in each filter bank, there are binarized images having various image sizes as well as circle sizes, and an inner region of the circle has a value of “0” or “255”.

Next, after generating the filter bank, an eyeball image captured by a camera is received (S2000).

The camera may be a CAM connected to a computer, a smart device, a medical camera, a head mount, and the like, and the eyeball image may be extracted from a user's face detected through the camera through a separate eyeball detection algorithm.

Next, an expected pupil area is extracted by calculating the input eye image with the binarized images of the filter bank (S3000).

In order to extract the expected pupil region, first, a convolutional operation is performed on the eyeball image and binarized images of the first filter bank to generate first convolutional images, and the eyeball image is converted into first convolutional images of the second filter bank. Second convolutional images are generated by performing a convolution operation with each of the binarized images (S3100).

Therefore, first convolution images in which the characteristics of the pupil position are highlighted are generated in the generated first filter bank, and second convolution images in which the characteristics of the eye region other than the pupil position are highlighted are generated in the second filter bank.

On the other hand, before performing the convolution operation, it is preferable to perform a padding operation on the eyeball image in order to output an image size equal to that of the eyeball image as a result of the convolutional computation.

Next, the second convolution images are inverted, and the first convolution images and the second convolution images are multiplied to extract multiplication images (S3200).

In detail, in the step of generating the convolution image (S3100), a first convolution image using a binarized image having the same image size and an inverted second convolution image are multiplied by each other.

On the other hand, by inverting the second convolution image, the existing dark part becomes brighter and the bright part becomes darker, so that the pupil area features are highlighted. In this state, when multiplied with the first convolution images, the pupil It has a characteristic that the area appears brighter than other areas of the eyeball.

Next, in order to extract the pixel coordinates (hereinafter referred to as "effective coordinates") of the multiplication image (hereinafter referred to as "effective multiplication image") having the highest pixel value among the extracted multiplication images and the effective multiplication image The image size of the used binarized image (hereinafter, referred to as "an effective binarized image") is extracted (S3300).

Next, pixel coordinates corresponding to the effective coordinates in the eyeball image are set as center coordinates of the effective binarized image, and an image size of the effective binarized image is extracted as an expected pupil area from the eyeball image (S3400).

After the expected pupil area is extracted, an actual pupil area is determined (S4000).

In detail, an edge image and a gradient image of the expected pupil area are extracted (S4100).

Here, it is preferable to use the Canny algorithm among various contour detection algorithms for the contour image.

In addition, a morphology operation may be performed on the contour image to remove noise such as unnecessary circles of a predetermined size and short lines in the image.

Also, the gradient image includes a vertical gradient image obtained by differentiating the expected pupil area in a vertical direction and a horizontal gradient image obtained by differentiating the expected pupil area in a horizontal direction, respectively.

In addition, the gradient image may be extracted using a Sobel algorithm, a Prewitt algorithm, a ROberts algorithm, a Laplacian algorithm, or the like.

Next, boundary regions having an elliptical shape are detected from the contour image (S4200).

Here, the algorithm for detecting the elliptical shape may use various known techniques such as an algorithm based on hub transformation, and the algorithm for detecting the ellipse is not limited.

Next, the number of pixels having a gradient value equal to or greater than a predetermined value is counted from the pixel coordinates of the gradient image corresponding to the boundary regions (S4300), and the boundary region having the largest number of counted pixels is extracted (S4400).

Next, an actual pupil area is detected by determining whether the extracted boundary area corresponds to a predetermined condition (S4500).

Here, a predetermined condition for determining the suitability of the extracted boundary area is as follows.

1) The ratio between the major axis and the minor axis of the extracted boundary area corresponds to a preset ratio or less

2) The size of the extracted boundary area is within the preset range

3) Changes in center coordinates and size of the boundary area extracted within a certain frame fall within a preset error range

That is, it is determined whether the extracted border area corresponds to a predetermined condition (S4510), and if all conditions are met, it is determined that it is suitable, and the extracted border area is extracted as an actual pupil area (S4520).

If any of the preset conditions are not met, the process of detecting the boundary areas is repeatedly performed.

Therefore, the image-based pupil detection method of the present invention first detects an expected pupil area and performs an operation to find an actual pupil area from the detected expected pupil area, thereby minimizing unnecessary calculation and erroneous detection.

In addition, after detecting an elliptical shape in the contour image in the expected pupil area, the location corresponding to the actual pupil area is detected using the size and ratio of the ellipse shape, the number of pixels containing a gradient value above a certain value, etc. However, since the pupil can be accurately detected regardless of the pupil's irregular shape, the reliability of the pupil measurement can be improved.

In addition, when the size and state of the pupil are measured based on the actual pupil area detected through the image-based pupil detection method of the present invention, the pupil detection accuracy is high, and high reliability of the pupil measurement result can be obtained.

As described above, the present invention has been shown and described with preferred embodiments, but is not limited to the above embodiments, and to those skilled in the art within the scope of not departing from the spirit of the present invention Various changes and modifications will be possible.

Claims (7)

generating a filter bank including binarized images in which circles of different sizes are generated;
Receiving an eye image captured by a camera;
extracting an expected pupil position (hereinafter referred to as "expected pupil area") by calculating the eyeball image with the binary images of the filter bank; and
extracting boundary regions having an elliptical shape from the expected pupil region, and determining a final pupil position (hereinafter referred to as "actual pupil region") by comparing with a predetermined condition among the boundary regions; Image-based pupil detection method characterized by.
According to claim 1,
Generating the filter bank:
generating a plurality of image backgrounds having various image sizes;
projecting circles of various sizes based on the center coordinates of the image backgrounds; and
A first filter bank including images in which the pixel value of the area corresponding to the inside of the circle projected on the image background is binarized to "0" and the pixel value of the other area to "255", and the image background is projected generating a second filter bank including images in which the pixel value inside the circle is set to "255" and the pixel value of the other area is binarized to "0"; pupil detection method
According to claim 2,
Extracting the expected pupil area:
First convolutional images are generated by performing a convolution operation on the eye image with binarized images of the first filter bank, and a convolutional operation is performed on the eyeball image and binarized images of the second filter bank, respectively, to obtain a convolutional product. generating 2 convolutional images;
inverting the second convolutional images;
extracting multiplication images by multiplying the convolution images by a first convolution image using a binarized image of the same size and an inverted second convolution image in the image generating step;
The pixel coordinates (hereinafter referred to as "effective coordinates") of the multiplication image (hereinafter referred to as "effective multiplication image") having the highest brightness value among the multiplication images and the binarized image used to extract the effective multiplication image extracting the size of (hereinafter referred to as “effective binarized image”); and
and setting pixel coordinates corresponding to the effective coordinates in the eyeball image as center coordinates of the effective binarized image, and extracting an image size of the effective binarized image as an expected pupil area from the eyeball image. An image-based pupil detection method.
According to claim 3,
Extracting the real pupil area:
extracting a contour image and a gradient image of the extracted expected pupil area;
detecting boundary regions having an elliptical shape from the contour image;
counting the number of pixels having a gradient value equal to or greater than a predetermined value at pixel coordinates of the gradient image corresponding to the boundary areas;
extracting a boundary area having the largest number of counted pixels; and
Including; determining whether the extracted boundary area corresponds to a predetermined condition;
An image-based pupil detection method characterized in that if the extracted boundary region meets a predetermined condition, it is extracted as an actual pupil region, and if it is not suitable, the step of detecting the boundary region is repeatedly performed.
According to claim 4,
The gradient image includes a vertical gradient image differentiated in a vertical direction and a horizontal gradient image differentiated in a horizontal direction.
According to claim 4,
The suitability determination step is performed when the ratio between the major axis and the minor axis of the extracted boundary region is less than or equal to a set ratio, the size is within a preset range, and the change in center coordinates and size of the extracted boundary region within a certain frame falls within the error range. Image-based pupil detection method characterized by judging by suitability
A computer program stored in a medium for performing the image-based pupil detection method of any one of claims 1 to 6 in combination with a computer.

Images