KR102225002B1 - Deep neural network based object detecting apparatus and method - Google Patents

Abstract

Disclosed are an eye blink detection apparatus and a method for detecting eye blinking capable of efficiently and reliably detecting eye blinking. An eye blink detection method according to an embodiment of the present invention includes: detecting an eye area in an input image frame based on a support vector machine; And determining an eye blinking state within the eye area based on the convolutional neural network model.

Description

DEEP NEURAL NETWORK BASED OBJECT DETECTING APPARATUS AND METHOD}

The present invention relates to an apparatus and method for detecting objects based on a deep neural network, and more particularly, to an apparatus for detecting eye blinking and a method for detecting eye blinking that can efficiently and reliably detect eye blinking.

Eye blink detection or eye tracking algorithms have various applications in a mobile environment (eg, a smartphone platform). For example, an eye blink detection algorithm can be used to prevent spoofing in a face recognition system. As another example, the eye blink detection systems may be used to advise smartphone users about their eye blink habits so as to alleviate computer vision disorders that many smartphone users suffer from.

There have been various studies on eye tracking and eye blink detection algorithms based on video input. However, conventional techniques have been developed for a desktop environment, and may not operate properly in a mobile/smart phone environment due to the limitation of computational resources of the smartphone and/or frequent change of the position of the eye due to the user's movement in the input video frame. .

The present invention is to provide an apparatus for detecting eye blinking, a method for detecting eye blinking, and a recording medium capable of efficiently and reliably detecting eye blinking.

In addition, the present invention is to provide an eye blink detection apparatus, an eye blink detection method, and a recording medium capable of detecting eye blink with high reliability using limited computational resources of a mobile terminal such as a smart phone.

The problem to be solved by the present invention is not limited to the problems mentioned above. Other technical problems not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

An eye blink detection method according to an aspect of the present invention includes: detecting an eye area in an input image frame based on a support vector machine; And determining an eye blinking state within the eye area based on the convolutional neural network model.

In the detecting of the eye area, the eye area may be detected based on the support vector machine and edge direction histogram features.

In determining the eye blinking state, the eye blinking state may be determined based on a LeNet-5 convolutional neural network model.

In determining the eye blinking state, any one of a closed eye state, an open eye state, and a background state for the eye area may be determined.

The eye blink detection method may further include setting a search area for eye detection in the input image frame.

In the detecting of the eye area, the eye area may be detected in the search area based on the support vector machine.

In the step of setting the search area, the search area may be set based on a direction measurement value measured by the mobile terminal that has obtained the input image frame.

The eye blink detection method includes, based on eye position information on images acquired by the mobile terminal and direction information of the mobile terminal at points when the images are acquired, the eye position information by regression analysis. It may further include determining a regression equation representing a relationship between the directional information and the direction information.

The setting of the search area may include estimating the eye position by applying the direction measurement value to the regression equation; And setting the search area around the estimated eye position.

The size of the search area may be set based on deviation information for the regression equation.

The detecting of the eye area may include scanning the search area with sliding windows having a plurality of different sizes; Calculating scores representing a similarity to an eye shape for the sliding windows based on the support vector machine; And detecting the eye position in the search area based on the scores.

In the detecting of the eye area, the eye area may be detected based on a sliding window having a smallest size among at least one sliding window having a score equal to or greater than a set reference value.

According to another aspect of the present invention, a computer-readable recording medium in which a program for executing the eye blink detection method is recorded is provided.

According to another aspect of the present invention, there is provided a detection unit configured to detect an eye area in an input image frame based on a support vector machine; And a determination unit configured to determine an eye blink state within the eye region based on a convolutional neural network model.

The detection unit may detect the eye region based on features of the support vector machine and edge direction histogram.

The determination unit may determine an eye blinking state from among a closed eye state, an open eye state, and a background state based on the LeNet-5 convolutional neural network model.

The eye blink detection apparatus may further include a setting unit configured to set a search area for eye detection in the input image frame.

The setting unit may be configured to set the search area based on a direction measurement value measured by the mobile terminal that has obtained the input image frame.

The detection unit may be configured to detect the eye area in the search area based on the support vector machine.

The setting unit, based on the eye position information on the images acquired by the mobile terminal and the direction information of the mobile terminal at the time points at which the images are acquired, the eye position information and the It can be configured to determine a regression equation representing the relationship between the direction information.

The setting unit estimates the eye position by applying the direction measurement value to the regression equation; Determining the size of the search area based on the deviation information for the regression equation; And it may be configured to set the search area based on the estimated eye position and the size of the search area.

The detection unit scans the search area with a plurality of sliding windows having different sizes; Calculating scores representing a similarity to an eye shape for the sliding windows based on the support vector machine; Detecting the eye position in the search area based on the scores; In addition, it may be configured to detect the eye region based on a sliding window having a smallest size among at least one sliding window having a score equal to or greater than a set reference value.

According to an embodiment of the present invention, an apparatus for detecting eye blinking, a method for detecting eye blinking, and a recording medium capable of efficiently and reliably detecting eye blinking are provided.

In addition, according to an embodiment of the present invention, an eye blink detection apparatus, an eye blink detection method, and a recording medium capable of detecting eye blink with high reliability using limited computational resources of a mobile terminal such as a smart phone are provided.

The effects of the present invention are not limited to the above-described effects. Effects not mentioned will be clearly understood by those of ordinary skill in the art from the present specification and the accompanying drawings.

1 is a flowchart of an eye blink detection method according to an embodiment of the present invention.
2 is a block diagram of an apparatus for detecting eye blinking according to an embodiment of the present invention.
3 is an exemplary diagram illustrating setting a search area and detecting an eye area in an input image frame according to an embodiment of the present invention.
4 is a state transition diagram of an eye blink detection method according to an embodiment of the present invention.
5 is a flowchart of step S10 of FIG. 1.
6 is a configuration diagram of a setting unit configuring an apparatus for detecting blinking eyes according to an exemplary embodiment of the present invention.
7 is a flowchart of step S20 of FIG. 1.
8 is a diagram showing the accuracy of a method for detecting eye blinking according to an embodiment of the present invention.

Other advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments to be described later in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and the present invention is only defined by the scope of the claims. Even if not defined, all terms (including technical or scientific terms) used herein have the same meaning as commonly accepted by universal technology in the prior art to which this invention belongs. A general description of known configurations may be omitted so as not to obscure the subject matter of the present invention. In the drawings of the present invention, the same reference numerals are used as much as possible for the same or corresponding configurations. In order to help the understanding of the present invention, some configurations in the drawings may be somewhat exaggerated or reduced.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise", "have" or "have" are intended to designate the existence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification. It is to be understood that the possibility of the presence or addition of other features, numbers, steps, actions, components, parts, or combinations thereof, or any further features, is not preliminarily excluded.

The'~ unit' used throughout this specification is a unit that processes at least one function or operation, and may mean, for example, a hardware component such as software, FPGA, or ASIC. However,'~ part' does not mean limited to software or hardware. The'~ unit' may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors.

As an example,'~ unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, and subs. Routines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and functions provided by the'~ units' may be performed separately by a plurality of elements and the'~ units', or may be integrated with other additional elements.

According to an embodiment of the present invention, there is provided an eye blink detection apparatus, an eye blink detection method, and a recording medium that can be used for efficient detection of eye blink in a smartphone platform with limited resources.

Real-time processing is required for reliable eye blink detection, and a throughput of 10 fps or more is required. Therefore, it may be difficult to apply the convolutional neural network model in the case of the eye blink detection algorithm compared to the case where it is applied to the eye tracking algorithm.

An eye blink detection method and an eye blink detection apparatus according to an embodiment of the present invention are provided by a hybrid learning system including a support vector machine and a convolutional neural network model. It detects eye-blinking.

According to an embodiment of the present invention, by a heterogeneous hybrid learning system using a support vector machine and a convolutional neural network, it is possible to efficiently and reliably detect the user's blinking, and use limited computational resources of a mobile terminal such as a smartphone. Thus, the blinking of an eye can be detected.

The eye blink detection method and the eye blink detection apparatus according to an embodiment of the present invention first detect an eye region in an input image frame based on a support vector machine, and then a support vector machine within the detected eye region based on a support vector machine. Instead, it determines the blinking state based on the convolutional neural network model.

In addition, the eye blink detection method and the eye blink detection apparatus according to an embodiment of the present invention are based on eye position information on images acquired by the mobile terminal and direction information measured by the mobile terminal, by regression analysis. A regression equation representing a relationship between eye position information and direction information is determined, and a search area is set by applying a direction measurement value of a mobile terminal that has acquired an input image frame to the regression equation. Accordingly, it is possible to accurately and efficiently set a search area for detecting the user's eye position by reflecting the user's habit of using the mobile terminal.

1 is a flowchart of an eye blink detection method according to an embodiment of the present invention. 2 is a block diagram of an apparatus for detecting eye blinking according to an embodiment of the present invention. 1 and 2, an eye blink detection method according to an embodiment of the present invention may be performed by the eye blink detection apparatus 100. The eye blink detection apparatus 100 may be provided to a user's mobile terminal. The eye blink detection device 100 may be mounted on a mobile terminal or installed as an application.

In an embodiment, input image frames may be obtained by a mobile terminal (eg, a smartphone). The input image frames may be input video/photo frames captured by a camera (not shown) of the mobile terminal. The input image frames are input to the eye blink detection apparatus 100. Steps (S10 to S30) of the eye blink detection method illustrated in FIG. 1 may be sequentially performed on successive input image frames, respectively.

The eye blink detection apparatus 100 may include a setting unit 110, a detection unit 120, a determination unit 130, a storage unit 140, and a direction measurement unit 150. 3 is an exemplary diagram illustrating setting a search area and detecting an eye area in an input image frame according to an embodiment of the present invention. 1 to 3, the setting unit 110 sets a region of interest (ROI) for eye detection in an input image frame (S10).

In an embodiment, the setting unit 110 may set the search area ROI by regression analysis based on a direction measurement value measured by the mobile terminal that has acquired the input image frame. A specific method for setting the search area ROI in the input image frame by regression analysis will be described later.

When the search area ROI is set in the input image frame, the detection unit 120 detects the eye area CR within the search area ROI of the input image frame based on the support vector machine (S20). In an embodiment, the detection unit 120 may detect the eye region CR based on a support vector machine and a Histogram of Oriented Gradients (HOG) feature.

In an embodiment, two eye regions CR may be detected within the search region ROI. Unlike the illustration of FIG. 3, it is possible to detect one eye area CR within two search areas ROI, or detect only one eye area CR within one search area ROI.

When the eye area CR is detected in the input image frame, the determination unit 130 determines a blinking state in the eye area CR based on the convolutional neural network model (S30). In an embodiment, the determination unit 130 may determine the user's eye blink state based on the LeNet-5 convolutional neural network model.

The determination unit 130 may determine an eye blinking state among a closed eye state, an open eye state, and a background state for the eye area CR.

The closed eye state means a state in which the user's eyes are closed in the input image frame, and the open eye state means the state in which the user's eyes are open in the input image frame. When the determination unit 130 classifies the eye area CR as a background class, it means that there is a high possibility that the eye cannot be detected as an incorrect area.

According to an embodiment of the present invention, firstly, after detecting the eye area CR by a support vector machine within the search area ROI of the input image frame, secondly, the eye area CR is detected using the entire search area ROI. Since the blinking state is detected using only the eye region CR detected based on the support vector machine without detecting the blinking state, it is possible to efficiently detect the blinking of the user's eyes.

According to an embodiment of the present invention, the advantages of the support vector machine and the convolutional neural network model can be obtained. In other words, by applying a support vector machine, it is possible to increase computational efficiency and obtain a high frame throughput, and at the same time, by applying a convolutional neural network model, eye blinking can be reliably detected. It is possible to minimize the detection of false-negative.

The storage unit 140 generates input image frames, a program (algorithm) for eye blink detection in the input image frames, a direction measurement value of a mobile terminal, regression analysis data for estimation of a search area, and regression analysis data. Various types of information, such as eye position information in the images for use and direction information of the mobile terminal at the time of capturing images, may be stored.

The direction measuring unit 150 is for measuring the direction of the mobile terminal, and may include, for example, an acceleration sensor such as a gyro sensor, a geomagnetic sensor, and the like. The direction measurement unit 150 may measure direction (position) information of the mobile terminal based on the 3-axis acceleration measurement value. The direction measurement value of the mobile terminal can be used to estimate the search area in the input image frame, which will be described later.

4 is a state transition diagram of an eye blink detection method according to an embodiment of the present invention. Referring to FIG. 4, the eye blink detection method according to an embodiment of the present invention may detect a user's eye blink from input image frames while switching between two states of a detection state and a tracking state.

The detection state is a state for detecting the user's eyes when the user's eyes are not detected. If the user's eyes are detected during the detection state, it is switched to the tracking state. The tracking state is a state for tracking the user's eyes when the user's eyes are detected. If it fails to detect the user's eyes during the tracking state, the detection state is switched.

When the user's eyes are not detected, steps for detecting eye blinking in the detection state are performed, and the initial search area in the detection state may be set as the entire area of the input image frame. When the user's eyes are detected, the search area may be set based on the detected (or estimated) position of the eyes.

In the case of failure to find the eye position within the search area in the tracking state, before switching to the search state, the search area is predicted and the search area is reset, and if the eye position is detected in the reset search area, the tracking state is maintained. do. If the eye position is not detected even in the reset search area, it may be switched to the search state.

To reset the search area, measurements of the direction sensor may be used. The position of the user's eyes in the input image frame may vary in various ways depending on various factors such as the type of smartphone (mobile terminal) and the user's phone usage habit.

In order to efficiently set the search area in consideration of factors such as the type of the mobile terminal and the user's habit, the setting unit 110 estimates the position of the eye based on regression analysis using the direction measurement value of the mobile terminal, A search area can be set based on the estimated eye position.

5 is a flowchart of step S10 of FIG. 1. 6 is a configuration diagram of a setting unit configuring an apparatus for detecting blinking eyes according to an exemplary embodiment of the present invention. 2, 5, and 6, the setting unit 110 may include a regression analysis unit 112 and a search area setting unit 114.

When eyes are detected in images or input image frames in the search state or tracking state, the position of the eyes in the images or input image frames is stored in the storage unit 140 together with the direction measurement value of the mobile terminal. .

The regression analysis unit 112 uses real-time regression analysis to correlate the direction measurement values input from the direction measurement unit 150 of the mobile terminal with the user's eye position in images or input image frames.

In an embodiment, the regression analysis unit 112 performs regression analysis based on eye position information on images acquired by the mobile terminal and direction information of the mobile terminal at the time points at which the images are acquired. Accordingly, a regression equation indicating a relationship between eye position information and direction information, and deviation information (standard deviation, variance, determination coefficient, etc.) of the regression equation may be determined (S12).

In an embodiment, the regression equation is linear coefficients representing a linear relationship between each axial posture of the mobile terminal, the X-axis (horizontal direction) and Y-axis (vertical direction) positions of the input image frame, and the error of the regression equation. It may include deviation information.

The search area setting unit 114 may estimate the eye position by applying the direction measurement value obtained by the mobile terminal to the regression equation, and set the search area around the estimated eye position (S14).

In an embodiment, the size of the search area may be set based on deviation information for the regression equation. The search area can be set so that the size (area) of the search area increases as the data deviation of the regression analysis increases.

If the size of the search area is adjusted according to the data deviation of the regression analysis, the size of the search area is increased for users with a habit of using a mobile terminal with a large data deviation, increasing the eye detection success rate, and improving the habit of using a mobile terminal with a small data deviation With respect to the possessed user, the size of the search area can be minimized to increase the computational efficiency for eye detection.

When eyes are detected within the search area, eye blinking is detected within the search area. If no eyes are detected in the search area set in step S14, a process of detecting eyes for the entire input image frame may be performed by switching from the tracking state to the search state.

The size of the eye object in the input image frame changes according to the distance between the camera and the user's eye. Therefore, in order to adaptively detect the eye area in the search area of the input image frame, adaptively to the distance between the camera and the user, the detection unit 120 detects the eye area by scanning a plurality of different sized sliding windows in the search area in parallel. can do.

7 is a flowchart of step S20 of FIG. 1. Referring to FIGS. 2 and 7, first, the detection unit 120 may scan the search area as a plurality of sliding windows having different sizes (S22). Each sliding window is set to have a size smaller than that of the search area.

For example, when the size of the input image frame is 240×360, the detection unit 120 parallelizes rectangular sliding windows having different sizes of 25×25, 28×28, and 31×31 in a 79×106 search area. Can be scanned.

The detection unit 120 calculates a score representing the similarity to the eye shape based on the support vector machine for each sliding window (S24), and determines the eye position in the search area based on the scores calculated for the sliding windows. It can be detected (S26).

In an embodiment, the detection unit 120 may detect the eye area based on a sliding window having the smallest size among at least one sliding window having a score equal to or greater than a set reference value. The detection unit 120 may determine the eye area for each sliding window by using a non-maximum suppression technique for the eye area.

That is, by allowing the determination unit 130 to detect eye blinking in the eye region having the minimum size, the computational throughput is reduced, and the determination unit 130 can detect eye blinking reliably at the same time computationally efficient in a resource-limited mobile terminal. In order to do so, the detection unit 120 selects an eye area of the sliding window having the smallest size from among the eye areas of the sliding windows having a high score with a high probability of being an eye.

ve 알고리즘 대비 25% 포인트 향상된 성능을 나타내었다. 이하에서 보다 상세히 본 발명의 실시예에 대해 설명한다.Experimental results for commercial smartphones show that the present invention has a high accuracy of 94% or more and a high throughput of 22 fps or more. Experimental results for the present invention are na

Compared to the ve algorithm, the performance is improved by 25% points. Hereinafter, embodiments of the present invention will be described in more detail.

Two classifiers including a support vector machine and a convolutional neural network (LeNet-5) were trained using a CEW data set consisting of open and closed eye image clips and a data set of background image clips.

The support vector machine classified open-eye and closed-eye images (a total of 9692 images) into a positive class, and 9810 background image clips were classified into a negative class, and learned. LeNet-5 was trained using 3 different labels: 4924 open eye images, 4870 closed eye images and 4905 background images.

The experiment was performed using an LG V20 smartphone (mobile terminal) with a Qualcomm Snapdragon 820 processor, and a video input resolution of 240×360 was used. When eyes are detected in the input image frame in the search step, the search area is set to a 79×106 area around the eye position.

8 is a diagram showing the accuracy of a method for detecting eye blinking according to an embodiment of the present invention. In FIG. 8, the accuracy and recall ratio were calculated based on the number of true positives, false positives, and actual positives, and the accuracy was true positive/(true). Positive + positive errors) count ratio, recall ratio is the true positive/actual positive number ratio. As shown in FIG. 8, the accuracy of the present invention was 94.4%, which was high, and was higher than the recall rate (89.7%).

According to an embodiment of the present invention, by a hybrid classification system combining two machine learning classifiers (a linear SVM classifier with HOG features, a LeNet-5 convolutional neural network model), eye blinking efficiently and reliably in a resource-limited mobile terminal (eye blinking) can be detected.

The method according to the embodiment of the present invention can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. Computer-readable recording media include volatile memories such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), Read Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Nonvolatile memory such as Electrically Erasable and Programmable ROM (EEPROM), flash memory device, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), floppy disk, hard disk, or The optical reading medium may be, for example, a storage medium such as a CD-ROM or a DVD, but is not limited thereto.

It should be understood that the above embodiments have been presented to aid the understanding of the present invention, and do not limit the scope of the present invention, and various deformable embodiments are also within the scope of the present invention. The technical protection scope of the present invention should be determined by the technical idea of the claims, and the technical protection scope of the present invention is not limited to the literal description of the claims itself, but a scope that has substantially equal technical value. It should be understood that it extends to the invention of.

100: eye blink detection device
110: setting unit
112: regression analysis unit
114: search area setting unit
120: detection unit
130: judgment unit
140: storage unit
150: direction measuring unit
ROI: navigation area
CR: eye area

Claims (17)

Setting a search area for eye detection in the input image frame;
Detecting an eye area within the search area of the input image frame based on a support vector machine; And
Based on the convolutional neural network model, comprising the step of determining an eye blinking state within the eye region,
The step of detecting the eye area,
Scanning the search area with a plurality of sliding windows having different sizes;
Calculating scores representing a similarity to an eye shape for the sliding windows based on the support vector machine; And
Including the step of detecting the eye region based on a sliding window having a smallest size among at least one sliding window having a score equal to or greater than a set reference value,
How to detect blinking. The method of claim 1,
The detecting of the eye area includes detecting the eye area based on the support vector machine and edge direction histogram features,
The determining of the eye blinking state may include determining the eye blinking state based on a LeNet-5 convolutional neural network model. The method of claim 1,
The determining of the eye blinking state includes determining one of a closed eye state, an open eye state, and a background state for the eye area. delete The method of claim 1,
The setting of the search area includes setting the search area based on a direction measurement value measured by a mobile terminal that has acquired the input image frame. The method of claim 5,
Between the eye position information and the direction information by regression analysis based on eye position information on the images acquired by the mobile terminal and the direction information of the mobile terminal at the time points at which the images are acquired. An eye blink detection method further comprising the step of determining a regression equation representing the relationship. The method of claim 6,
The step of setting the search area,
Estimating the eye position by applying the direction measurement value to the regression equation; And
And setting the search area around the estimated eye position. The method of claim 6,
The size of the search area is set based on deviation information for the regression equation. delete delete A computer-readable recording medium on which a program for executing the blink detection method according to any one of claims 1 to 3 and 5 to 8 is recorded. A setting unit configured to set a search area for eye detection in the input image frame;
A detection unit configured to detect an eye area within the search area of the input image frame based on a support vector machine; And
Based on the convolutional neural network model, including a determination unit configured to determine an eye blinking state within the eye region,
The detection unit,
Scanning the search area with a plurality of sliding windows having different sizes;
Calculating scores representing a similarity to an eye shape for the sliding windows based on the support vector machine;
Detecting the eye position in the search area based on the scores; And
Configured to detect the eye region based on a sliding window having a smallest size among at least one sliding window having a score equal to or greater than a set reference value,
Eye blink detection device. The method of claim 12,
The detection unit detects the eye region based on the support vector machine and edge direction histogram features,
The determining unit, based on the LeNet-5 convolutional neural network model, determines an eye blinking state from among a closed eye state, an open eye state, and a background state. The method of claim 12,
The setting unit is configured to set the search area based on a direction measurement value measured by the mobile terminal that has obtained the input image frame. The method of claim 14,
The setting unit, based on the eye position information on the images acquired by the mobile terminal and the direction information of the mobile terminal at the time points at which the images are acquired, the eye position information and the An eye blink detection device configured to determine a regression equation representing a relationship between direction information. The method of claim 15,
The setting unit,
Estimating the eye position by applying the direction measurement to the regression equation;
Determining the size of the search area based on the deviation information for the regression equation; And
An apparatus for detecting eye blinking, configured to set the search area based on the estimated eye position and the size of the search area. delete

Images