KR20200017576A - Deep neural network based object detecting apparatus and method - Google Patents

Abstract

Disclosed are an eye blinking detecting apparatus capable of efficiently and reliably detecting eye blinking and an eye blinking detecting method. According to an embodiment of the present invention, the eye blinking detecting method comprises the steps of: detecting an eye region in an input image frame based on a support vector machine; and determining an eye blinking state in the eye region based on a convolutional neural network model.

Description

DEEP NEURAL NETWORK BASED OBJECT DETECTING APPARATUS AND METHOD}

The present invention relates to an apparatus and method for detecting a deep neural network-based object, and more particularly, to an eye blink detection device and an eye blink detection method capable of detecting eye blinking efficiently and reliably.

Blink detection or eye tracking algorithms have a variety of applications in mobile environments (eg, smartphone platforms). For example, a blink blink algorithm may be used to prevent spoofing in a facial recognition system. As another example, eye blink detection systems can be used to advise smartphone users about their blinking habits to alleviate the computer vision disease suffered by many smartphone users.

Various studies have been conducted on eye tracking and blink blink detection algorithms based on video input. However, conventional techniques have been developed for desktop environments and may not work properly in mobile / smartphone environments due to the limited computing resources of the smartphone and / or frequent eye changes due to user movement in input video frames. .

The present invention provides an eye blink detection device, an eye blink detection method, and a recording medium capable of detecting eye blinks efficiently and reliably.

The present invention also provides an eye blink detection device, an eye blink detection method, and a recording medium capable of detecting eye blink with high reliability by using limited computing resources of a mobile terminal such as a smart phone.

The problem to be solved by the present invention is not limited to the above-mentioned problem. Other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

An eye blink detection method according to an aspect of the present invention includes detecting an eye region in an input image frame based on a support vector machine; And determining an eye blink state within the eye region based on the convolutional neural network model.

The detecting of the eye area may detect the eye area based on the support vector machine and an edge direction histogram feature.

In the determining of the blink state, the blink state may be determined based on a LeNet-5 convolutional neural network model.

The determining of the eye blink state may determine any one of a closed eye state, an open eye state, and a background state of the eye area.

The eye blink detection method may further include setting a search area for eye detection in the input image frame.

The detecting of the eye region may detect the eye region based on the support vector machine in the search region.

In the setting of the search area, the search area may be set based on a direction measurement value measured by the mobile terminal acquiring the input image frame.

The eye blink detection method is based on eye position information on images acquired by the mobile terminal and the direction information of the mobile terminal at the time points at which the images are obtained, the eye position information by regression analysis. And determining a regression equation representing a relationship between the direction information and the direction information.

The setting of the search area may include estimating the eye position by applying the direction measurement value to the regression equation; And setting the search area based on the estimated eye position.

The size of the search region may be set based on the deviation information on the regression equation.

The detecting of the eye area may include scanning the search area with a plurality of sliding windows having different sizes; Calculating, for the sliding windows, scores indicating similarity with an eye shape based on the support vector machine; And detecting the eye position in the search area based on the scores.

The detecting of the eye area may detect the eye area based on a sliding window having a smallest size among at least one sliding window having a score equal to or greater than a set reference value.

According to another aspect of the present invention, a computer-readable recording medium having a program recorded thereon for executing the eye blink detection method is provided.

According to still another aspect of the present invention, there is provided an apparatus, comprising: a detector configured to detect an eye region in an input image frame based on a support vector machine; And a determination unit configured to determine an eye blink state in the eye region based on the convolutional neural network model.

The detector may detect the eye region based on the support vector machine and an edge direction histogram feature.

The determination unit may determine the eye blink state among the closed eye state, the opened eye state, and the background state based on the LeNet-5 convolution neural network model.

The eye blink detection device may further include a setting unit configured to set a search area for eye detection in the input image frame.

The setting unit may be configured to set the search area based on a direction measurement value measured by the mobile terminal acquiring the input image frame.

The detector may be configured to detect the eye region based on the support vector machine in the search region.

The setting unit, based on the eye position information on the images obtained by the mobile terminal and the direction information of the mobile terminal at the time when the images are obtained, the eye position information and the by the regression analysis It can be configured to determine a regression equation representing the relationship between the direction information.

The setting unit estimates the eye position by applying the direction measurement value to the regression equation; Determine the size of the search region based on the deviation information for the regression equation; The search region may be configured based on the estimated eye position and the size of the search region.

The detecting unit scans the search area into a plurality of sliding windows having different sizes; Calculating scores for the sliding windows indicating similarity with an eye shape based on the support vector machine; Detect the eye position in the search area based on the scores; The eye region may be detected based on a sliding window having a smallest size among at least one sliding window having a score equal to or greater than a set reference value.

According to an embodiment of the present invention, an eye blink detection apparatus, an eye blink detection method, and a recording medium capable of detecting eye blinking efficiently and reliably are provided.

According to an embodiment of the present invention, there is provided an eye blink detection device, an eye blink detection method, and a recording medium capable of detecting eye blink with high reliability by using limited computing resources of a mobile terminal such as a smart phone.

The effects of the present invention are not limited to the effects described above. Effects that are not mentioned will be clearly understood by those skilled in the art from the present specification and the accompanying drawings.

1 is a flowchart of an eye blink detection method according to an exemplary embodiment of the present invention.
2 is a block diagram of the eye blink detection apparatus according to an embodiment of the present invention.
FIG. 3 is an exemplary diagram illustrating setting a search area in an input image frame and detecting an eye area according to an embodiment of the present invention.
4 is a state transition diagram of an eye blink detection method according to an exemplary embodiment of the present invention.
5 is a flowchart of step S10 of FIG.
6 is a configuration diagram of a setting unit constituting an eye blink detection apparatus according to an embodiment of the present invention.
7 is a flowchart of step S20 of FIG.
8 is a view showing the accuracy of the eye blink detection method according to an embodiment of the present invention.

Other advantages and features of the present invention, and methods of achieving them will become apparent with reference to the following embodiments in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and the present invention is only defined by the scope of the claims. If not defined, all terms used herein (including technical or scientific terms) have the same meaning as commonly accepted by universal techniques in the prior art to which this invention belongs. General descriptions of known configurations may be omitted so as not to obscure the subject matter of the present invention. In the drawings of the present invention, the same reference numerals are used for the same or corresponding configurations. In order to help the understanding of the present invention, some of the components in the drawings may be somewhat exaggerated or reduced.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise", "have" or "include" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, one Or any other feature or number, step, operation, component, part, or combination thereof.

As used throughout the present specification, '~ part' is a unit for processing at least one function or operation, and may mean, for example, a hardware component such as software, FPGA, or ASIC. However, "~" is not meant to be limited to software or hardware. May be configured to reside in an addressable storage medium or may be configured to play one or more processors.

As an example, '~' means components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, and subs. Routines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided by the component and the '~' may be performed separately by the plurality of components and the '~', or may be integrated with other additional components.

According to an embodiment of the present invention, there is provided an eye blink detection device, an eye blink detection method, and a recording medium that can be used for efficient detection of eye blinks in a smartphone platform with limited resources.

Real-time processing is required for reliable blink detection, and a throughput of 10 fps or more is required. Therefore, in the case of the eye blink detection algorithm, it may be difficult to apply the convolutional neural network model to the eye tracking algorithm.

An eye blink detection method and an eye blink detection apparatus according to an embodiment of the present invention are provided by a heterogeneous hybrid learning system including a support vector machine and a convolutional neural network model. Detect eye-blinking.

According to an embodiment of the present invention, the heterogeneous hybrid learning system based on the support vector machine and the convolutional neural network can detect the user's eye blink efficiently and reliably, and utilize the limited computing resources of the mobile terminal such as a smartphone. Eye blink can be detected.

An eye blink detection method and an eye blink detection apparatus according to an embodiment of the present invention first detects an eye region in an input image frame based on a support vector machine, and then a support vector machine in the detected eye region based on the support vector machine. Instead, the blink state is determined based on the convolutional neural network model.

In addition, the eye blink detection method and the eye blink detection apparatus according to an embodiment of the present invention by the regression analysis based on the eye position information on the images obtained by the mobile terminal and the direction information measured by the mobile terminal A regression equation indicating a relationship between eye position information and direction information is determined, and a search area is set by applying a direction measurement value of a mobile terminal obtained with an input image frame to the regression equation. Accordingly, it is possible to accurately and efficiently set the search area for detecting the eye position of the user by reflecting the user's habit of using the mobile terminal.

1 is a flowchart of an eye blink detection method according to an exemplary embodiment of the present invention. 2 is a block diagram of the eye blink detection apparatus according to an embodiment of the present invention. 1 and 2, the eye blink detection method according to an embodiment of the present invention may be performed by the eye blink detection apparatus 100. The eye blink detection apparatus 100 may be provided to a user's mobile terminal. The eye blink detection apparatus 100 may be mounted on a mobile terminal or installed as an app.

In an embodiment, input image frames may be obtained by a mobile terminal (eg, a smartphone). The input image frames may be input video / photo frames taken by a camera (not shown) of the mobile terminal. The input image frames are input to the blinking apparatus 100. Steps S10 to S30 of the eye blink detection method illustrated in FIG. 1 may be sequentially performed on successive input image frames.

The eye blink detection apparatus 100 may include a setting unit 110, a detection unit 120, a determination unit 130, a storage unit 140, and a direction measuring unit 150. FIG. 3 is an exemplary diagram illustrating setting a search area in an input image frame and detecting an eye area according to an embodiment of the present invention. 1 to 3, the setting unit 110 sets a region of interest (ROI) for eye detection in an input image frame (S10).

In an embodiment, the setting unit 110 may set the search area ROI by regression analysis based on the direction measurement value measured by the mobile terminal that obtained the input image frame. A detailed method for setting the search area ROI in the input image frame by regression analysis will be described later.

When the search region ROI is set in the input image frame, the detector 120 detects the eye region CR in the search region ROI of the input image frame based on the support vector machine (S20). In an embodiment, the detector 120 may detect the eye region CR based on the support vector machine and the Histogram of Oriented Gradients (HOG) feature.

In an embodiment, two eye regions CR may be detected within the search region ROI. Unlike the illustration of FIG. 3, it is also possible to detect one eye area CR in each of two search areas ROI, or only one eye area CR in one search area ROI.

When the eye region CR is detected in the input image frame, the determination unit 130 determines the eye blink state in the eye region CR based on the convolutional neural network model (S30). In an embodiment, the determination unit 130 may determine the blinking state of the user based on the LeNet-5 convolutional neural network model.

The determination unit 130 may determine the eye blink state of any one of a closed eye state, an opened eye state, and a background state with respect to the eye region CR.

The closed eye state means a state in which the user closed the eyes in the input image frame, and the open eye state means a state in which the user opened the eyes in the input image frame. When the determiner 130 classifies the eye region CR as a background class, it means that the eye is likely to be an incorrect region that cannot be detected.

According to an embodiment of the present invention, the eye region CR is first detected by the support vector machine in the search region ROI of the input image frame, and then the second eye is used by using the entire search region ROI. Since the eye blink state is detected using only the eye region CR detected based on the support vector machine without detecting the blink state, the eye blink of the user can be detected efficiently.

According to the embodiment of the present invention, the advantages of the support vector machine and the advantages of the convolutional neural network model can be obtained. That is, by applying the support vector machine, it is possible to increase the computational efficiency and to obtain high frame rate, and to detect the eye blink reliably by applying the convolutional neural network model. False-negative detection can be minimized.

The storage unit 140 generates input image frames, a program (algorithm) for detecting eye blinks in the input image frames, direction measurement values of the mobile terminal, regression analysis data for estimating a search area, and regression analysis data. Various information, such as eye position information in the images and direction information of the mobile terminal at the time of capturing the images, may be stored.

The direction measuring unit 150 measures the direction of the mobile terminal, and may include, for example, an acceleration sensor such as a gyro sensor, a geomagnetic sensor, or the like. The direction measuring unit 150 may measure the direction (posture) information of the mobile terminal based on the 3-axis acceleration measurement value. The direction measurement value of the mobile terminal may be used to estimate the search area in the input image frame, which will be described later.

4 is a state transition diagram of an eye blink detection method according to an exemplary embodiment of the present invention. Referring to FIG. 4, the eye blink detection method according to an exemplary embodiment of the present invention may detect eye blink of a user from input image frames while switching between two states of a detection state and a tracking state.

The detection state is a state for detecting the eyes of the user when the eyes of the user are not detected. If the user's eyes are detected during the detection state, the state is switched to the tracking state. The tracking state is a state for tracking the eyes of the user when the eyes of the user are detected. If it fails to detect the user's eye during the tracking state, it switches to the detection state.

If the user's eye is not detected, steps for detecting eye blink in the detection state are performed, and the initial search area in the detection state may be set to the entire area of the input image frame. When the user's eye is detected, the search area may be set based on the location of the detected (or estimated) eye.

If the tracking state fails to locate the eye within the search area, reset the search area by predicting the eye position before switching to the search state, and remain tracking if the eye position is detected in the reset search area. do. If the eye position is not detected even in the reset search area, it may be switched to the search state.

For resetting the search area, measurements of the direction sensor can be used. The position of the user's eyes in the input image frame may vary depending on various factors such as the type of smartphone (mobile terminal) and the user's phone usage habit.

In order to efficiently set the search area in consideration of factors such as the type of the mobile terminal and the user's habit, the setting unit 110 estimates the eye position based on the regression analysis using the direction measurement value of the mobile terminal, The search region may be set based on the estimated eye position.

5 is a flowchart of step S10 of FIG. 6 is a configuration diagram of a setting unit constituting an eye blink detection apparatus according to an exemplary embodiment of the present invention. 2, 5, and 6, the setting unit 110 may include a regression analyzing unit 112 and a search region setting unit 114.

In the search or tracking state, when the eye is detected in the images or input picture frames, the position of the eye in the images or input picture frames is stored in the storage 140 together with the orientation measurement of the mobile terminal. .

The regression analyzer 112 uses real-time regression analysis to associate the eye position of the user in the images or input image frames with the direction measurement values input from the direction measurer 150 of the mobile terminal.

In an embodiment, the regression analysis unit 112 performs regression analysis on the basis of the eye position information on the images acquired by the mobile terminal and the direction information of the mobile terminal at the time points at which the images were obtained. As a result, a regression equation indicating a relationship between eye position information and direction information and deviation information (standard deviation, variance, coefficient of determination, etc.) of the regression equation can be determined (S12).

In an embodiment, the regression equations represent linear coefficients representing the linear relationship between each axial attitude of the mobile terminal, the position of the X axis (horizontal direction) and the Y axis (vertical direction) of the input image frame, and the error of the regression equation Deviation information may be included.

The search region setting unit 114 may estimate the eye position by applying the direction measurement value obtained by the mobile terminal to the regression equation, and set the search region around the estimated eye position (S14).

In an embodiment, the size of the search region may be set based on the deviation information on the regression equation. The search region may be set such that the size (area) of the search region increases as the data deviation of the regression analysis increases.

If the size of the search area is adjusted according to the data deviation of the regression analysis, the user who has a habit of using the mobile terminal with a large data deviation may increase the size of the search area to increase eye detection success rate, With respect to the user, the size of the search area can be minimized to increase the computational efficiency for eye detection.

When eyes are detected in the search area, eye blinks are detected in the search area. If the eye is not detected in the search area set in step S14, a process of detecting the eye with respect to the entire input image frame by switching from the tracking state to the search state may be performed.

The size of the object of the eye in the input image frame varies with the distance between the camera and the user's eye. Accordingly, in order to detect an appropriate eye area in the search area of the input image frame, the detection unit 120 detects the eye area by scanning a plurality of sliding windows of different sizes in parallel to the search area, adaptively to the distance between the camera and the user. can do.

7 is a flowchart of step S20 of FIG. 2 and 7, first, the detector 120 may scan the search area into sliding windows having a plurality of different sizes (S22). Each sliding window is set to have a size smaller than the search area.

For example, if the size of the input image frame is 240 × 360, the detector 120 parallelizes the rectangular sliding windows having different sizes of 25 × 25, 28 × 28, 31 × 31 in a 79 × 106 search area. You can scan.

The detector 120 calculates, for each sliding window, a score indicating a similarity to the eye shape based on the support vector machine (S24), and calculates an eye position in the search area based on the scores calculated for the sliding windows. It can detect (S26).

In an embodiment, the detector 120 may detect an eye region based on a sliding window having the smallest size among at least one sliding window having a score equal to or greater than a set reference value. The detector 120 may determine the eye region for each sliding window by using a non-maximum suppression technique.

That is, the determination unit 130 detects eye blinking for the eye area having a minimum size, thereby reducing arithmetic throughput, and the determination unit 130 can efficiently and reliably detect eye blinking in a resource-limited mobile terminal. In order to ensure that the detection unit 120 selects the eye area of the sliding window having the smallest size among the eye areas of the sliding windows having a high score that is likely to be the eye.

ve 알고리즘 대비 25% 포인트 향상된 성능을 나타내었다. 이하에서 보다 상세히 본 발명의 실시예에 대해 설명한다.Experimental results on commercial smartphones show that the present invention has a high accuracy of 94% or higher and a high throughput of 22 fps or higher. Experimental results for the present invention na in commercial smartphones

25% point improvement compared to ve algorithm. Hereinafter, embodiments of the present invention will be described in more detail.

Two classifiers, including a support vector machine and a convolutional neural network (LeNet-5), were trained using a CEW data set consisting of floating eye and closed eye image clips and a data set of background image clips.

The support vector machine classified the opened and closed eye images (9692 images in total) into a positive class and the 9810 background image clips into a negative class to learn. LeNet-5 was trained using three different labels: 4924 open eye images, 4870 closed eye images and 4905 background images.

The experiment was performed using an LG V20 smartphone (mobile terminal) with a Qualcomm Snapdragon 820 processor, using a video input resolution of 240 × 360. If eyes were detected in the input image frame in the search step, the search area was set to 79 x 106 area around the eye position.

8 is a view showing the accuracy of the eye blink detection method according to an embodiment of the present invention. In FIG. 8, the precision and recall rate were calculated based on the number of true positives, false positives, and actual positives, and the accuracy was true / (true). The positive rate plus false positive rate and recall rate are the true positive / actual positive number ratios. As shown in FIG. 8, the accuracy of the present invention was high as 94.4% and was higher than the recall rate (89.7%).

According to an embodiment of the present invention, by a hybrid classification system combining two machine learning classifiers (linear SVM classifier with HOG features, LeNet-5 convolutional neural network model), it blinks efficiently and reliably in resource limited mobile terminals. eye blinking can be detected.

The method according to an embodiment of the present invention may be implemented as a program that can be executed in a computer, for example, and may be implemented in a general-purpose digital computer which operates the program using a computer-readable recording medium. The computer-readable recording medium may include volatile memory such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), Nonvolatile memory, such as electrically erasable and programmable ROM (EEPROM), flash memory device, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), floppy disk, hard disk, or Optical reading media may be, for example, but not limited to, a storage medium in the form of CD-ROM, DVD, or the like.

The above embodiments are presented to aid the understanding of the present invention, and do not limit the scope of the present invention, from which it should be understood that various modifications are within the scope of the present invention. The technical protection scope of the present invention should be defined by the technical spirit of the claims, and the technical protection scope of the present invention is not limited to the literary description of the claims, but the scope of the technical equivalents is substantially equal. It is to be understood that the invention extends to the invention.

100: blinking device
110: setting unit
112: regression analysis unit
114: search area setting unit
120: detector
130: judgment
140: storage unit
150: direction measuring unit
ROI: navigation area
CR: eye area

Claims (17)

Detecting an eye region in the input image frame based on the support vector machine; And
And determining a blink state in the eye region based on a convolutional neural network model. The method of claim 1,
The detecting of the eye region may include detecting the eye region based on the support vector machine and an edge direction histogram feature.
The determining of the eye blink state, the eye blink detection method for determining the eye blink state based on the LeNet-5 convolutional neural network model. The method of claim 1,
The determining of the eye blinking state may include determining one of a closed eye state, an open eye state, and a background state of the eye area. The method of claim 1,
Setting a search area for eye detection in the input image frame;
The detecting of the eye area may include detecting the eye area based on the support vector machine in the search area. The method of claim 4, wherein
The setting of the search area may include setting the search area based on a direction measurement value measured by the mobile terminal acquiring the input image frame. The method of claim 5,
Based on the eye position information on the images acquired by the mobile terminal and the direction information of the mobile terminal at the time points at which the images were obtained, between the eye position information and the direction information by regression analysis. Determining a regression equation representing the relationship. The method of claim 6,
Setting the search area,
Estimating the eye position by applying the direction measurement to the regression equation; And
And setting the search area around the estimated eye position. The method of claim 4, wherein
And a size of the search area is set based on the deviation information on the regression equation. The method of claim 4, wherein
Detecting the eye area,
Scanning the search area with a plurality of different sliding windows;
Calculating, for the sliding windows, scores indicating similarity with an eye shape based on the support vector machine; And
And detecting the eye position in the search area based on the scores. The method of claim 9,
The detecting of the eye area may include detecting the eye area based on a sliding window having a smallest size among at least one sliding window having a score equal to or greater than a set reference value. A computer-readable recording medium having recorded thereon a program for executing the eye blink detection method according to any one of claims 1 to 10. A detection unit configured to detect an eye region in the input image frame based on the support vector machine; And
And a determination unit configured to determine an eye blink state within the eye region based on a convolutional neural network model. The method of claim 12,
The detection unit detects the eye region based on the support vector machine and the edge direction histogram feature,
The determination unit, based on the LeNet-5 convolutional neural network model, the eye blink detection device for determining the eye blink state from the closed eye state, open eye state and background state. The method of claim 12,
A setting unit configured to set a search area for eye detection in the input image frame,
The setting unit is configured to set the search area based on a direction measurement value measured by the mobile terminal that obtained the input image frame,
And the detection unit is configured to detect the eye region based on the support vector machine in the search region. The method of claim 14,
The setting unit, based on the eye position information on the images obtained by the mobile terminal and the direction information of the mobile terminal at the time when the images are obtained, the eye position information and the by the regression analysis And a blink blink detection device configured to determine a regression equation representing the relationship between the direction information. The method of claim 15,
The setting unit,
Apply the direction measurement to the regression equation to estimate the eye position;
Determine the size of the search region based on the deviation information for the regression equation; And
And configured to set the search area based on the estimated eye position and the size of the search area. The method according to any one of claims 14 to 16,
The detection unit,
Scan the search area into a plurality of sliding windows of different sizes;
Calculating scores for the sliding windows indicating similarity to an eye shape based on the support vector machine;
Detect the eye position in the search area based on the scores; And
The eye blink detection device is configured to detect the eye area based on a sliding window of the smallest size among at least one sliding window having a score equal to or greater than a set reference value.

Images