CN111488844B - Eye state detection method, device, equipment and storage medium - Google Patents
Eye state detection method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111488844B CN111488844B CN202010301042.4A CN202010301042A CN111488844B CN 111488844 B CN111488844 B CN 111488844B CN 202010301042 A CN202010301042 A CN 202010301042A CN 111488844 B CN111488844 B CN 111488844B
- Authority
- CN
- China
- Prior art keywords
- target
- eye
- state
- image
- edge feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/193—Preprocessing; Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
Abstract
The invention relates to the technical field of image recognition, and discloses a method, a device, equipment and a storage medium for detecting eye states. The eye state detection method comprises the following steps: acquiring a target image to be detected, wherein the target image comprises a target eye; extracting features of the target image to be detected by adopting a deep learning model to obtain a plurality of edge feature points, wherein the plurality of edge feature points comprise a plurality of edge feature point coordinates; fitting a plurality of edge feature point coordinates by adopting a least square method to obtain a first target parameter and a second target parameter; the state of the target eye is acquired based on the first target parameter and the second target parameter, and the state of the target eye is an open-eye state or a closed-eye state.
Description
Technical Field
The present invention relates to the field of image recognition technologies, and in particular, to a method, an apparatus, a device, and a storage medium for detecting an eye state.
Background
With the development of technologies such as image big data and deep learning, technologies such as face recognition and the like are very mature, and demands of people on man-machine interaction, facial expression interaction, face capture and the like are greatly improved.
Most of the existing eye recognition technologies are based on face recognition technologies, and most of facial expression researches are rough, for example, facial expressions only recognize smiles, laughter, anxiety, pain or sedation and the like; partial eye recognition techniques may detect eye conditions or locate the position of the eye.
In the prior art, methods for detecting eye states are mainly classified into a feature-based analysis method and a pattern-based classification method, wherein the feature-based analysis method mainly comprises algorithms such as template matching and eyelid detection, and the pattern-based classification method mainly extracts features of eye areas. However, in practical application, these algorithms are easily affected by changes in illumination, changes in face, and the like, and there are problems that the detection accuracy of eye states is low and that large amounts of data are insufficient in the process of recognizing eye states.
Disclosure of Invention
The invention mainly aims to solve the problems of low detection precision of the eye state and insufficient large data volume in the process of recognizing the eye state.
The first aspect of the present invention provides a method for detecting an eye state, including: acquiring a target image to be detected, wherein the target image to be detected comprises a target eye; extracting features of the target image to be detected by adopting a deep learning model to obtain a plurality of edge feature points, wherein the edge feature points comprise edge feature point coordinates; fitting the edge feature point coordinates by using a least square method to obtain a first target parameter and a second target parameter; and acquiring the state of a target eye based on the first target parameter and the second target parameter, wherein the state of the target eye is an open-eye state or a closed-eye state.
Optionally, in a first implementation manner of the first aspect of the present invention, the feature extracting, by using a deep learning model, the target image to be detected to obtain a plurality of edge feature points, where the edge feature points include edge feature point coordinates including: preprocessing the target image to be detected to obtain a preprocessed target image; inputting the preprocessed target image into a convolutional neural network for convolution to obtain a first target image; pooling the first target image in the convolutional neural network to obtain a second target image; performing nonlinear mapping on the second target image to obtain a target characteristic image; and extracting a plurality of edge feature points corresponding to the target eyes from the target feature image, wherein the edge feature points comprise edge feature point coordinates.
Optionally, in a second implementation manner of the first aspect of the present invention, the fitting the coordinates of the plurality of edge feature points by using a least square method to obtain a first target parameter and a second target parameter includes: in the target image to be detected, performing ellipse fitting on the edge feature points by adopting the least square method based on the edge feature point coordinates to obtain a target ellipse equation; and extracting a first target parameter and a second target parameter from the target elliptic equation.
Optionally, in a third implementation manner of the first aspect of the present invention, the acquiring, based on the first target parameter and the second target parameter, a state of a target eye, where the state of the target eye is an open-eye state or a closed-eye state includes: performing eye closure ratio calculation on the first target parameter and the second target parameter to obtain a target eye closure ratio; and acquiring a target eye state identifier according to the target eye closing ratio, and determining the state of the target eye based on the target eye state identifier, wherein the target eye state identifier is a first eye state identifier or a second eye state identifier, and the state of the target eye is a open eye state or a closed eye state.
Optionally, in a fourth implementation manner of the first aspect of the present invention, according to the target eye closing ratio, a target eye state identifier is obtained, and a state of the target eye is determined based on the target eye state identifier, where the target eye state identifier is a first eye state identifier or a second eye state identifier, and the state of the target eye is a open eye state or a closed eye state includes: reading the target eye closure ratio and judging whether the target eye closure ratio is greater than or equal to a target closure ratio threshold; if the target eye closing ratio is greater than or equal to the target closing ratio threshold, a first eye state identification is obtained, and the state of the target eye is determined to be a tension state based on the first eye state identification; and if the target eye closing ratio is smaller than the target closing ratio threshold, obtaining a second eye state identification, and determining that the state of the target eye is an eye closing state based on the second eye state identification.
Optionally, in a fifth implementation manner of the first aspect of the present invention, before the acquiring a target image to be detected, where the target image to be detected includes a target eye, the method for detecting an eye state further includes: establishing a standard image library, wherein the standard image library comprises a plurality of eye-opening state standard images and a plurality of eye-closing state standard images; a target closure ratio threshold is acquired based on the plurality of open eye state standard images and the plurality of closed eye state standard images.
Optionally, in a sixth implementation manner of the first aspect of the present invention, the acquiring the target closure ratio threshold based on the plurality of eye state standard images and the plurality of eye closure state standard images includes: initializing the number of error samples and a closure ratio critical value to obtain a lower bound value of the number of error samples and an initial closure ratio threshold value; respectively selecting a plurality of closure ratio reference thresholds within a preset range according to preset standards; counting the number of error samples corresponding to each closing ratio reference threshold in the plurality of eye-opening state standard images and the plurality of eye-closing state standard images according to each closing ratio reference threshold in the plurality of closing ratio reference thresholds, so as to obtain a plurality of reference error sample numbers; comparing the infinitesimal value of the error sample number with the plurality of reference error sample numbers to obtain a target reference error sample number, and adjusting the infinitesimal value of the error sample number to the target reference error sample number to obtain a target error sample number, wherein the target reference error sample number is the reference error sample number smaller than or equal to the infinitesimal value of the error sample number; and determining a target closure ratio reference threshold value from the closure ratio reference thresholds based on the target error sample number, and adjusting the initial closure ratio threshold value to the target closure ratio reference threshold value to obtain a target closure ratio threshold value.
A second aspect of the present invention provides a detection apparatus for eye state, comprising: the image acquisition module is used for acquiring a target image to be detected, wherein the target image to be detected comprises a target eye; the feature extraction module is used for extracting features of the target image to be detected by adopting a deep learning model to obtain a plurality of edge feature points, wherein the edge feature points comprise edge feature point coordinates; the parameter acquisition module is used for fitting the coordinates of the edge feature points by adopting a least square method to obtain a first target parameter and a second target parameter; the eye state acquisition module is used for acquiring the state of a target eye based on the first target parameter and the second target parameter, wherein the state of the target eye is an open-eye state or a closed-eye state.
Optionally, in a first implementation manner of the second aspect of the present invention, the feature extraction module is specifically configured to: preprocessing the target image to be detected to obtain a preprocessed target image; inputting the preprocessed target image into a convolutional neural network for convolution to obtain a first target image; pooling the first target image in the convolutional neural network to obtain a second target image; performing nonlinear mapping on the second target image to obtain a target characteristic image; and extracting a plurality of edge feature points corresponding to the target eyes from the target feature image, wherein the edge feature points comprise edge feature point coordinates.
Optionally, in a second implementation manner of the second aspect of the present invention, the parameter obtaining module is specifically configured to: in the target image to be detected, performing ellipse fitting on the edge feature points by adopting the least square method based on the edge feature point coordinates to obtain a target ellipse equation; and extracting a first target parameter and a second target parameter from the target elliptic equation.
Optionally, in a third implementation manner of the second aspect of the present invention, the eye state obtaining module specifically includes: a closing ratio calculating unit, configured to perform an eye closing ratio calculation on the first target parameter and the second target parameter, to obtain a target eye closing ratio; the eye state acquisition unit is used for acquiring a target eye state identifier according to the target eye closing ratio, determining the state of the target eye based on the target eye state identifier, wherein the target eye state identifier is a first eye state identifier or a second eye state identifier, and the state of the target eye is a open eye state or a closed eye state.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the eye state obtaining unit is specifically configured to: reading the target eye closure ratio and judging whether the target eye closure ratio is greater than or equal to a target closure ratio threshold; if the target eye closing ratio is greater than or equal to the target closing ratio threshold, a first eye state identification is obtained, and the state of the target eye is determined to be a tension state based on the first eye state identification; and if the target eye closing ratio is smaller than the target closing ratio threshold, obtaining a second eye state identification, and determining that the state of the target eye is an eye closing state based on the second eye state identification.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the device for detecting an eye state further includes: the image library establishing module is used for establishing a standard image library, and the standard image library comprises a plurality of eye state standard images and a plurality of eye closing state standard images; and the target threshold acquisition module is used for acquiring a target closure ratio threshold based on the plurality of eye opening state standard images and the plurality of eye closing state standard images.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the target threshold obtaining module is specifically configured to: initializing the number of error samples and a closure ratio critical value to obtain a lower bound value of the number of error samples and an initial closure ratio threshold value; respectively selecting a plurality of closure ratio reference thresholds within a preset range according to preset standards; counting the number of error samples corresponding to each closing ratio reference threshold in the plurality of eye-opening state standard images and the plurality of eye-closing state standard images according to each closing ratio reference threshold in the plurality of closing ratio reference thresholds, so as to obtain a plurality of reference error sample numbers; comparing the infinitesimal value of the error sample number with the plurality of reference error sample numbers to obtain a target reference error sample number, and adjusting the infinitesimal value of the error sample number to the target reference error sample number to obtain a target error sample number, wherein the target reference error sample number is the reference error sample number smaller than or equal to the infinitesimal value of the error sample number; and determining a target closure ratio reference threshold value from the closure ratio reference thresholds based on the target error sample number, and adjusting the initial closure ratio threshold value to the target closure ratio reference threshold value to obtain a target closure ratio threshold value.
A third aspect of the present invention provides a detection apparatus for eye state, comprising: a memory and at least one processor, the memory having instructions stored therein, the memory and the at least one processor being interconnected by a line; the at least one processor invokes the instructions in the memory to cause the eye state detection device to perform the eye state detection method described above.
A fourth aspect of the present invention provides a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the above-described eye state detection method.
In the technical scheme provided by the invention, a target image to be detected is obtained, wherein the target image to be detected comprises a target eye; extracting features of the target image to be detected by adopting a deep learning model to obtain a plurality of edge feature points, wherein the edge feature points comprise edge feature point coordinates; fitting the edge feature point coordinates by using a least square method to obtain a first target parameter and a second target parameter; and acquiring the state of a target eye based on the first target parameter and the second target parameter, wherein the state of the target eye is an open-eye state or a closed-eye state. In the embodiment of the invention, the deep learning model is adopted to extract the characteristic points of the target image to be detected, the target eye closing ratio value is obtained based on the characteristic point coordinate calculation, and the target eye state is obtained according to the target eye closing ratio value, so that the problem of insufficient large data amount in the process of identifying the eye state is solved, and the accuracy of eye state detection is improved.
Drawings
FIG. 1 is a schematic diagram of an embodiment of a method for detecting eye states according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of another embodiment of a method for detecting eye states according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an embodiment of an eye state detection device according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another embodiment of an eye state detection device according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an embodiment of an eye state detection apparatus according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a method, a device, equipment and a storage medium for detecting eye states, which are used for extracting characteristic points of a target image to be detected by adopting a deep learning model, calculating based on characteristic point coordinates to obtain a target eye closing ratio value, and obtaining the target eye states according to the target eye closing ratio value, so that the problem of insufficient large data amount in the process of identifying the eye states is solved, and the accuracy of eye state detection is improved.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
For ease of understanding, a specific flow of an embodiment of the present invention is described below, referring to fig. 1, and one embodiment of a method for detecting an eye state in an embodiment of the present invention includes:
101. acquiring a target image to be detected, wherein the target image to be detected comprises a target eye;
the server acquires a target image to be detected including a target eye.
The server acquires images from terminals such as a mobile phone, a computer or a camera, and the target image to be detected can be an image from a shot eye or an image for locating an eye area in a face image.
It should be noted that, the format of the target image to be detected is not limited in this embodiment, and the size of the target image to be detected may be a conventional size (length is 24CM and width is 14 CM), and the size of the target image to be detected is not limited in this embodiment.
It is to be understood that the execution body of the present invention may be an eye state detection device, and may also be a terminal or a server, which is not limited herein. The embodiment of the invention is described by taking a server as an execution main body as an example.
102. Extracting features of a target image to be detected by adopting a deep learning model to obtain a plurality of edge feature points, wherein the edge feature points comprise edge feature point coordinates;
And the server inputs the target image to be detected into the deep learning model for feature extraction to obtain a plurality of edge feature points comprising edge feature point coordinates.
The server performs a series of processing on the target image to be detected to obtain a processed target image, inputs the preprocessed target image into a convolutional neural network for feature extraction to obtain a plurality of edge feature points distributed around the eyes, and each edge feature point corresponds to one edge feature point coordinate.
For example, the target image to be detected is subjected to gray level processing, normalization and other processing to obtain a processed target image, and the preprocessed target image is input into a convolutional neural network to perform feature extraction to obtain P 0 、P 1 、P 2 、P 3 、P 4 、P 5 、P 6 And P 7 The 8 edge feature points and the 8 edge feature point pairs8 edge feature point coordinates, P 0 、P 1 、P 2 、P 3 、P 4 、P 5 、P 6 And P 7 For 8 edge feature points distributed around the target eye.
103. Fitting a plurality of edge feature point coordinates by adopting a least square method to obtain a first target parameter and a second target parameter;
and the server performs ellipse fitting on the plurality of edge characteristic points by adopting a least square method and a plurality of edge characteristic point coordinates, so as to obtain a first target parameter and a second target parameter.
The least square method is also called as a least squares method, and is a mathematical optimization technology. The method can simply and conveniently calculate unknown data by using a least square method by minimizing the square sum of errors to find the optimal function matching of the data, and the square sum of errors between the calculated data and actual data is minimized. The least squares method can also be used for curve fitting, and other optimization problems can also be expressed by the least squares method by minimizing energy or maximizing entropy. In short, the least squares method is similar to gradient descent, and is a common method for solving unconstrained optimization problems, and can also be used for curve fitting to solve regression problems. The least square method is essentially to minimize the "mean square error", which is 1/m of the sum of squares of the residuals, m being the number of samples, and the mean square error is the most commonly used performance metric in regression tasks.
In the scheme, a least square method is used for curve fitting, a server uses the least square method to carry out ellipse fitting on a plurality of edge characteristic points, a target ellipse equation can be obtained by fitting on a target eye based on the plurality of edge characteristic points and the least square method, and a first target parameter and a second target parameter are obtained from the target ellipse equation by the server.
104. The state of the target eye is acquired based on the first target parameter and the second target parameter, and the state of the target eye is an open-eye state or a closed-eye state.
The server acquires the open-eye state or the closed-eye state of the target eye according to the first target parameter and the second target parameter.
The server calculates a ratio according to the first target parameter and the second target parameter, and judges whether the state of the target eye is an open-eye state or a closed-eye state based on the parameter ratio. For example, the first target parameter is 5, the second target parameter is 4, the ratio of the second target parameter 4 divided by the first target parameter 5 is 0.8, and the server determines whether the target eye is in an open-eye state or a closed-eye state according to the ratio result of 0.8.
In the embodiment of the invention, the deep learning model is adopted to extract the characteristic points of the target image to be detected, the target eye closing ratio value is obtained based on the characteristic point coordinate calculation, and the target eye state is obtained according to the target eye closing ratio value, so that the problem of insufficient large data amount in the process of identifying the eye state is solved, and the accuracy of eye state detection is improved.
Referring to fig. 2, another embodiment of a method for detecting an eye state according to an embodiment of the present invention includes:
201. Establishing a standard image library, wherein the standard image library comprises a plurality of eye-opening state standard images and a plurality of eye-closing state standard images;
the server establishes a standard image library including a plurality of open-eye state standard images and a plurality of closed-eye state standard images.
The standard image library including a plurality of open-eye state standard images and a plurality of closed-eye state standard images is used to obtain the target closing ratio threshold.
202. Acquiring a target closure ratio threshold based on the plurality of open-eye state standard images and the plurality of closed-eye state standard images;
the server calculates a target closure ratio threshold according to the plurality of eye state standard images and the plurality of eye closure state standard images.
Specifically, initializing the number of error samples and a closure ratio critical value by a server to obtain a lower bound value of the number of error samples and an initial closure ratio threshold value; the server respectively selects a plurality of closing ratio reference thresholds within a preset range according to preset standards; the server calculates the corresponding error sample number in the plurality of eye-opening state standard images and the plurality of eye-closing state standard images according to each closing ratio reference threshold value to obtain a plurality of reference error sample numbers; selecting the reference error sample number smaller than or equal to the infinitesimal value of the error sample number from the multiple reference sample numbers as a target reference error sample number, and adjusting the infinitesimal value of the error sample number to be the same value as the target reference error sample number by the server to obtain the target error sample number; and finally, according to the target error sample number, the server determines a target closure ratio reference threshold corresponding to the target error sample number in a plurality of closure ratio reference thresholds, and adjusts the initial closure ratio threshold to be the same value as the target closure ratio reference threshold to obtain the target closure ratio threshold.
For ease of understanding, the following description is provided in connection with specific scenarios:
the server initializes the threshold of the closing ratio to 0, and then initializes the number of error samples to obtain an infinitesimal value of 30, and the specific function is as follows: initializing wrong 0=inf, thresh 0=0, wherein wrong0 is the number of error samples, and thresh0 is a closure ratio threshold; the server takes a closing ratio reference threshold value according to a preset standard of 0.001 at intervals within a preset range of 0-1 to obtain a plurality of closing ratio reference values of 0.001, 0.002 and 0.003. 0.003 … 1, and then counts the number of error samples in a plurality of eye-opening state standard images and a plurality of eye-closing state standard images according to each closing ratio reference value to obtain a plurality of reference error sample numbers, such as 15, 47, 50 and the like. When one of the reference error sample numbers 17 is smaller than the infinit value 30 of the error sample number, the server adjusts the infinit value of the error sample number to the reference error sample number 17 to obtain a target error sample number 17; the server selects a corresponding target closure ratio threshold value from a plurality of closure ratio reference threshold values according to the target error sample number 17, and the server adjusts the initial closure ratio threshold value 0 to be the target closure ratio threshold value 0.5 on the assumption that the closure ratio threshold value corresponding to the target error sample number 17 is 0.5.
203. Acquiring a target image to be detected, wherein the target image to be detected comprises a target eye;
the server acquires a target image to be detected including a target eye.
The server acquires images from terminals such as a mobile phone, a computer or a camera, and the target image to be detected can be an image from a shot eye or an image for locating an eye area in a face image.
It should be noted that, the format of the target image to be detected is not limited in this embodiment, and the size of the target image to be detected may be a conventional size (length is 24CM and width is 14 CM), and the size of the target image to be detected is not limited in this embodiment.
204. Extracting features of a target image to be detected by adopting a deep learning model to obtain a plurality of edge feature points, wherein the edge feature points comprise edge feature point coordinates;
and the server inputs the target image to be detected into the deep learning model for feature extraction to obtain a plurality of edge feature points comprising edge feature point coordinates.
The server performs a series of processing on the target image to be detected to obtain a processed target image, inputs the preprocessed target image into a convolutional neural network for feature extraction to obtain a plurality of edge feature points distributed around the eyes, and each edge feature point corresponds to one edge feature point coordinate.
For example, the target image to be detected is subjected to gray level processing, normalization and other processing to obtain a processed target image, and the preprocessed target image is input into a convolutional neural network to perform feature extraction to obtain P 0 、P 1 、P 2 、P 3 、P 4 、P 5 、P 6 And P 7 The 8 edge feature points and 8 edge feature point coordinates corresponding to the 8 edge feature points, P 0 、P 1 、P 2 、P 3 、P 4 、P 5 、P 6 And P 7 For 8 edge feature points distributed around the target eye.
Specifically, a server preprocesses a target image to be detected to obtain a preprocessed target image; secondly, the server inputs the preprocessed target image into a convolution layer of a convolution neural network to carry out convolution to obtain a first target image; then the server inputs the first target image into a pooling layer of the convolutional neural network for downsampling treatment to obtain a second target image; the server non-linearly maps the second target image into a target feature image; finally, the server extracts a plurality of edge feature points comprising the coordinates of the edge feature points from the target feature image.
In this embodiment, the preprocessing is a graying process and a normalizing process, and the image contrast after the graying process is strong, so that feature extraction is facilitated, and the normalizing process sets the image pixel value in the range of [0,1], so that feature extraction is facilitated. The convolutional neural network is CNN, which is a feedforward neural network comprising convolutional calculation and having a depth structure, and is one of representative algorithms of deep learning. The CNN has characteristic learning capability, can carry out translation invariant classification on input information according to a hierarchical structure of the CNN, and can realize the processing of a target image to be detected according to the capability of the CNN, so that a target characteristic image is obtained, and a server can detect edge characteristic points and corresponding point coordinates around eyes in the target characteristic image.
205. Fitting a plurality of edge feature point coordinates by adopting a least square method to obtain a first target parameter and a second target parameter;
and the server performs ellipse fitting on the plurality of edge characteristic points by adopting a least square method and a plurality of edge characteristic point coordinates, so as to obtain a first target parameter and a second target parameter.
The least square method is also called as a least squares method, and is a mathematical optimization technology. The method can simply and conveniently calculate unknown data by using a least square method by minimizing the square sum of errors to find the optimal function matching of the data, and the square sum of errors between the calculated data and actual data is minimized. The least squares method can also be used for curve fitting, and other optimization problems can also be expressed by the least squares method by minimizing energy or maximizing entropy. In short, the least squares method is similar to gradient descent, and is a common method for solving unconstrained optimization problems, and can also be used for curve fitting to solve regression problems. The least square method is essentially to minimize the "mean square error", which is 1/m of the sum of squares of the residuals, m being the number of samples, and the mean square error is the most commonly used performance metric in regression tasks.
In the scheme, a least square method is used for curve fitting, a server uses the least square method to carry out ellipse fitting on a plurality of edge characteristic points, a target ellipse equation can be fitted on a target eye based on the edge characteristic points and the least square method, and the server obtains a first target parameter and a second target parameter based on the target ellipse equation.
Specifically, in the target image to be detected, the server performs a preset function and a plurality of edge feature point coordinate pairs P according to the least square method 0 、P 1 、P 2 、P 3 、P 4 、P 5 、P 6 And P 7 The 8 edge feature points are subjected to ellipse fitting to obtain a target ellipse equation, and the corresponding preset function is as follows:the specific process for solving the target elliptic equation is as follows: the server calculates the minimum value of F (A, B, C, D, E, F) according to the 8 edge feature point coordinates and the preset function, calculates the values of A, B, C, D, E and F according to the minimum value of F (A, B, C, D, E, F), and finally calculates a target elliptic equation according to known A, B, C, D, E and F to obtain the target elliptic equation as follows: />After the target elliptic equation is calculated, the server reads the first target parameter a and the second target parameter b from the target elliptic equation.
206. The state of the target eye is acquired based on the first target parameter and the second target parameter, and the state of the target eye is an open-eye state or a closed-eye state.
The server acquires the open-eye state or the closed-eye state of the target eye according to the first target parameter and the second target parameter.
The server calculates a ratio according to the first target parameter and the second target parameter, and judges whether the state of the target eye is an open-eye state or a closed-eye state based on the parameter ratio. For example, the first target parameter is 5, the second target parameter is 4, the ratio of the second target parameter 4 divided by the first target parameter 5 is 0.8, and the server determines whether the target eye is in an open-eye state or a closed-eye state according to the ratio result of 0.8.
Specifically, the server calculates a ratio of the eye closure ratio of the first target parameter to the eye closure ratio of the second target parameter to obtain a target eye closure ratio; the server obtains a target eye state identification based on the target eye state identification and determines a first eye state identification or a second eye state identification based on the target eye state identification.
The specific steps of determining the first eye state identifier or the second eye state identifier based on the target eye state identifier are as follows: the server reads the target eye closure ratio and judges whether the target eye closure ratio is greater than or equal to a target closure ratio threshold; when the target eye closure ratio is greater than or equal to the target closure ratio threshold, the server obtains a first eye state identifier capable of determining that the state of the target eye is a open eye state; when the target eye closure ratio is less than the target closure ratio threshold, the server obtains a second eye state identification capable of determining that the state of the target eye is the closed eye state.
For ease of understanding, the following description is provided in connection with the specific case:
assume that the relationship of the target eye closure ratio, target closure ratio threshold, and eye state identification is as follows:
where EOCRA is eye closure ratio, f (EOCRA) is state of the target eye, bestEOCRA target closure ratio threshold, 1 is first eye state identification, 0 is second eye state identification.
Assuming that the target eye closure ratio threshold is 0.5, the first target parameter is 5, and the second target parameter is 4, the ratio of the second target parameter to the first target parameter is calculated to obtain a target eye closure ratio of 0.8. The target eye closure ratio 0.8 is greater than the target eye closure ratio threshold value 0.5, and the server obtains a first eye state identification, and determines that the state of the target eye is a open eye state according to the first eye state identification.
In the embodiment of the invention, the deep learning model is adopted to extract the characteristic points of the target image to be detected, the target eye closing ratio value is obtained based on the characteristic point coordinate calculation, and the target eye state is obtained according to the target eye closing ratio value, so that the problem of insufficient large data amount in the process of identifying the eye state is solved, and the accuracy of eye state detection is improved.
The method for detecting an eye state in the embodiment of the present invention is described above, and the following describes a device for detecting an eye state in the embodiment of the present invention, referring to fig. 3, and one embodiment of the device for detecting an eye state in the embodiment of the present invention includes:
an image acquisition module 301, configured to acquire a target image to be detected, where the target image to be detected includes a target eye;
the feature extraction module 302 is configured to perform feature extraction on the target image to be detected by using a deep learning model, so as to obtain a plurality of edge feature points, where the edge feature points include edge feature point coordinates;
a parameter obtaining module 303, configured to fit the plurality of edge feature point coordinates by using a least square method, so as to obtain a first target parameter and a second target parameter;
an eye state obtaining module 304, configured to obtain a state of a target eye based on the first target parameter and the second target parameter, where the state of the target eye is an open-eye state or a closed-eye state.
In the embodiment of the invention, the deep learning model is adopted to extract the characteristic points of the target image to be detected, the target eye closing ratio value is obtained based on the characteristic point coordinate calculation, and the target eye state is obtained according to the target eye closing ratio value, so that the problem of insufficient large data amount in the process of identifying the eye state is solved, and the accuracy of eye state detection is improved.
Referring to fig. 4, another embodiment of an eye state detection device according to an embodiment of the present invention includes:
an image acquisition module 301, configured to acquire a target image to be detected, where the target image to be detected includes a target eye;
the feature extraction module 302 is configured to perform feature extraction on the target image to be detected by using a deep learning model, so as to obtain a plurality of edge feature points, where the edge feature points include edge feature point coordinates;
a parameter obtaining module 303, configured to fit the plurality of edge feature point coordinates by using a least square method, so as to obtain a first target parameter and a second target parameter;
an eye state obtaining module 304, configured to obtain a state of a target eye based on the first target parameter and the second target parameter, where the state of the target eye is an open-eye state or a closed-eye state.
Optionally, the feature extraction module 302 may be further specifically configured to:
preprocessing the target image to be detected to obtain a preprocessed target image;
inputting the preprocessed target image into a convolutional neural network for convolution to obtain a first target image;
pooling the first target image in the convolutional neural network to obtain a second target image;
Performing nonlinear mapping on the second target image to obtain a target characteristic image;
and extracting a plurality of edge feature points corresponding to the target eyes from the target feature image, wherein the edge feature points comprise edge feature point coordinates.
Optionally, the parameter obtaining module 303 may be further specifically configured to:
in the target image to be detected, performing ellipse fitting on the edge feature points by adopting the least square method based on the edge feature point coordinates to obtain a target ellipse equation;
and extracting a first target parameter and a second target parameter from the target elliptic equation.
Optionally, the eye state acquisition module 304 includes:
a closing ratio calculating unit 3041, configured to perform an eye closing ratio calculation on the first target parameter and the second target parameter to obtain a target eye closing ratio;
an eye state obtaining unit 3042, configured to obtain a target eye state identifier according to the target eye closing ratio, and determine a state of the target eye based on the target eye state identifier, where the target eye state identifier is a first eye state identifier or a second eye state identifier, and the state of the target eye is a open eye state or a closed eye state.
Optionally, the eye state acquiring unit 3042 may be specifically configured to:
reading the target eye closure ratio and judging whether the target eye closure ratio is greater than or equal to a target closure ratio threshold;
if the target eye closing ratio is greater than or equal to the target closing ratio threshold, a first eye state identification is obtained, and the state of the target eye is determined to be a tension state based on the first eye state identification;
and if the target eye closing ratio is smaller than the target closing ratio threshold, obtaining a second eye state identification, and determining that the state of the target eye is an eye closing state based on the second eye state identification.
Optionally, the device for detecting an eye state further includes:
a gallery creation module 305, configured to create a standard image gallery, where the standard image gallery includes a plurality of eye-open state standard images and a plurality of eye-closed state standard images;
a target threshold acquisition module 306 for acquiring a target closure ratio threshold based on the plurality of open eye state standard images and the plurality of closed eye state standard images.
Optionally, the target threshold obtaining module 306 may be further specifically configured to:
initializing the number of error samples and a closure ratio critical value to obtain a lower bound value of the number of error samples and an initial closure ratio threshold value;
Respectively selecting a plurality of closure ratio reference thresholds within a preset range according to preset standards;
counting the number of error samples corresponding to each closing ratio reference threshold in the plurality of eye-opening state standard images and the plurality of eye-closing state standard images according to each closing ratio reference threshold in the plurality of closing ratio reference thresholds, so as to obtain a plurality of reference error sample numbers;
comparing the infinitesimal value of the error sample number with the plurality of reference error sample numbers to obtain a target reference error sample number, and adjusting the infinitesimal value of the error sample number to the target reference error sample number to obtain a target error sample number, wherein the target reference error sample number is the reference error sample number smaller than or equal to the infinitesimal value of the error sample number;
and determining a target closure ratio reference threshold value from the closure ratio reference thresholds based on the target error sample number, and adjusting the initial closure ratio threshold value to the target closure ratio reference threshold value to obtain a target closure ratio threshold value.
In the embodiment of the invention, the deep learning model is adopted to extract the characteristic points of the target image to be detected, the target eye closing ratio value is obtained based on the characteristic point coordinate calculation, and the target eye state is obtained according to the target eye closing ratio value, so that the problem of insufficient large data amount in the process of identifying the eye state is solved, and the accuracy of eye state detection is improved.
The above detailed description of the eye state detection device in the embodiment of the present invention from the point of view of the modularized functional entity in fig. 3 and fig. 4 is described below in detail from the point of view of hardware processing.
Fig. 5 is a schematic structural diagram of an eye state detection device 500 according to an embodiment of the present invention, where the eye state detection device 500 may have a relatively large difference due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing application programs 533 or data 532. Wherein memory 520 and storage medium 530 may be transitory or persistent storage. The program stored on the storage medium 530 may include one or more modules (not shown), each of which may include a series of instruction operations in the eye state detection apparatus 500. Still further, the processor 510 may be configured to communicate with the storage medium 530 and execute a series of instruction operations in the storage medium 530 on the eye state detection device 500.
The eye state detection device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input/output interfaces 560, and/or one or more operating systems 531, such as Windows Serve, mac OS X, unix, linux, freeBSD, and the like. It will be appreciated by those skilled in the art that the eye condition detection device structure shown in fig. 5 does not constitute a limitation of the eye condition detection device and may include more or fewer components than shown, or may combine certain components, or may be a different arrangement of components.
The present invention also provides a computer readable storage medium, which may be a non-volatile computer readable storage medium, or a volatile computer readable storage medium, having stored therein instructions that, when executed on a computer, cause the computer to perform the steps of the method for detecting an eye state.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (9)
1. A method for detecting an eye state, the method comprising:
acquiring a target image to be detected, wherein the target image to be detected comprises a target eye;
extracting features of the target image to be detected by adopting a deep learning model to obtain a plurality of edge feature points, wherein the edge feature points comprise edge feature point coordinates;
fitting the edge feature point coordinates by using a least square method to obtain a target elliptic equation, and reading a first target parameter and a second target parameter of the target elliptic equation;
acquiring a state of a target eye based on the first target parameter, the second target parameter and a target closure ratio threshold, wherein the state of the target eye is an open-eye state or a closed-eye state;
and extracting features of the target image to be detected by adopting a deep learning model to obtain a plurality of edge feature points, wherein the edge feature points comprise edge feature point coordinates, and the edge feature point coordinates comprise:
preprocessing the target image to be detected to obtain a preprocessed target image;
inputting the preprocessed target image into a CNN convolutional neural network to carry out convolution to obtain a first target image;
Pooling the first target image in the CNN convolutional neural network to obtain a second target image;
performing nonlinear mapping on the second target image to obtain a target characteristic image;
and extracting a plurality of edge feature points corresponding to the target eyes from the target feature image, wherein the edge feature points comprise edge feature point coordinates.
2. The method according to claim 1, wherein fitting the plurality of edge feature point coordinates using a least squares method to obtain a target elliptic equation, and reading a first target parameter and a second target parameter of the target elliptic equation comprises:
in the target image to be detected, performing ellipse fitting on the edge feature points by adopting the least square method based on the edge feature point coordinates to obtain a target ellipse equation;
and extracting a first target parameter and a second target parameter from the target elliptic equation.
3. The method according to claim 1, wherein the acquiring the state of the target eye based on the first target parameter, the second target parameter, and a target closure ratio threshold, the state of the target eye being an open-eye state or a closed-eye state includes:
Performing eye closure ratio calculation on the first target parameter and the second target parameter to obtain a target eye closure ratio;
and acquiring a target eye state identifier according to the target eye closing ratio and a target closing ratio threshold, and determining the state of the target eye based on the target eye state identifier, wherein the target eye state identifier is a first eye state identifier or a second eye state identifier, and the state of the target eye is an open eye state or a closed eye state.
4. The method according to claim 3, wherein the obtaining a target eye state identifier according to the target eye closure ratio and a target closure ratio threshold, and determining the state of the target eye based on the target eye state identifier, the target eye state identifier being a first eye state identifier or a second eye state identifier, the state of the target eye being a open eye state or a closed eye state, comprises:
reading the target eye closure ratio and judging whether the target eye closure ratio is greater than or equal to a target closure ratio threshold;
if the target eye closing ratio is greater than or equal to the target closing ratio threshold, a first eye state identification is obtained, and the state of the target eye is determined to be a tension state based on the first eye state identification;
And if the target eye closing ratio is smaller than the target closing ratio threshold, obtaining a second eye state identification, and determining that the state of the target eye is an eye closing state based on the second eye state identification.
5. The method according to any one of claims 1 to 4, characterized in that before the acquisition of the target image to be detected, the target image to be detected including the target eye, the method further comprises:
establishing a standard image library, wherein the standard image library comprises a plurality of eye-opening state standard images and a plurality of eye-closing state standard images;
a target closure ratio threshold is acquired based on the plurality of open eye state standard images and the plurality of closed eye state standard images.
6. The method of claim 5, wherein the obtaining a target closure ratio threshold based on the plurality of open eye state standard images and the plurality of closed eye state standard images comprises:
initializing the number of error samples and a closure ratio critical value to obtain a lower bound value of the number of error samples and an initial closure ratio threshold value;
respectively selecting a plurality of closure ratio reference thresholds within a preset range according to preset standards;
Counting the number of error samples corresponding to each closing ratio reference threshold in the plurality of eye-opening state standard images and the plurality of eye-closing state standard images according to each closing ratio reference threshold in the plurality of closing ratio reference thresholds, so as to obtain a plurality of reference error sample numbers;
comparing the infinitesimal value of the error sample number with the plurality of reference error sample numbers to obtain a target reference error sample number, and adjusting the infinitesimal value of the error sample number to the target reference error sample number to obtain a target error sample number, wherein the target reference error sample number is the reference error sample number smaller than or equal to the infinitesimal value of the error sample number;
and determining a target closure ratio reference threshold value from the closure ratio reference thresholds based on the target error sample number, and adjusting the initial closure ratio threshold value to the target closure ratio reference threshold value to obtain a target closure ratio threshold value.
7. An eye state detection device, characterized in that the eye state detection device comprises:
the image acquisition module is used for acquiring a target image to be detected, wherein the target image to be detected comprises a target eye;
The feature extraction module is used for extracting features of the target image to be detected by adopting a deep learning model to obtain a plurality of edge feature points, wherein the edge feature points comprise edge feature point coordinates;
the parameter acquisition module is used for fitting the coordinates of the edge feature points by adopting a least square method to obtain a target elliptic equation, and reading a first target parameter and a second target parameter of the target elliptic equation;
an eye state acquisition module, configured to acquire a state of a target eye based on the first target parameter, the second target parameter, and a target closure ratio threshold, where the state of the target eye is an open-eye state or a closed-eye state;
the feature extraction module is specifically configured to:
preprocessing the target image to be detected to obtain a preprocessed target image;
inputting the preprocessed target image into a CNN convolutional neural network to carry out convolution to obtain a first target image;
pooling the first target image in the CNN convolutional neural network to obtain a second target image;
performing nonlinear mapping on the second target image to obtain a target characteristic image;
and extracting a plurality of edge feature points corresponding to the target eyes from the target feature image, wherein the edge feature points comprise edge feature point coordinates.
8. An eye state detection apparatus, characterized in that the eye state detection apparatus comprises: a memory and at least one processor, the memory having instructions stored therein, the memory and the at least one processor being interconnected by a line;
the at least one processor invokes the instructions in the memory to cause the eye state detection device to perform the eye state detection method of any one of claims 1-6.
9. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the method of detecting an eye state according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010301042.4A CN111488844B (en) | 2020-04-16 | 2020-04-16 | Eye state detection method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010301042.4A CN111488844B (en) | 2020-04-16 | 2020-04-16 | Eye state detection method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111488844A CN111488844A (en) | 2020-08-04 |
| CN111488844B true CN111488844B (en) | 2023-10-20 |
Family
ID=71812792
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010301042.4A Active CN111488844B (en) | 2020-04-16 | 2020-04-16 | Eye state detection method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111488844B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103400471A (en) * | 2013-08-12 | 2013-11-20 | 电子科技大学 | Detecting system and detecting method for fatigue driving of driver |
| CN108108684A (en) * | 2017-12-15 | 2018-06-01 | 杭州电子科技大学 | A kind of attention detection method for merging line-of-sight detection |
| CN109460704A (en) * | 2018-09-18 | 2019-03-12 | 厦门瑞为信息技术有限公司 | A kind of fatigue detection method based on deep learning, system and computer equipment |
| CN109583292A (en) * | 2018-10-11 | 2019-04-05 | 杭州电子科技大学 | A kind of visibility region detection method |
-
2020
- 2020-04-16 CN CN202010301042.4A patent/CN111488844B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103400471A (en) * | 2013-08-12 | 2013-11-20 | 电子科技大学 | Detecting system and detecting method for fatigue driving of driver |
| CN108108684A (en) * | 2017-12-15 | 2018-06-01 | 杭州电子科技大学 | A kind of attention detection method for merging line-of-sight detection |
| CN109460704A (en) * | 2018-09-18 | 2019-03-12 | 厦门瑞为信息技术有限公司 | A kind of fatigue detection method based on deep learning, system and computer equipment |
| CN109583292A (en) * | 2018-10-11 | 2019-04-05 | 杭州电子科技大学 | A kind of visibility region detection method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111488844A (en) | 2020-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7912253B2 (en) | Object recognition method and apparatus therefor | |
| Kawaguchi et al. | Detection of eyes from human faces by Hough transform and separability filter | |
| US7995805B2 (en) | Image matching apparatus, image matching method, computer program and computer-readable storage medium | |
| CN110659665B (en) | Model construction method of different-dimension characteristics and image recognition method and device | |
| WO2015149696A1 (en) | Method and system for extracting characteristic of three-dimensional face image | |
| CN109376604B (en) | Age identification method and device based on human body posture | |
| CN101408929A (en) | Multiple-formwork human face registering method and apparatus for human face recognition system | |
| CN111445459A (en) | Image defect detection method and system based on depth twin network | |
| WO2013122009A1 (en) | Reliability level acquisition device, reliability level acquisition method and reliability level acquisition program | |
| CN112784712B (en) | Missing child early warning implementation method and device based on real-time monitoring | |
| CN109145704B (en) | Face portrait recognition method based on face attributes | |
| CN116386120A (en) | Noninductive monitoring management system | |
| CN111178187A (en) | Face recognition method and device based on convolutional neural network | |
| CN114359787A (en) | Target attribute identification method and device, computer equipment and storage medium | |
| CN111382638A (en) | Image detection method, device, equipment and storage medium | |
| WO2015131710A1 (en) | Method and device for positioning human eyes | |
| CN111488844B (en) | Eye state detection method, device, equipment and storage medium | |
| CN112101293A (en) | Facial expression recognition method, device, equipment and storage medium | |
| CN110866470A (en) | Face anti-counterfeiting detection method based on random image characteristics | |
| WO2002080088A1 (en) | Method for biometric identification | |
| CN113269136B (en) | Off-line signature verification method based on triplet loss | |
| Zaghetto et al. | Touchless multiview fingerprint quality assessment: rotational bad-positioning detection using Artificial Neural Networks | |
| KR100711223B1 (en) | Face recognition method using Zernike/LDA and recording medium storing the method | |
| Pujol et al. | A connectionist computational method for face recognition | |
| CN117133022B (en) | Color image palm print recognition method and device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |