CN110428908B - Eyelid motion function evaluation system based on artificial intelligence - Google Patents
Eyelid motion function evaluation system based on artificial intelligence Download PDFInfo
- Publication number
- CN110428908B CN110428908B CN201910704037.5A CN201910704037A CN110428908B CN 110428908 B CN110428908 B CN 110428908B CN 201910704037 A CN201910704037 A CN 201910704037A CN 110428908 B CN110428908 B CN 110428908B
- Authority
- CN
- China
- Prior art keywords
- video
- detected object
- face
- movement
- eyelid
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Biomedical Technology (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Public Health (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Molecular Biology (AREA)
- Epidemiology (AREA)
- Computing Systems (AREA)
- Primary Health Care (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Databases & Information Systems (AREA)
- Pathology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Evolutionary Biology (AREA)
- Ophthalmology & Optometry (AREA)
- Heart & Thoracic Surgery (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Veterinary Medicine (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an eyelid movement abnormity evaluation system based on artificial intelligence, which comprises: the detected object acquisition module: the method comprises the steps of obtaining a face video only containing a detected object from an input face action video of the detected object; eye and designated part positioning module: the eye movement video and the specified part linkage video only comprising the detected object are obtained; TSN model: the eye movement video and the specified part linkage video are processed, and action signals of the eyes and the specified part are output; a probability output module: the probability signal is used for outputting a probability signal of judging the occurrence of eyelid movement abnormality by the computer of each frame of picture; and the evaluation module is used for obtaining the abnormal grade according to the probability signal of the occurrence of the eyelid movement abnormality and the eyelid movement abnormality probability judgment mechanism. The invention has the advantages of nationwide acceptability, convenience, accuracy, objectivity and repeatability, and strong clinical usability.
Description
Technical Field
The invention relates to the field of eyelid movement abnormal degree evaluation, in particular to an eyelid movement function evaluation system based on artificial intelligence.
Background
The existing techniques for observing eyelid movement abnormality mainly comprise neuromuscular recorder technology, brainstem Magnetic Resonance Angiography (MRA) examination and wearing of novel glasses for recording eyelid movement frequency function. The neuromuscular recording instrument technology has the defects of complex technical operation, time and labor waste and low adaptability to the old and children; meanwhile, eyelid movement dysfunction caused by different reasons is different from pathogenesis inducement, frequency, binocular symmetry, eyelid closure degree and the like, so that the measurement result is easily influenced by subjective factors of detection personnel, the result lacks repeatability and stability, and early detection in a large range in the crowd is difficult to carry out. The examination of the magnetic resonance angiography of the brainstem (MRA) can only be used to determine whether blepharospasm is associated with blood vessels with abnormal facial nerves, and is expensive and requires a high degree of patient compliance. Wear the novel glasses of record eyelid movement frequency function, can only take notes the closed condition of eyelid, can not take notes the condition of removing the linkage of eyes and other positions, and glasses are with high costs and the popularization dynamics is lower, and a lot of children are lower to the cooperation degree of wearing glasses, and can not present the most natural state of using the eye.
The prior art has higher requirements on testing environment, detection equipment and detection personnel, and has higher cost of manpower, material resources and the like.
The prior art has limitations in application range: the neuromuscular recorder needs to place a stimulating electrode on the supraorbital nerve and the middle part of the lower eyelid of both eyes is mediated by the conductive paste, so as to record the motion waveform of the orbicularis muscles of both eyes. However, the result has larger error due to the low fit of the examinee, especially children; the brain stem Magnetic Resonance Angiography (MRA) technology mainly observes the correlation between blood vessel abnormality at the brain stem part and the facial nerve brain stem. The cause of the disease is that facial nerve is pressed by blood vessels or tumors in cerebellar and pontocerebral horn. The application range is limited, and the method is mainly used for excluding differential diagnosis; wearing the novel glasses with the function of recording eyelid movement frequency requires that the examinee can continuously wear the glasses, so that the examinee is easy to show unnatural states such as tension, curiosity and the like in detection, and in addition, the recording process of eyelid movement reflex can be seriously influenced by random taking off and wearing the glasses, thereby seriously influencing the accuracy and the effectiveness of the inspection result. The test results of the prior art are subjective: in the prior art, the recording modes of eyelid movement frequency, binocular symmetry, eyelid closing force and the like caused by different pathogenic causes are different, so that the interpretation of the measurement result is influenced by subjective factors of detection personnel, and the repeatability of the measurement result is low.
The prior art test environment has limitations: due to the requirements of the examination process on test equipment and environment, the existing examination and evaluation technology cannot be developed in the daily life environment, so that large-scale examination of dry eye, eyelid movement reflex abnormality, blepharospasm and the like cannot be developed at the present stage, such as examination outside a doctor's office and in daily life.
In conclusion, due to the particularity of the measurement population (children, the elderly and hyperthyroidism patients are common), the traditional measurement means limits the measurement of the eyelid function, so that the large-area eyelid function screening is difficult to develop in the population on the basis of the prior art, a large number of patients with eyelid dysfunction miss the optimal treatment opportunity, irreversible corneal or conjunctival injuries are caused finally, and great pressure is caused to family, medical and social resources. Therefore, achieving large-scale early screening of eyelid function in a population is an important prerequisite to avoid visual impairment and to identify facial neuropathy.
Deep learning techniques have been widely used in the field of computer vision. As one of deep learning techniques, a CNN (Convolutional Neural Network) model greatly improves the accuracy of image classification, and brings qualitative changes to an image classification task. A suitable CNN model may be designed for any one database, and may be used to train the samples in the database, thereby obtaining the association between the samples in the database and the labels of the samples. Here, the samples in the database may be images or videos.
The early-stage research of the technical team of the invention discovers that when eye diseases such as xerophthalmia and ocular surface inflammation and the like of patients with eyelid movement abnormality occur and facial nerve, trigeminal nerve and brainstem lesions, the facial movement modality of the patients with eyelid movement abnormality is different from that of normal people. When corresponding eye diseases or neuropathy occur, different facial motion modes of normal people appear in the early stage of the diseases, the eye diseases or the neuropathy are reflected in that frequent blinks are larger than 15 times per mi, or forced blinks or upper and lower eyelids are incompletely closed during blinking, and key parts such as eyebrows, nasal wings, mouths and the like can be linked to move when the disease is serious, for example, the mouths are enlarged after the forced blinking. Therefore, the general eyelid movement detection device can perform detection evaluation on the eyes only, which causes a technical problem that the evaluation result is inaccurate. The technical scheme of the invention trains an artificial intelligence algorithm to recognize actions by utilizing a large amount of video data collected in the early stage and indicating normal and abnormal eyelid functions, and judges whether the eyelid movement of the examinee is abnormal or not by recognizing the actions of eyes and key parts.
Disclosure of Invention
The invention aims to solve the technical problems that the existing eyelid movement abnormality detection equipment has higher requirements on a test environment, detection equipment and detection personnel, the cost of manpower and material resources is higher, and the detection result is not accurate enough on one side; the eyelid movement function evaluation system based on artificial intelligence is high in all-people acceptability, convenience, objectivity, accuracy and repeatability and clinical usability.
In order to solve the above problems, there is provided an eyelid movement function evaluating system based on artificial intelligence, comprising:
the detected object acquisition module: the system comprises a video acquisition unit, a video processing unit and a control unit, wherein the video acquisition unit is used for acquiring a face feature area of each frame of a detected object from an input face action video of the detected object, cutting off the face feature area of each frame of a non-detected object and acquiring a face video only containing the detected object;
eye and designated part positioning module: eyes and designated parts used for locating and image processing the facial video, the designated part has one or more key parts corresponding to the examined object; cutting to generate an eye movement video and a designated part linkage video of the detected object;
the TSN model based on the convolutional neural network: the eye movement video and the specified part linkage video are processed, and action signals of the eyes and the specified part are output;
a probability output module: the action signal output by the TSN is used as input for outputting a probability signal for judging the occurrence of eyelid movement abnormality by a computer of each frame of picture; the probability signal of the abnormal eyelid movement comprises a probability signal of abnormal eye movement when the eyelid movement is abnormal and a probability signal of abnormal linkage movement of each designated part;
the evaluation module is used for obtaining an abnormal grade according to the probability signal that the eye movement is abnormal, the probability signal that each appointed part moves and abnormal linkage action occurs and an eyelid movement abnormal probability judgment mechanism and judging the severity of the eyelid movement abnormality of the detected object;
the examined object acquisition module comprises:
the fast-RCNN neural network model is used for positioning the face area of each frame of the detected object in the face action video and inputting the face area into the face recognition model frame by frame;
the face recognition model comprises a left face, a front face and a right face classification model and is used for recognizing the facial features of the detected object in different directions in each face region;
the eye and designated part positioning module comprises a CNN structure of CPM, a CNN structure of PAF and an image processing module.
Particularly, the duration of the face motion video is more than or equal to 5 min.
Specifically, the face motion video is a video in which the face of the subject is located at the center of the lens.
Specifically, a specific method for obtaining a facial feature region of each frame of a subject from an input facial motion video of the subject is as follows:
(1) inputting the facial action video into a fast-RCNN neural network model, positioning the face region of each frame of the detected object in the facial action video, and inputting the face region of each frame into a face recognition model;
(2) the face recognition model recognizes the facial features of the detected object in different directions in each frame of face region;
(3) and (3) performing the step (1) and the step (2) on each frame of the facial motion video to obtain a facial feature region of the detected object of each frame.
Specifically, the designated and critical areas include the head, nose, mouth, and eyebrows.
Specifically, the specific method for locating the designated part of the face video is as follows:
(1) obtaining a heat map of joint points of the eyes, the eyebrows, the nose and the mouth of the face video through a CNN structure of CPM to judge the joint points;
(2) obtaining 2D vector geometry through a CNN structure of the PAF to code the motion directions of the eyebrows, the nose and the mouth of the face video and position the eyebrows, the nose and the mouth;
(3) learning a video containing eyes through an image processing module, and positioning the eyes; and intercepting an image of the face video at the moment of eyelid closure, analyzing the degree of eyelid closure in the image, and processing the eyelid part and the pupil part of the image by using an image gray processing technology.
Particularly, the method for processing the eye movement video and the designated part linkage video and outputting the action signal by the TSN model is realized by adopting a basic framework of a BN-acceptance v3 convolutional neural network, and the specific method is as follows:
(1) sparse sampling is carried out on the input eye movement video and the designated part linkage video to generate a series of short segments as input;
(2) and carrying out convolution operation on the RGB format image and the optical flow format image with proper sizes in the short segment through a time flow convolution neural network and a space flow convolution neural network respectively to obtain action signals of the eyes and the designated part, and outputting the action signals.
Specifically, the specific method for outputting the probability signal of the occurrence of the eyelid movement abnormality by the probability output module is as follows:
(1) taking action signals output by a TSN model as input, training a machine learning classifier, combining the category score output of a plurality of short segments to obtain consensus on category hypothesis among the short segments, obtaining a segment consensus function G, and predicting probability signals of each behavior category of the whole segment of video belonging to eyes and a specified part by a prediction function H based on the consensus; the behavior types comprise the behavior type of abnormal actions of eyes when the eyelid movement of the detected object is abnormal and the behavior type of abnormal linkage actions of each appointed part;
(2) and outputting a probability signal of abnormal action of eyes and a probability signal of abnormal linkage action of each designated part.
Particularly, the eyelid movement abnormality level judgment mechanism comprises an eye movement judgment mechanism and a linkage action judgment mechanism;
the eye movement judging mechanism is that a threshold value P signal is set to divide the eye movement into two grades of abnormal movement and abnormal movement, and the two grades are compared with a probability signal of abnormal movement of the eyes; if the probability signal of the abnormal action of the eyes is higher than the threshold P signal, the eyes are judged to have abnormal grade; if the probability signal of the abnormal action of the eyes is lower than the threshold P signal, the abnormal grade is the abnormal grade;
when the grade of the judging mechanism for judging the abnormal eye movement is an abnormal grade, executing a linkage action judging mechanism;
the linkage action judging mechanism is used for setting a threshold signal of abnormal motion of each designated part and is divided into a light level and a heavy level; if the probability signal of abnormal linkage action of the movement of one designated part is higher than the threshold signal of abnormal movement of the designated part, the abnormal grade is a light grade, otherwise, the abnormal grade is a heavy grade.
The method for evaluating according to the eyelid movement function evaluating system comprises the following steps:
s1, obtaining a face video only containing the detected object from the input face action video of the detected object through a detected object obtaining module;
s2, positioning the eyes and the designated parts of the face video through an eye and designated part positioning module, and cutting to generate an eye movement video and a designated part linkage video of the detected object;
s3, processing the eye movement video and the specified part linkage video through the TSN model and outputting action signals of the eyes and the specified part;
s4, inputting the action signal output by the TSN into a probability output module, and outputting a probability signal for judging the occurrence of eyelid movement abnormality by the computer of each frame of picture;
and S5, obtaining an abnormal grade through an evaluation module according to the probability signal of the occurrence of the eyelid movement abnormality and the eyelid movement abnormality probability judgment mechanism, and judging the severity of the eyelid movement abnormality of the detected object.
The invention has the beneficial effects that:
1. the invention has the advantages of national acceptability: in the traditional detection scheme, because the matching degree of the children or the old is low in the non-detection process, the emotions such as tension, fear and the like are often shown during detection, and the accuracy of the detection result is reduced along with the resistance behaviors such as crying and screaming. According to the technical scheme, detection methods which can be accepted by all ages are researched and developed, the form of recording facial action expression videos by the camera is adopted, and the degree of adaptability and accuracy of the examinee to the test process are improved.
2. The invention has the advantages of convenience: the traditional detection scheme is limited by conditions such as detection equipment, environment, testers and the like, and cannot be developed on a large scale. The technical scheme of the invention has low requirements on detection conditions, the detection process is convenient and quick, the large-scale screening of people can be realized, and the long-term tracking and evaluation of patients can be realized.
3. The invention has objectivity and repeatability: the traditional detection side examinee is often difficult to cooperate with measurement, and the measurement result caused by subjective factors of measurement personnel and the like is lack of objectivity and is often unrepeatable. According to the method, the artificial intelligence technology is used for action recognition through the high relevance of the eyelid movement and the facial expression of the examinee. The stability and repeatability of the artificial intelligence technology are utilized, so that the inspection result is more objective.
4. The invention has the following accuracy: the eye movement detection method and the eye movement detection device combine the eye movement and the linkage movement of the key part for action recognition, adopt an eyelid movement abnormality grade judgment mechanism to obtain the abnormality grade, and judge the severity of the eyelid movement abnormality of the detected object, and are more accurate compared with the detection and evaluation of the eye by a common eyelid movement detection device.
5. The invention has strong clinical usability: the method can analyze the condition of the eyelid movement function in the video of the examined person by observing the most natural eyelid movement of the patient in various natural states (near vision, far vision, outside of the clinic, inside of the clinic and the like) and simultaneously utilizing an artificial intelligence method, thereby excluding the diagnosis of doctors and checking the subjectivity of the operation of the doctors and further obtaining the most ideal diagnosis means.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a block diagram of a system according to an embodiment of the invention;
FIG. 2 is a flow chart of a method of an embodiment of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, and the scope of the present invention will be more clearly and clearly defined.
As shown in fig. 1, the eyelid movement function evaluation system based on artificial intelligence of the present embodiment includes: the system comprises a detected object acquisition module, an eye and designated part positioning module, a TSN (time delay network) model, a probability output module and an evaluation module.
The detected object acquisition module: the method is used for obtaining the facial feature region of each frame of the detected object from the input facial motion video of the detected object, cutting off the facial feature region of each frame of the non-detected object and obtaining the facial video only containing the detected object.
Eye and designated part positioning module: eyes and designated parts used for locating and image processing the facial video, the designated part has one or more key parts corresponding to the examined object; and cutting to generate an eye movement video and a designated part linkage video of the detected object.
The TSN model based on the convolutional neural network: the eye movement video and the designated part linkage video are processed, and the eye movement video and the designated part action signals are output.
A probability output module: the action signal output by the TSN is used as input for outputting a probability signal for judging the occurrence of eyelid movement abnormality by a computer of each frame of picture; the probability signal of the abnormal eyelid movement comprises a probability signal of abnormal eye movement when the eyelid movement is abnormal and a probability signal of abnormal linkage movement of each designated part;
and the evaluation module is used for obtaining an abnormal grade according to the probability signal that the eye movement is abnormal, the probability signal that each appointed part moves and the probability judgment mechanism of the eyelid movement abnormality, and judging the severity of the eyelid movement abnormality of the detected object.
The examined object acquisition module comprises:
the fast-RCNN neural network model is used for positioning the face area of each frame of the detected object in the face action video and inputting the face area into the face recognition model frame by frame;
and the face recognition model comprises a left face classification model, a front face classification model and a right face classification model and is used for recognizing the facial features of the detected object in different directions in each face region.
The eye and designated part positioning module comprises a CNN structure of CPM, a CNN structure of PAF and an image processing module.
As shown in fig. 2, the method for evaluating the eyelid movement function evaluating system according to the above embodiment includes the following steps:
s1, importing or directly inputting the facial motion video of the detected object to a detected object acquisition module.
The recording of the facial motion video needs to be performed in an environment where the detected object is in a natural state and the ambient light is sufficient. The camera, not limited to a mobile phone camera, a computer camera and a hidden camera are arranged on a proper height position, so that the detected object is opposite to the camera. The hidden camera is mainly installed in a doctor's office or a waiting room and used for recording facial action expressions of a patient during waiting and talking with a doctor; the mobile phone or the computer front camera is mainly used for facial movements of a detected person in near vision, and can be used in hospitals and homes. When shooting is used and the detected object is a child, the mobile phone is placed at a proper height, corresponding software is opened, the head of the detected object and the distance between the head of the detected object and the mobile phone are adjusted according to the guidance of the software, so that the face of the detected object is opposite to the front camera of the mobile phone, and the face of the detected object is ensured to be positioned in the center of the screen of the mobile phone in the video recording process; at the moment, a start key of the software is clicked, the software plays a section of animation, meanwhile, the front-facing camera automatically starts to collect the video of the detected object when the detected object watches the mobile phone, and the duration of the recorded facial action video is more than 5 min. After the facial action video is shot, the facial action video can be imported or directly input into a detected object acquisition module in application software for realizing the embodiment.
S2, obtaining a face video only containing the detected object through a detected object obtaining module, and specifically comprising the following steps:
s21, positioning the face area of each frame of the detected object in the face action video through a fast-RCNN neural network model of the detected object acquisition module, and inputting the face area into a face recognition model frame by frame.
The Fast-RCNN neural Network model is a neural Network model for extracting a target detection Region based on deep learning, firstly, a candidate Region generation Network (RPN) is utilized to extract a face candidate Region, then, the face candidate Region is subjected to convolution operation to extract face features, and finally, a Fast Region convolution neural Network (Fast R-CNN) of joint training is utilized to perform face recognition to obtain a face Region.
And S22, the face characteristics of the detected object with different directions in each frame of face region of the face recognition model.
The left-side, front-side and right-side face classification models of the present embodiment have been deeply trained on a large number of facial images of the left-side, front-side and right-side faces, and are capable of recognizing facial features of subjects in different orientations.
S23, performing the step S21 and the step S22 on each frame of the facial motion video, and obtaining the facial feature region of each frame of the detected object.
S24, the detected object acquisition module cuts off the face feature area of each frame of the non-detected object to obtain a face video only containing the detected object.
And S3, positioning the eyes and the designated parts of the face video through the eye and designated part positioning module, and cutting to generate an eye movement video and a designated part linkage video. The designated sites include the head, nose, mouth and eyebrows.
The specific method for positioning the head, the nose, the mouth, the eyebrows and the eyes of the face video by the eye and designated part positioning module comprises the following steps:
and S31, obtaining a heat map of joint points of the eyes, the eyebrows, the nose and the mouth of the face video through the CNN structure of the CPM to judge the joint points.
Cpm (volumetric position algorithms), which is an algorithm for applying deep learning to human body posture analysis, is derived from position estimation to learn image features and image-dependent (image-dependent) spatial models to estimate human body posture. Pose animation is a full convolution network, with the input being a human body posture map, and the output being n heat maps (heatmaps) representing the responses of n joint points (body part). The CNN structure of CPM of the present embodiment has been largely trained including depth training of color video images of eyes and eyebrows, nose, and mouth, and input of face video, and a heat map of joint points of eyes and eyebrows, nose, and mouth can be obtained.
And S32, obtaining 2D vector geometry through the CNN structure of the PAF to code the motion directions of the eyebrows, the nose and the mouth of the face video, and positioning the eyebrows, the nose and the mouth.
The paf (part Affinity fields) is an algorithm applied to human body posture estimation, and can encode the positions and the directions of joint points from the joint points of a heat map through a part contact strategy, so that 2D vector geometry encoding and limb quick matching are realized, and the method is widely applied to real-time multi-person 2D posture estimation. The CNN structure of CPM of this embodiment has been trained on a large number of videos and heat maps, from which heat maps and joint points available for the CNN structure of CPM, the eyebrows, nose and mouth of facial videos are encoded by 2D vector geometry.
S33, learning a video containing eyes through an image processing module, and positioning the eyes; and intercepting an image of the face video at the moment of eyelid closure, analyzing the degree of eyelid closure in the image, and processing the eyelid part and the pupil part of the image through an image gray processing technology.
The image gray processing technology can adopt any one of a floating point algorithm, an integer method, a shift method and an average value method, after the gray value is obtained, R, G, B values in the original RGB format image are uniformly replaced by the gray value, and the gray image can be obtained.
And S4, processing the eye movement video and the specified part linkage video through the TSN model and outputting action signals of the eyes and the specified part.
The method for processing the eye movement video and the designated part linkage video and outputting the action signal by the TSN model is realized by adopting a basic framework of a BN-initiation v3 convolutional neural network, and the specific method is as follows:
s41, sparsely sampling from the input eye movement video and the designated part linkage video to generate a series of short segments as input;
and S42, performing convolution operation on the RGB format image and the optical flow format image with proper sizes in the short segment through a time flow convolution neural network and a space flow convolution neural network respectively to obtain action signals of the eyes and the designated part, and outputting the action signals.
The TSN (Temporal Segment Network) model is used for action recognition in videos, modeling is based on a long-range Temporal structure, and learning is guaranteed to be effective and efficient when the whole video is used by combining a sparse Temporal sampling strategy (sparse Temporal sampling strategy) and video-level supervision (video-level supervision). In this embodiment, the TSN model is used to perform motion recognition on the eye movement video and the designated portion linkage video by using a sparse time sampling strategy, and a time flow convolution neural network and a space flow convolution neural network are used to perform convolution operation to obtain a motion signal.
And S5, inputting the action signal output by the TSN into a probability output module, and outputting a probability signal for judging the occurrence of eyelid movement abnormality by a computer of each frame of picture.
The eyelid movement disorder includes frequent blinking more than 15 times/min, or incomplete closure of upper and lower eyelids during forced blinking or blinking, and linkage actions accompanied with several key parts of eyebrow, forehead, nose wing and mouth, such as mouth enlargement and eyebrow wrinkle.
Therefore, the probability signal of the occurrence of the eyelid movement abnormality includes a probability signal of the occurrence of an abnormal movement of the eye when the eyelid movement abnormality occurs and a probability signal of the occurrence of an abnormal interlocking movement for each designated portion.
The specific method for outputting the probability signal of the occurrence of the eyelid movement abnormality by the probability output module is as follows:
s5.1, taking the action signal output by the TSN model as input, training a machine learning classifier, combining the category scores of a plurality of short segments to output so as to obtain consensus on category hypothesis among the short segments, obtaining a segment consensus function G (the segment consensus function), and predicting a probability signal of the whole segment of video belonging to each behavior category by using a prediction function H based on the consensus; the behavior types comprise the behavior type of abnormal movement of eyes when the eyelid movement abnormality occurs in the detected object and the behavior type of abnormal linkage movement of each appointed part.
And S5.2, outputting a probability signal of abnormal actions of eyes and a probability signal of abnormal linkage actions of each designated part.
And S6, obtaining an abnormal grade through an evaluation module according to the probability signal of the occurrence of the eyelid movement abnormality and an eyelid movement abnormality probability judgment mechanism, and judging the severity of the eyelid movement abnormality of the detected object.
The eyelid movement abnormity grade judging mechanism comprises an eye movement judging mechanism and a linkage action judging mechanism;
the eye movement judging mechanism is that a threshold value P signal is set to divide the eye movement into two grades of abnormal movement and abnormal movement, and the two grades are compared with a probability signal of abnormal movement of the eyes; if the probability signal of the abnormal action of the eyes is higher than the threshold P signal, the eyes are judged to have abnormal grade; if the probability signal of the abnormal action of the eyes is lower than the threshold P signal, the eyes are in a normal grade;
when the grade of the judging mechanism for judging the abnormal eye movement is an abnormal grade, executing a linkage action judging mechanism;
the linkage action judging mechanism is used for setting a threshold signal of abnormal motion of each designated part and is divided into a light level and a heavy level; if the probability signal of abnormal linkage action of the movement of one designated part is higher than the threshold signal of abnormal movement of the designated part, the abnormal grade is a light grade, otherwise, the abnormal grade is a heavy grade.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, various changes or modifications may be made by the patentees within the scope of the appended claims, and within the scope of the invention, as long as they do not exceed the scope of the invention described in the claims.
Claims (7)
1. An eyelid movement function assessment system based on artificial intelligence, comprising:
the detected object acquisition module: the system comprises a video acquisition unit, a video processing unit and a video processing unit, wherein the video acquisition unit is used for acquiring a face feature area of each frame of a detected object from an input face action video of the detected object, cutting off the face feature area of each frame of the detected object, and acquiring a face video only containing the detected object;
eye and designated part positioning module: the eye and the designated part are used for positioning and image processing the facial video, and the designated part has one or more key parts corresponding to the detected object; cutting to generate an eye movement video and a designated part linkage video of the detected object; the designated parts and the key parts comprise the head, the nose, the mouth and the eyebrows;
the TSN model based on the convolutional neural network: the eye movement video and the specified part linkage video are processed, and action signals of the eyes and the specified part are output;
a probability output module: the action signal output by the TSN is used as input for outputting a probability signal for judging the occurrence of eyelid movement abnormality by a computer of each frame of picture; the probability signal of the abnormal eyelid movement comprises a probability signal of abnormal eye movement when the eyelid movement is abnormal and a probability signal of abnormal linkage movement of each designated part; the specific method for outputting the probability signal of the occurrence of the eyelid movement abnormality by the probability output module is as follows: (1) taking action signals output by a TSN model as input, training a machine learning classifier, combining the category score output of a plurality of short segments to obtain consensus on category hypothesis among the short segments, obtaining a segment consensus function G, and predicting probability signals of each behavior category of the whole segment of video belonging to eyes and a specified part by a prediction function H based on the consensus; the behavior types comprise a behavior type of abnormal movement of eyes when eyelid movement abnormality occurs in the detected object and a behavior type of abnormal linkage movement of each appointed part; (2) outputting a probability signal of abnormal actions of eyes and a probability signal of abnormal linkage actions of each designated part;
an evaluation module: the eyelid movement abnormality judging mechanism is used for judging the severity of the eyelid movement abnormality of the detected object according to the abnormality probability signal of the eye movement, the abnormality linkage action probability signal of the movement of each appointed part and the eyelid movement abnormality probability judging mechanism to obtain the abnormality grade;
the examined object acquisition module comprises:
the fast-RCNN neural network model is used for positioning the face area of each frame of the detected object in the face action video and inputting the face area into the face recognition model frame by frame;
the face recognition model comprises a left face, a front face and a right face classification model and is used for recognizing the facial features of the detected object in different directions in each face region;
the eye and designated part positioning module comprises a CNN structure of CPM, a CNN structure of PAF and an image processing module.
2. The system according to claim 1, wherein said system comprises: the duration of the facial motion video is more than or equal to 5 min.
3. The system according to claim 1, wherein said system comprises: the face motion video is a video of the face of the detected object in the center of the lens.
4. The system according to claim 1, wherein said system comprises: the specific method for obtaining the facial feature region of each frame of the detected object from the input facial motion video of the detected object comprises the following steps:
(1) inputting the facial action video into a fast-RCNN neural network model, positioning the face region of each frame of the detected object in the facial action video, and inputting the face region of each frame into a face recognition model;
(2) the face recognition model recognizes the facial features of the detected object in different directions in each frame of face region;
(3) and (3) performing the step (1) and the step (2) on each frame of the facial motion video to obtain a facial feature region of the detected object of each frame.
5. The system according to claim 1, wherein said system comprises: the specific method for positioning the designated part of the face video comprises the following steps:
(1) obtaining a heat map of the eyes and joint points of a designated part of the face video through a CNN structure of CPM to judge the joint points;
(2) obtaining 2D vector geometry through a CNN structure of the PAF to encode the motion direction of the appointed part of the face video and locate the appointed part;
(3) learning a video containing eyes through an image processing module, and positioning the eyes; and intercepting an image of the face video at the moment of eyelid closure, analyzing the degree of eyelid closure in the image, and processing the eyelid part and the pupil part of the image through an image gray processing technology.
6. The system according to claim 1, wherein said system comprises: the method for processing the eye movement video and the designated part linkage video and outputting the action signal by the TSN model is realized by adopting a basic framework of a BN-initiation v3 convolutional neural network, and the specific method is as follows:
(1) sparse sampling is carried out on the input eye movement video and the designated part linkage video to generate a series of short segments as input;
(2) and carrying out convolution operation on the RGB format image and the optical flow format image with proper sizes in the short segment through a time flow convolution neural network and a space flow convolution neural network respectively to obtain action signals of the eyes and the designated part, and outputting the action signals.
7. The method for evaluating an eyelid movement function evaluating system according to any one of claims 1-6, comprising the steps of:
s1, obtaining a face video only containing the detected object from the input face action video of the detected object through a detected object obtaining module;
s2, positioning the eyes and the designated parts of the face video through an eye and designated part positioning module, and cutting to generate an eye movement video and a designated part linkage video of the detected object;
s3, processing the eye movement video and the specified part linkage video through the TSN model and outputting action signals of the eyes and the specified part;
s4, inputting the action signal output by the TSN into a probability output module, and outputting a probability signal for judging the occurrence of eyelid movement abnormality by the computer of each frame of picture;
and S5, obtaining an abnormal grade according to the probability signal of the occurrence of the eyelid movement abnormality and the eyelid movement abnormality probability judgment mechanism through the evaluation module, and judging the severity of the eyelid movement abnormality of the detected object.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910704037.5A CN110428908B (en) | 2019-07-31 | 2019-07-31 | Eyelid motion function evaluation system based on artificial intelligence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910704037.5A CN110428908B (en) | 2019-07-31 | 2019-07-31 | Eyelid motion function evaluation system based on artificial intelligence |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110428908A CN110428908A (en) | 2019-11-08 |
| CN110428908B true CN110428908B (en) | 2021-12-17 |
Family
ID=68413529
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910704037.5A Active CN110428908B (en) | 2019-07-31 | 2019-07-31 | Eyelid motion function evaluation system based on artificial intelligence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110428908B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111128369A (en) * | 2019-11-18 | 2020-05-08 | 创新工场(北京)企业管理股份有限公司 | Method and device for evaluating Parkinson's disease condition of patient |
| CN111243236A (en) * | 2020-01-17 | 2020-06-05 | 南京邮电大学 | Fatigue driving early warning method and system based on deep learning |
| CN112133391B (en) * | 2020-09-17 | 2024-01-26 | 吾征智能技术(北京)有限公司 | Disease cognitive system based on abnormal eyebrows of human body |
| CN112163556A (en) * | 2020-10-16 | 2021-01-01 | 苏州体素信息科技有限公司 | Child vision disorder screening system and method based on deep learning algorithm |
| CN113569655A (en) * | 2021-07-02 | 2021-10-29 | 广州大学 | Red eye patient identification system based on eye color monitoring |
| CN113674507A (en) * | 2021-08-18 | 2021-11-19 | 建信金融科技有限责任公司 | Infant monitoring method and device, computer equipment and computer-readable storage medium |
| CN113706564B (en) * | 2021-09-23 | 2023-07-18 | 苏州大学 | Meibomian gland segmentation network training method and device based on multiple supervision modes |
| CN116386106A (en) * | 2023-03-16 | 2023-07-04 | 宁波星巡智能科技有限公司 | Intelligent infant head recognition method, device and equipment during sleep-accompanying infant |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8952819B2 (en) * | 2013-01-31 | 2015-02-10 | Lytx, Inc. | Direct observation event triggering of drowsiness |
| CN109344808A (en) * | 2018-07-24 | 2019-02-15 | 中山大学中山眼科中心 | A kind of eyes image processing system based on deep learning |
| CN109640785A (en) * | 2016-04-08 | 2019-04-16 | 维扎瑞尔股份公司 | For obtaining, assembling and analyzing vision data with the method and system of the eyesight performance of evaluator |
-
2019
- 2019-07-31 CN CN201910704037.5A patent/CN110428908B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8952819B2 (en) * | 2013-01-31 | 2015-02-10 | Lytx, Inc. | Direct observation event triggering of drowsiness |
| CN109640785A (en) * | 2016-04-08 | 2019-04-16 | 维扎瑞尔股份公司 | For obtaining, assembling and analyzing vision data with the method and system of the eyesight performance of evaluator |
| CN109344808A (en) * | 2018-07-24 | 2019-02-15 | 中山大学中山眼科中心 | A kind of eyes image processing system based on deep learning |
Non-Patent Citations (2)
| Title |
|---|
| "基于卷积神经网络的人眼状态检测";黄洁媛等;《扬州大学学报（自然科学版）》;20180831;第21卷(第3期);全文 * |
| "基于眼动序列分析的眨眼检测";高宁等;《计算机工程与应用》;20190307;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110428908A (en) | 2019-11-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110428908B (en) | Eyelid motion function evaluation system based on artificial intelligence | |
| CN113420624B (en) | Non-contact fatigue detection method and system | |
| CN111326253A (en) | Method for evaluating multi-modal emotional cognitive ability of patients with autism spectrum disorder | |
| CN110335266B (en) | Intelligent traditional Chinese medicine visual inspection image processing method and device | |
| CN109712710B (en) | Intelligent infant development disorder assessment method based on three-dimensional eye movement characteristics | |
| CN110269587B (en) | Infant motion analysis system and infant vision analysis system based on motion | |
| Wang et al. | Screening early children with autism spectrum disorder via response-to-name protocol | |
| CN112472089A (en) | System and method for judging reliability of psychological test based on eye movement technology | |
| CN109805944B (en) | Children's ability analytic system that shares feelings | |
| CN108305680B (en) | Intelligent Parkinson's disease auxiliary diagnosis method and device based on multivariate biological characteristics | |
| JP2023508339A (en) | Ocular system for deception detection | |
| CN111887867A (en) | Method and system for analyzing character formation based on expression recognition and psychological test | |
| Qin et al. | Improving reliability and accuracy of vibration parameters of vocal folds based on high-speed video and electroglottography | |
| CN115299947A (en) | Psychological scale confidence evaluation method and system based on multi-modal physiological data | |
| CN109938727A (en) | Non-human primate 3D vision stimulation test system and method | |
| CN114565957A (en) | Consciousness assessment method and system based on micro expression recognition | |
| CN110403631A (en) | A kind of Noninvasive intracranial pressure measurement method based on continuous ultrasound image | |
| CN115813343A (en) | Child behavior abnormity evaluation method and system | |
| CN115444422A (en) | Eye movement data-based real environment psychological load assessment method and system | |
| Fedullo et al. | A machine learning approach for a vision-based van-herick measurement system | |
| CN115661101A (en) | Premature infant retinopathy detection system based on random sampling and deep learning | |
| CN115497621A (en) | Old person cognitive status evaluation system | |
| CN111259743B (en) | Training method and system for myopia image deep learning recognition model | |
| CN115223232A (en) | Eye health comprehensive management system | |
| CN114240934A (en) | Image data analysis method and system based on acromegaly |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |