CN112256132A - Man-machine interaction system for gradually-frozen person design - Google Patents
Man-machine interaction system for gradually-frozen person design Download PDFInfo
- Publication number
- CN112256132A CN112256132A CN202011172716.1A CN202011172716A CN112256132A CN 112256132 A CN112256132 A CN 112256132A CN 202011172716 A CN202011172716 A CN 202011172716A CN 112256132 A CN112256132 A CN 112256132A
- Authority
- CN
- China
- Prior art keywords
- blink
- human
- picture
- module
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000003993 interaction Effects 0.000 title claims abstract description 48
- 210000001508 eye Anatomy 0.000 claims abstract description 75
- 230000004397 blinking Effects 0.000 claims abstract description 16
- 210000000744 eyelid Anatomy 0.000 claims description 27
- 238000010586 diagram Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 6
- 238000000034 method Methods 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 3
- 238000003062 neural network model Methods 0.000 claims description 2
- 210000005252 bulbus oculi Anatomy 0.000 abstract description 7
- 230000008014 freezing Effects 0.000 abstract description 3
- 238000007710 freezing Methods 0.000 abstract description 3
- 230000002452 interceptive effect Effects 0.000 abstract description 2
- 208000003443 Unconsciousness Diseases 0.000 description 2
- 230000000474 nursing effect Effects 0.000 description 2
- 208000024891 symptom Diseases 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention discloses a human-computer interaction system designed for a gradually frozen person, which comprises a human face acquisition module, a human eye positioning module, a blink recognition module and an interaction module; the face acquisition module acquires a face picture in real time; the human eye positioning module analyzes the human face picture and calculates to obtain the coordinate position of the human eye key point in the picture; the blink identification module judges whether to blink or not based on the coordinate position of the key point of the human eye in the picture; the interaction module is used for completing human-computer interaction based on blinking actions. This interactive system for freezing people designs is different from in the past and interacts through tracking eyeball, but through blink and computer interaction, has reduced the interaction degree of difficulty, has promoted the stability of system.
Description
Technical Field
The invention belongs to the technical field of human-computer interaction, and particularly relates to a human-computer interaction system designed for a frozen person.
Background
In the prior art, a nursing system for a patient suffering from 'a gradually frozen disease' mainly realizes interaction with a computer by tracking the eyeball position of the patient. However, the eyeball tracking robustness and real-time performance are poor due to the fact that the area and the motion amplitude of the eyeball are small; in addition, the eyeball tracking cannot judge whether the patient moves the eyeball consciously, which is easy to cause misjudgment.
Meanwhile, the method requires that the sight line of the patient is kept at the same position of the screen for a long time, and the operation is fatigue.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a human-computer interaction system designed for the gradually frozen person, so that the patient can realize interaction with a computer by blinking and can type and communicate with the outside only by facing a camera without wearing any equipment. The stability is higher, is difficult for the misjudgment to can distinguish whether the patient is for knowingly blinking, it is comparatively convenient comfortable to use.
In order to achieve the technical purpose, the technical scheme adopted by the invention is as follows:
a man-machine interaction system designed for a person suffering from gradually freezing comprises a face acquisition module, a human eye positioning module, a blink recognition module and an interaction module;
the face acquisition module acquires a face picture in real time;
the human eye positioning module analyzes the human face picture and calculates to obtain the coordinate position of the human eye key point in the picture;
the blink identification module judges whether to blink or not based on the coordinate position of the key point of the human eye in the picture;
the interaction module is used for completing human-computer interaction based on blinking actions.
In order to optimize the technical scheme, the specific measures adopted further comprise:
the human eye positioning module analyzes the human face picture, and calculates to obtain the coordinate positions of the following six key points of human eyes in the picture:
a left canthus keypoint, a right canthus keypoint, a left upper eyelid keypoint, a right upper eyelid keypoint, a left lower eyelid keypoint, and a right lower eyelid keypoint.
The human eye positioning module analyzes the human face picture based on the cascade deep neural network model;
the input of each order of the cascade deep neural network is a whole picture, and the whole picture is firstly input through a similarity change matrix TtCorrecting the picture and inputting the picture St-1Variation to give StThe variation formula is as follows:
and then, calculating the changed picture by using an H (X, Y) function to obtain a key point heat map H (i), wherein the calculation formula is as follows:
meanwhile, calculating a characteristic diagram of an input picture, inputting the picture into a full-connection layer, using a linear rectification function as an activation function, and finally performing up-sampling to obtain a characteristic diagram fc;
the formula for the ReLU function is as follows:
f(x)＝max(0,x)
wherein S ist-1For inputting pictures, TtIs a similarity variation matrix, StFor corrected pictures,. DELTA.StIs StAnd St-1Offset between, SiThe coordinate of the key point is adopted, and X is the output value of the full connecting layer;
the feature map fc, the key point heat maps H (i) and the transformed pictures are input into the next layer of network, and a human eye key point detection model is finally obtained through multi-layer iteration, so that human eye key points can be marked.
The blink identification module calculates a ratio DR of a distance between upper and lower eyelids of human eyes and a distance between left and right canthus based on the coordinate position of the key point of the human eye in the picture, and is used for judging whether to blink:
for left corner key S1, right corner key S2, left upper eyelid key S3, right upper eyelid key S4, left lower eyelid key S5, right lower eyelid key S6, the DR calculation is as follows:
x and y respectively refer to the abscissa and ordinate values of the key point.
In the blink identification module, a group of data of DR when eyes are opened and a group of data of DR when eyes are closed are measured, Kalman filtering is carried out, the stable DR value interval when eyes are closed is obtained, the stable DR value interval when eyes are opened, and a blink DR threshold value is set according to the stable DR value interval when eyes are closed and the stable DR value interval when eyes are opened;
when the DR is changed from being higher than the blink DR threshold value to being lower than the blink DR threshold value, the eyes are detected to be closed, and then the eyes are judged to be open when the DR is changed from being lower than the blink DR threshold value to being higher than the blink DR threshold value, and the whole process is that the eyes are blinked once.
The filter formula for performing kalman filtering is as follows:
wherein the content of the first and second substances,
is an estimated value of current DR, alpha is Kalman gain, XkIs a measure of the current DR that,
is an estimate of the last DR.
In the blink identification module, an effective blink judgment time threshold is further set, in order to distinguish whether the user is consciously blinking, each time the user detects that the user is open and closed, the continuous time duration of which the DR is lower than the blink DR threshold is recorded, if the continuous time of the closed eye is lower than the effective blink judgment time threshold, the user is judged to be unconscious blinking, and if the continuous time of the closed eye is higher than the effective blink judgment time threshold, the user is judged to be effective interaction.
26 letters and communication phrases are uniformly arranged on the interaction module;
when the method is used, each line is selected in a circulating mode on the interface of the interaction module, each character key of the line is selected in a circulating mode after blinking is detected, the selection is determined when blinking is detected again, the letters appear in the output box, one-time input is completed, circulation is continued, and the next interaction is waited.
The invention has the following beneficial effects:
this interactive system for freezing people designs is different from in the past and interacts through tracking eyeball, but through blink and computer interaction, has reduced the interaction degree of difficulty, has promoted the stability of system.
And the input mode of the graphical human-computer interaction interface improves the blink input efficiency, reduces the blink interaction times and brings better experience for patients.
The system can help the patients with the gradually frozen symptoms to communicate easily, express the mind of the heart in time and keep the open heart state. And the patient can express the demand subjectively, lets medical personnel can treat the nursing patient more comprehensively, improves patient's recuperation environment.
Drawings
FIG. 1 is a schematic structural view of the present invention;
FIG. 2 is a schematic diagram of a cascaded deep neural network structure in an embodiment;
FIG. 3 is a diagram illustrating key point detection in a human eye according to an embodiment;
FIG. 4 is an interaction module graphical user interface in an embodiment.
Detailed Description
Embodiments of the present invention are described in further detail below with reference to the accompanying drawings.
Referring to fig. 1, the human-computer interaction system designed for the gradually frozen person comprises a face acquisition module, a human eye positioning module, a blink recognition module and an interaction module;
the face acquisition module acquires a face picture in real time;
the human eye positioning module analyzes the human face picture and calculates to obtain the coordinate position of the human eye key point in the picture;
the blink identification module judges whether to blink or not based on the coordinate position of the key point of the human eye in the picture;
the interaction module is used for completing human-computer interaction based on blinking actions.
In an embodiment, the human eye positioning module analyzes a human face picture, and calculates to obtain the following coordinate positions of six key points of human eyes in the picture:
a left canthus keypoint, a right canthus keypoint, a left upper eyelid keypoint, a right upper eyelid keypoint, a left lower eyelid keypoint, and a right lower eyelid keypoint.
In the embodiment, the human eye positioning module analyzes a human face picture based on a cascade Deep neural Network (Deep Alignment Network) model, and the Network structure is shown in fig. 2;
the input of each order of the cascade deep neural network is a whole picture, and the whole picture is firstly input through a similarity change matrix TtCorrecting the picture and inputting the picture St-1Variation to give StThe variation formula is as follows:
and then, calculating the changed picture by using an H (X, Y) function to obtain a key point heat map H (i), wherein the calculation formula is as follows:
meanwhile, calculating a characteristic diagram of an input picture, namely inputting the picture into a full connection layer (dense layer), using a linear rectification function (ReLU) as an activation function, and finally performing up-sampling to obtain a characteristic diagram fc;
the formula for the ReLU function is as follows:
f(x)＝max(0,x)
the feature map fc, the key point heat maps H (i) and the transformed pictures are input into the next layer of network, and a human eye key point detection model is finally obtained through multi-layer iteration, so that 6 human eye key points can be accurately marked, as shown in fig. 3.
In the embodiment, the coordinate positions of six key points of the human eyes in one picture can be obtained by using the trained human eye key point detection model.
For the left corner key point S1, the right corner key point S2, the left upper eyelid key point S3, the right upper eyelid key point S4, the left lower eyelid key point S5, and the right lower eyelid key point S6, the blink detection algorithm first calculates the ratio DR of the distance between the upper and lower eyelids to the distance between the left and right corners of the eye, and the calculation formula is as follows:
x and y respectively refer to the abscissa and ordinate values of the key point.
In an embodiment, in the blink identification module, in order to filter out the influence of illumination and angle on DR, a set of data of DR when eyes are open and a set of data of DR when eyes are closed are measured, and kalman filtering is performed to obtain a stable DR value interval when eyes are closed, where the DR value interval is: [0.22,0.31], the stable DR interval at eye-open is: [0.61,0.69], setting a blink DR threshold value according to a stable DR value interval when eyes are closed and a stable DR value interval when eyes are open;
when the DR is changed from being higher than the blink DR threshold value by 0.5 to being lower than the blink DR threshold value by 0.5, the eyes are detected to be closed, and when the DR is changed from being lower than the blink DR threshold value by 0.5 to being higher than the blink DR threshold value by 0.5, the eyes are judged to be open, and the whole process is that the eyes are blinked once.
In the embodiment, the filter formula when performing kalman filtering is as follows:
wherein the content of the first and second substances,
is an estimated value of current DR, alpha is Kalman gain, XkIs a measure of the current DR that,
is an estimate of the last DR.
In an embodiment, the blink identification module further sets a valid blink judgment time threshold, and in order to distinguish whether the user is a conscious blink, each time a change from an open eye to a closed eye is detected, the user starts to record a continuous time duration with a DR lower than the blink DR threshold by 0.5, and if the closed eye continuous time is lower than the valid blink judgment time threshold 600ms, the user is determined to be unconscious blink, and if the closed eye continuous time is higher than the valid blink judgment time threshold 600ms, the user is determined to be a valid interaction.
In an embodiment, the interaction module develops an interaction interface through Qt, as shown in fig. 4, 26 letters and some commonly used communication phrases are uniformly arranged on the interface by using blinking as an interaction mode.
In order to facilitate the use of patients with 'gradually frozen symptoms' and reduce the blinking frequency, when the system is used, the interface of the interaction module circularly selects each row, each character key of the row is circularly selected after blinking is detected, when blinking is detected again, the selection is determined, letters appear in the output box, one input is completed, and circulation is continued to be started to wait for the next interaction.
The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may be made by those skilled in the art without departing from the principle of the invention.
Claims (8)
1. A man-machine interaction system designed for a frozen person is characterized by comprising a face acquisition module, a human eye positioning module, a blink recognition module and an interaction module;
the face acquisition module acquires a face picture in real time;
the human eye positioning module analyzes the human face picture and calculates to obtain the coordinate position of the human eye key point in the picture;
the blink identification module judges whether to blink or not based on the coordinate position of the key point of the human eye in the picture;
the interaction module is used for completing human-computer interaction based on blinking actions.
2. The human-computer interaction system designed for the gradually frozen person according to claim 1, wherein the human eye positioning module analyzes the human face picture, and calculates the coordinate positions of the following six key points of human eyes in the picture:
a left canthus keypoint, a right canthus keypoint, a left upper eyelid keypoint, a right upper eyelid keypoint, a left lower eyelid keypoint, and a right lower eyelid keypoint.
3. The human-computer interaction system designed for the frozen person according to claim 1, wherein the human eye positioning module analyzes the human face picture based on a cascaded deep neural network model;
the input of each order of the cascade deep neural network is a whole picture, and the whole picture is firstly input through a similarity change matrix TtCorrecting the picture and inputting the picture St-1Variation to give StThe variation formula is as follows:
St＝Tt -1(TtSt-1+ΔSt)
and then, calculating the changed picture by using an H (X, Y) function to obtain a key point heat map H (i), wherein the calculation formula is as follows:
meanwhile, calculating a characteristic diagram of an input picture, inputting the picture into a full-connection layer, using a linear rectification function as an activation function, and finally performing up-sampling to obtain a characteristic diagram fc;
the formula for the ReLU function is as follows:
f(x)＝max(0,x)
wherein S ist-1For inputting pictures, TtIs a similarity variation matrix, StFor corrected pictures,. DELTA.StIs StAnd St-1Offset between, SiThe coordinate of the key point is adopted, and X is the output value of the full connecting layer;
the feature map fc, the key point heat maps H (i) and the transformed pictures are input into the next layer of network, and a human eye key point detection model is finally obtained through multi-layer iteration, so that human eye key points can be marked.
4. The human-computer interaction system designed for the gradually frozen human according to any one of claims 1 to 3, wherein the blink identification module calculates a ratio DR of a distance between upper and lower eyelids and a distance between left and right canthus of human eye based on the coordinate positions of the key points of human eye in the picture for determining whether to blink:
for left corner key S1, right corner key S2, left upper eyelid key S3, right upper eyelid key S4, left lower eyelid key S5, right lower eyelid key S6, the DR calculation is as follows:
x and y respectively refer to the abscissa and ordinate values of the key point.
5. The system of claim 1, wherein the blink recognition module measures a set of data of DR when the eyes are open and a set of data of DR when the eyes are closed, performs kalman filtering to obtain a stable interval of DR values when the eyes are closed, and sets a threshold value of blink DR according to the stable interval of DR values when the eyes are closed and the stable interval of DR values when the eyes are open;
when the DR is changed from being higher than the blink DR threshold value to being lower than the blink DR threshold value, the eyes are detected to be closed, and then the eyes are judged to be open when the DR is changed from being lower than the blink DR threshold value to being higher than the blink DR threshold value, and the whole process is that the eyes are blinked once.
6. The human-computer interaction system designed for the gradually frozen person according to claim 5, wherein a filter formula in Kalman filtering is as follows:
wherein the content of the first and second substances,
is an estimated value of current DR, alpha is Kalman gain, XkIs a measure of the current DR that,
is an estimate of the last DR.
7. The system of claim 5 or 6, wherein the blink identification module further sets a valid blink judgment time threshold, and in order to distinguish whether the eye is intentionally blinked, each time a change from open to closed eye is detected, records a duration of time that DR is below the blink DR threshold, and determines that the eye is unintentionally blinked if the duration of time that the eye is closed is below the valid blink judgment time threshold, and determines that the eye is effectively interacted if the duration of time is greater than the valid blink judgment time threshold.
8. The human-computer interaction system designed for the frozens according to claim 1, wherein 26 letters and communication phrases are uniformly arranged on the interaction module;
when the method is used, each line is selected in a circulating mode on the interface of the interaction module, each character key of the line is selected in a circulating mode after blinking is detected, the selection is determined when blinking is detected again, the letters appear in the output box, one-time input is completed, circulation is continued, and the next interaction is waited.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011172716.1A CN112256132A (en) | 2020-10-28 | 2020-10-28 | Man-machine interaction system for gradually-frozen person design |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011172716.1A CN112256132A (en) | 2020-10-28 | 2020-10-28 | Man-machine interaction system for gradually-frozen person design |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112256132A true CN112256132A (en) | 2021-01-22 |
Family
ID=74261655
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011172716.1A Pending CN112256132A (en) | 2020-10-28 | 2020-10-28 | Man-machine interaction system for gradually-frozen person design |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112256132A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114529972A (en) * | 2022-02-22 | 2022-05-24 | 山西医科大学第一医院 | Autonomous call processing method and system for amyotrophic lateral sclerosis patient |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109840565A (en) * | 2019-01-31 | 2019-06-04 | 成都大学 | A kind of blink detection method based on eye contour feature point aspect ratio |
| CN111476196A (en) * | 2020-04-23 | 2020-07-31 | 南京理工大学 | Facial action-based nursing demand identification method for old disabled people |
-
2020
- 2020-10-28 CN CN202011172716.1A patent/CN112256132A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109840565A (en) * | 2019-01-31 | 2019-06-04 | 成都大学 | A kind of blink detection method based on eye contour feature point aspect ratio |
| CN111476196A (en) * | 2020-04-23 | 2020-07-31 | 南京理工大学 | Facial action-based nursing demand identification method for old disabled people |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114529972A (en) * | 2022-02-22 | 2022-05-24 | 山西医科大学第一医院 | Autonomous call processing method and system for amyotrophic lateral sclerosis patient |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210280322A1 (en) | Wearable-based certification of a premises as contagion-safe | |
| CN110458101B (en) | Criminal personnel sign monitoring method and equipment based on combination of video and equipment | |
| CN102542281B (en) | Non-contact biometric feature identification method and system | |
| CN107085715A (en) | A kind of television set intelligently detects the dormant system and method for user | |
| CN109375765B (en) | Eyeball tracking interaction method and device | |
| CN103902958A (en) | Method for face recognition | |
| CN110119672A (en) | A kind of embedded fatigue state detection system and method | |
| TW202038133A (en) | System and method for rapidly locating iris using deep learning | |
| Lopez-Martinez et al. | Physiological and behavioral profiling for nociceptive pain estimation using personalized multitask learning | |
| CN110263691A (en) | Head movement detection method based on android system | |
| Ghani et al. | GazePointer: A real time mouse pointer control implementation based on eye gaze tracking | |
| Wu et al. | Appearance-based gaze block estimation via CNN classification | |
| WO2021203718A1 (en) | Method and system for facial recognition | |
| CN110705454A (en) | Face recognition method with living body detection function | |
| CN104898971B (en) | A kind of mouse pointer control method and system based on Visual Trace Technology | |
| Pandey et al. | Assistance for paralyzed patient using eye motion detection | |
| CN112256132A (en) | Man-machine interaction system for gradually-frozen person design | |
| Zhang et al. | Artificial intelligence in physiological characteristics recognition for internet of things authentication | |
| Tian et al. | A severity score for retinopathy of prematurity | |
| Perez et al. | Real-time iris detection on coronal-axis-rotated faces | |
| Zhang et al. | Biometric authentication via finger photoplethysmogram | |
| Rafik et al. | Application of metaheuristic for optimization of iris Image segmentation by using evaluation Hough Transform and methods Daugman | |
| JP5942138B2 (en) | Psychosomatic recovery device and psychosomatic recovery system using the same | |
| CN108771539A (en) | A kind of detection method and its device of the contactless heart rate based on camera shooting | |
| CN112527103B (en) | Remote control method and device for display equipment, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210122 |
|
| RJ01 | Rejection of invention patent application after publication |