CN116665281B - Key emotion extraction method based on doctor-patient interaction - Google Patents
Key emotion extraction method based on doctor-patient interaction Download PDFInfo
- Publication number
- CN116665281B CN116665281B CN202310773657.0A CN202310773657A CN116665281B CN 116665281 B CN116665281 B CN 116665281B CN 202310773657 A CN202310773657 A CN 202310773657A CN 116665281 B CN116665281 B CN 116665281B
- Authority
- CN
- China
- Prior art keywords
- emotion
- patient
- key
- image
- detection model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000008451 emotion Effects 0.000 title claims abstract description 156
- 230000003993 interaction Effects 0.000 title claims abstract description 35
- 238000000605 extraction Methods 0.000 title claims abstract description 22
- 238000001514 detection method Methods 0.000 claims abstract description 52
- 238000000034 method Methods 0.000 claims abstract description 22
- 230000001815 facial effect Effects 0.000 claims abstract description 18
- 230000008909 emotion recognition Effects 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims description 6
- 238000012937 correction Methods 0.000 claims description 3
- 238000007405 data analysis Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 abstract description 9
- 230000006996 mental state Effects 0.000 abstract description 9
- 230000002996 emotional effect Effects 0.000 abstract description 3
- 238000003745 diagnosis Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000005070 sampling Methods 0.000 description 6
- 230000008921 facial expression Effects 0.000 description 5
- 230000003340 mental effect Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000010195 expression analysis Methods 0.000 description 2
- 208000020016 psychiatric disease Diseases 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 206010013954 Dysphoria Diseases 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a key emotion extraction method based on doctor-patient interaction, which comprises the following steps of: acquiring facial image data of a patient to form a first image set; a face detection model is built, and face detection and recognition are carried out on the image frames in the first image set through the face detection model, so that a second image set is formed; constructing an emotion frame detection model, and carrying out emotion recognition on the second image set through the emotion frame detection model to form a third image set; and constructing an emotion key frame detection model, and extracting the emotion key frames of the patient from the third image set through the emotion key frame detection model so as to realize the identification of the real-time emotion of the patient and the extraction of the emotion key frames in the doctor-patient interaction process. The invention solves the technical problems that the traditional mode of acquiring the emotional state of the patient through off-line evaluation by medical staff is not convenient enough and the mental state evaluation of the patient is not accurate enough.
Description
Technical Field
The invention relates to the technical field of intelligent medical services, in particular to a key emotion extraction method based on doctor-patient interaction.
Background
Mental disorders have become one of the focus of social attention in modern medicine. As the population ages and the pace of life increases, various stress and emotion problems become one of the daily challenges facing people. Mental state assessment is an important problem in the medical field and can be used for assisting in diagnosing mental diseases such as depression, dysphoria and the like, and emotion recognition is an important part of mental state assessment of patients.
Currently, researchers explore emotion recognition of patients based on doctor-patient interaction patterns by using various technical means such as facial expression analysis, speech emotion recognition, physiological signal monitoring and the like; for example, by analyzing the facial expressions of a doctor and patient, the emotions of the doctor and patient are classified and identified by computer vision technology; through monitoring the voice and physiological signals of the patient, the emotion characteristics are extracted from the voice and physiological signals by combining a machine learning technology, and the emotion of the patient is further identified and classified. Facial expression emotion recognition is an important branch in the fields of computer vision and artificial intelligence, and aims to analyze facial expressions through computer vision and pattern recognition technologies so as to judge the emotional state of a person. However, in practical applications, the mental state is still evaluated by using an offline doctor-patient interaction mode, and the emotion recognition accuracy through facial expression analysis, voice emotion recognition or physiological signal detection is not high, so that a large amount of data needs to be acquired. Therefore, it is needed to provide a key emotion extraction method based on doctor-patient interaction, which solves the technical problems that the existing medical staff is not convenient enough to acquire the emotion state of the patient through offline evaluation and the mental state evaluation of the patient is not accurate enough.
Disclosure of Invention
The invention mainly aims to provide a key emotion extraction method based on doctor-patient interaction, and aims to solve the technical problems that an existing medical staff is not convenient enough to acquire the emotion state of a patient through offline evaluation and the mental state evaluation of the patient is not accurate enough.
In order to achieve the above purpose, the present invention provides a key emotion extraction method based on doctor-patient interaction, wherein the key emotion extraction method based on doctor-patient interaction includes the following steps:
S1, acquiring facial image data of a patient to form a first image set;
s2, constructing a face detection model, and carrying out face detection and recognition on the image frames in the first image set through the face detection model to form a second image set;
s3, constructing an emotion frame detection model, and carrying out emotion recognition on the second image set through the emotion frame detection model to form a third image set;
S4, constructing an emotion key frame detection model, and extracting the emotion key frames of the patient from the third image set through the emotion key frame detection model so as to realize the identification of the real-time emotion of the patient and the extraction of the emotion key frames in the doctor-patient interaction process.
In one preferred embodiment, the step S1 acquires facial image data of the patient to form a first image set, specifically:
The method comprises the steps of obtaining facial image data of a patient through image acquisition equipment, carrying out data analysis on the facial image data to obtain a plurality of image frames, and carrying out normalization processing on the plurality of image frames to form a first image set.
In one preferred embodiment, the face detection model adopts RETINAFACE network model architecture.
In one preferred embodiment, the face detection model includes a linear rectification function, and the linear rectification function is used for performing pixel correction processing on a plurality of image frames in the first image set.
In one of the preferred schemes, the face detection model comprises a PFLD face key point recognition algorithm.
In one preferred embodiment, the emotion frame detection model includes ResNet a network, and the ResNet network is configured to perform classification and identification of emotion of the patient on the second image set, so as to obtain an emotion prediction result.
In one of the preferred embodiments, the emotion prediction result includes a natural emotion, a happy emotion, a sad emotion, a vital emotion, a fear emotion, a surprise emotion and a frigid emotion.
In one preferred embodiment, the step S4 is performed on the third image set by using the emotion key frame detection model to extract an emotion key frame of the patient, and specifically includes:
storing emotion prediction results and time labels of each image frame to an emotion linked list;
judging whether the emotion prediction results of the current moment and the image frame at the previous moment are consistent, if not, setting the current image frame as an emotion key frame, updating an emotion linked list, and if so, executing the next step;
and counting the frequency value of the emotion prediction result of each image frame in the emotion linked list, if the frequency value of the emotion prediction result of the current image frame in continuous time is positioned in an emotion frequency threshold value interval, setting the current image frame as an emotion key frame, updating the emotion linked list, and otherwise, setting the current image frame as a common frame.
In one preferred embodiment, the threshold region of emotion frequency is (7, 10).
In one preferred embodiment, the step S4 further includes, after completing the recognition of the real-time emotion of the patient and extracting the emotion key frame:
And encrypting the emotion prediction result and the time tag of the patient by adopting an encryption algorithm, and uploading the generated hash value to a blockchain network for storage.
In the technical scheme of the invention, the key emotion extraction method based on doctor-patient interaction comprises the following steps of: acquiring facial image data of a patient to form a first image set; a face detection model is built, and face detection and recognition are carried out on the image frames in the first image set through the face detection model, so that a second image set is formed; constructing an emotion frame detection model, and carrying out emotion recognition on the second image set through the emotion frame detection model to form a third image set; and constructing an emotion key frame detection model, and extracting the emotion key frames of the patient from the third image set through the emotion key frame detection model so as to realize the identification of the real-time emotion of the patient and the extraction of the emotion key frames in the doctor-patient interaction process. The invention solves the technical problems that the traditional mode of acquiring the emotional state of the patient through off-line evaluation by medical staff is not convenient enough and the mental and psychological state evaluation of the patient is not accurate enough.
In the invention, resNet networks are adopted to identify the emotion of the patient, the identified emotion prediction result is used for extracting the emotion key frame, and the mental state of the patient is evaluated based on the emotion key frame and the corresponding time label, so that the accuracy and the instantaneity of the mental state evaluation are improved.
In the invention, after the identification of the real-time emotion and key frame of the patient in the doctor-patient interaction process is completed, the emotion prediction result and time label of the patient are encrypted, the encrypted hash value is uploaded to the blockchain network, and the blockchain network is adopted to perform the uplink operation on the emotion prediction result and time label of the patient, so as to protect the privacy of the patient.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings may be obtained from the structures shown in these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic diagram of a key emotion extraction method based on doctor-patient interaction according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a face detection model according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of emotion frame recognition for a patient according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating emotion key frame recognition according to an embodiment of the present invention;
Fig. 5 is a schematic diagram of a face detection and recognition result according to an embodiment of the present invention;
FIG. 6 is a schematic diagram showing the results of emotion recognition for a patient according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a result of emotion key frame recognition according to an embodiment of the present invention.
The achievement of the object, functional features and advantages of the present invention will be further described with reference to the drawings in connection with the embodiments.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the invention. All other embodiments, based on the embodiments of the invention, which are apparent to those of ordinary skill in the art without inventive faculty, are intended to be within the scope of the invention.
Furthermore, descriptions such as those referred to as "first," "second," and the like, are provided for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implying an order of magnitude of the indicated technical features in the present disclosure. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature.
Moreover, the technical solutions of the embodiments of the present invention may be combined with each other, but it is necessary to be based on the fact that those skilled in the art can implement the embodiments, and when the technical solutions are contradictory or cannot be implemented, it should be considered that the combination of the technical solutions does not exist, and is not within the scope of protection claimed by the present invention.
Referring to fig. 1, according to an aspect of the present invention, the present invention provides a key emotion extraction method based on doctor-patient interaction, wherein the key emotion extraction method based on doctor-patient interaction includes the following steps:
S1, acquiring facial image data of a patient to form a first image set;
s2, constructing a face detection model, and carrying out face detection and recognition on the image frames in the first image set through the face detection model to form a second image set;
s3, constructing an emotion frame detection model, and carrying out emotion recognition on the second image set through the emotion frame detection model to form a third image set;
S4, constructing an emotion key frame detection model, and extracting the emotion key frames of the patient from the third image set through the emotion key frame detection model so as to realize the identification of the real-time emotion of the patient and the extraction of the emotion key frames in the doctor-patient interaction process.
Specifically, in this embodiment, the meta-diagnosis room is a new medical scenario, a virtual doctor-patient interaction diagnosis and treatment environment is generated by using high-tech equipment and information technology means, and medical resources are concentrated in one room to provide a medical scenario of one-stop medical service. The meta-diagnosis room comprises a doctor workstation, a patient bed, diagnosis and treatment equipment, information technology equipment and other functional areas, and a doctor can check medical record data of a patient, monitor physiological information of the patient, perform diagnosis, treatment and other works on the workstation; the patient can receive diagnosis and treatment, monitor physiological information and the like on the bed; the diagnosis and treatment equipment comprises various medical equipment such as electrocardiographs, sphygmomanometers, thermometers and the like; the information technology equipment comprises video monitoring equipment, remote medical equipment, intelligent diagnosis equipment and the like, and can realize functions of remote medical treatment, information sharing and the like between doctors and patients. The meta-office can provide more humanized medical services, so that interaction between a patient and a doctor is tighter, the medical efficiency can be improved, the medical cost is reduced, the medical experience is improved, and a new thought is provided for the development of medical technology. According to the invention, doctor-patient interaction is performed in a meta-diagnosis room scene, and facial image data of a patient is obtained, so that emotion key frames of the patient are extracted in real time, the accuracy and efficiency of mental state assessment are improved, and a better and accurate diagnosis basis is provided for doctors.
Specifically, in this embodiment, before the step S1 of acquiring the facial image data of the patient, the method further includes: the method comprises the steps of guiding a patient to start acquisition equipment, starting an AI digital human doctor, asking the patient according to a mental assessment scale through the AI digital human doctor, performing interaction such as emotion perception, game interaction and the like, recording current image information of the patient, namely facial image data of the patient through the acquisition equipment, uploading the image information to a server in real time, closing the acquisition equipment if the AI digital human doctor asks to end according to the mental assessment scale, ending doctor-patient interaction, enabling the patient to confirm to end, and enabling the acquisition equipment to automatically close if the patient does not confirm, wherein the acquisition equipment does not capture any facial data beyond a time threshold, the time threshold is 10 minutes, and the method is not specifically limited and can be specifically set according to requirements; if not, continuously asking questions according to the psychological assessment scale; in the present invention, the acquisition device is an image capturing device, and the present invention is not particularly limited, and may be specifically set as needed.
Specifically, in this embodiment, the step S1 acquires facial image data of the patient to form a first image set, specifically: the method comprises the steps of obtaining facial image data of a patient through image acquisition equipment, carrying out data analysis on the facial image data to obtain a plurality of image frames, and carrying out normalization processing on the plurality of image frames to form a first image set.
Specifically, in this embodiment, referring to fig. 2 and fig. 5, the face detection model adopts RETINAFACE network model architecture; wherein, the RETINAFACE network model architecture adopts a pyramid structure; the face detection model comprises a linear rectification function, wherein the linear rectification function is used for carrying out pixel correction processing on a plurality of image frames in a first image set, and replacing all negative values in a feature image in the first image set with 0, namely replacing all negative values in the plurality of image frames in the first image set with 0, and forming feature images Conv1_x, conv2_x, conv3_x, conv4_x and Conv5_x through convolution operation from bottom to top; the method comprises the steps of adopting 1X 1 convolution operation with the step length of 1 for a characteristic layer Conv5_x to form a characteristic layer M5, carrying out up-sampling operation on the characteristic layer M5, fusing the characteristic layer with the characteristic layer after adopting the 1X 1 convolution operation with the step length of 1 for the characteristic layer Conv4_x, and carrying out up-sampling operation on the fused characteristic layer once to form a characteristic layer M4; after the up-sampling operation is carried out on the feature layer M4 once, the feature layer M4 is fused with the feature layer after the 1 multiplied by 1 convolution operation with the step length of 1 is adopted on the feature layer Conv3_x, and the feature layer M3 is formed after the up-sampling operation is carried out on the fused feature layer again; after the feature layer M3 is subjected to one-time up-sampling operation, the feature layer M3 is fused with the feature layer subjected to 1 gamma 1 convolution operation with the step length of 1 on the feature layer Conv2_x, and the fused feature layer is subjected to one-time up-sampling operation to form a feature layer M2; performing 3 gamma 3 convolution operation with the step length of 2 on the M2, M3, M4, M5 and Conv5_x characteristic layers to form effective characteristic layers P2, P3, P4, P5 and P6, and finally performing classification, regression and face key point prediction on the effective characteristic layers P2, P3, P4, P5 and P6 respectively, wherein a classification task outputs classification class (cls) of face pixels; the regression task outputs the position (box) of the face in the image, comprising four vertex coordinates; the invention adopts the PFLD face key point recognition algorithm to predict the face key points and output a plurality of values, in the invention, the PFLD face key point recognition algorithm is adopted to predict the face key points (landmark), ten values are output, and the coordinates of eyes, nose tips and mouth angles are respectively positioned.
Specifically, in this embodiment, referring to fig. 3 and fig. 6, the emotion frame detection model includes ResNet networks, and the ResNet networks are configured to perform classification and identification of emotion of the patient on the second image set, so as to obtain an emotion prediction result; the ResNet network carries out iterative training through a data set which completes emotion recognition in advance so as to have the function of emotion classification recognition, the invention is not particularly limited, and the iterative training is carried out by adopting a conventional technical means; and inputting face images in the second image set identified by the face detection model into the ResNet network which is already pre-trained, so as to obtain emotion prediction results of the patient at each moment and form a third image set.
Specifically, in this embodiment, the emotion prediction result includes a natural emotion, a happy emotion, a sad emotion, a vital emotion, a fear emotion, a surprise emotion, a frigid emotion, and the like, and the present invention is not particularly limited, and may be specifically set as needed.
Specifically, in this embodiment, referring to fig. 4 and fig. 7, in step S4, patient emotion key frame extraction is performed on the third image set through the emotion key frame detection model, specifically: storing emotion prediction results and time labels of each image frame to an emotion linked list; judging whether the emotion prediction results of the current moment and the image frame at the previous moment are consistent, if not, setting the current image frame as an emotion key frame, updating an emotion linked list, and if so, executing the next step; and counting the frequency value of the emotion prediction result of each image frame in the emotion linked list, if the frequency value of the emotion prediction result of the current image frame in continuous time is positioned in an emotion frequency threshold value interval, setting the current image frame as an emotion key frame, updating the emotion linked list, and otherwise, setting the current image frame as a common frame.
Specifically, in the present embodiment, the emotion frequency threshold region is (7, 10); the present invention is not particularly limited, and may be specifically set as needed.
Specifically, in this embodiment, after the step S3 completes the identification of the real-time emotion and key frame of the patient in the doctor-patient interaction process, the method further includes: encrypting the emotion prediction result and the time tag of the patient, the identity information of the patient, the diagnosis and treatment ID and the like by adopting an encryption algorithm, and uploading the generated hash value to a blockchain network for storage so as to better protect the privacy of the patient; in the present invention, the encryption algorithm is a national encryption algorithm, and the present invention is not particularly limited, and may be specifically set as required.
The foregoing description of the preferred embodiments of the present invention should not be construed as limiting the scope of the invention, but rather as utilizing equivalent structural changes made in the description of the present invention and the accompanying drawings or directly/indirectly applied to other related technical fields under the inventive concept of the present invention.
Claims (8)
1. The key emotion extraction method based on doctor-patient interaction is characterized by comprising the following steps of:
S1, acquiring facial image data of a patient to form a first image set;
s2, constructing a face detection model, and carrying out face detection and recognition on the image frames in the first image set through the face detection model to form a second image set;
S3, constructing an emotion frame detection model, and carrying out emotion recognition on the second image set through the emotion frame detection model to form a third image set; the emotion frame detection model comprises ResNet networks, and the ResNet networks are used for carrying out emotion classification and identification on the second image set so as to obtain emotion prediction results;
S4, constructing an emotion key frame detection model, and extracting an emotion key frame of the patient from the third image set through the emotion key frame detection model so as to realize the identification of the real-time emotion of the patient and the extraction of the emotion key frame in the doctor-patient interaction process; the method comprises the following steps:
storing emotion prediction results and time labels of each image frame to an emotion linked list;
judging whether the emotion prediction results of the current moment and the image frame at the previous moment are consistent, if not, setting the current image frame as an emotion key frame, updating an emotion linked list, and if so, executing the next step;
and counting the frequency value of the emotion prediction result of each image frame in the emotion linked list, if the frequency value of the emotion prediction result of the current image frame in continuous time is positioned in an emotion frequency threshold value interval, setting the current image frame as an emotion key frame, updating the emotion linked list, and otherwise, setting the current image frame as a common frame.
2. The method for extracting key emotion based on doctor-patient interaction according to claim 1, wherein the step S1 is to acquire facial image data of a patient to form a first image set, specifically:
The method comprises the steps of obtaining facial image data of a patient through image acquisition equipment, carrying out data analysis on the facial image data to obtain a plurality of image frames, and carrying out normalization processing on the plurality of image frames to form a first image set.
3. A method for extracting key emotion based on doctor-patient interaction according to any one of claims 1-2, wherein the face detection model adopts RETINAFACE network model architecture.
4. A method of key emotion extraction based on doctor-patient interaction as claimed in any one of claims 1-2, wherein said face detection model includes a linear rectification function for performing pixel correction processing on a number of image frames in the first image set.
5. A method of key emotion extraction based on doctor-patient interaction as recited in any of claims 1-2, wherein said face detection model includes PFLD face key point recognition algorithm.
6. The method for extracting key emotion based on doctor-patient interaction of claim 1, wherein the emotion prediction result includes natural emotion, happy emotion, sad emotion, vital emotion, fear emotion, surprise emotion and frigid emotion.
7. A key emotion extraction method based on doctor-patient interaction as claimed in claim 1, characterized in that the emotion frequency threshold region is (7, 10).
8. The method for extracting key emotion based on doctor-patient interaction of claim 7, wherein after the step S4 is completed for identifying the real-time emotion of the patient and extracting the emotion key frame, further comprises:
And encrypting the emotion prediction result and the time tag of the patient by adopting an encryption algorithm, and uploading the generated hash value to a blockchain network for storage.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310773657.0A CN116665281B (en) | 2023-06-28 | 2023-06-28 | Key emotion extraction method based on doctor-patient interaction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310773657.0A CN116665281B (en) | 2023-06-28 | 2023-06-28 | Key emotion extraction method based on doctor-patient interaction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116665281A CN116665281A (en) | 2023-08-29 |
| CN116665281B true CN116665281B (en) | 2024-05-10 |
Family
ID=87727956
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310773657.0A Active CN116665281B (en) | 2023-06-28 | 2023-06-28 | Key emotion extraction method based on doctor-patient interaction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116665281B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117671774B (en) * | 2024-01-11 | 2024-04-26 | 好心情健康产业集团有限公司 | Face emotion intelligent recognition analysis equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107169426A (en) * | 2017-04-27 | 2017-09-15 | 广东工业大学 | A kind of detection of crowd's abnormal feeling and localization method based on deep neural network |
| CN109190487A (en) * | 2018-08-07 | 2019-01-11 | 平安科技（深圳）有限公司 | Face Emotion identification method, apparatus, computer equipment and storage medium |
| CN110287895A (en) * | 2019-04-17 | 2019-09-27 | 北京阳光易德科技股份有限公司 | A method of emotional measurement is carried out based on convolutional neural networks |
| CN110781810A (en) * | 2019-10-24 | 2020-02-11 | 合肥盛东信息科技有限公司 | Face emotion recognition method |
| CN110807394A (en) * | 2019-10-23 | 2020-02-18 | 上海能塔智能科技有限公司 | Emotion recognition method, test driving experience evaluation method, device, equipment and medium |
| CN112163459A (en) * | 2020-09-04 | 2021-01-01 | 三峡大学 | Face abnormal emotion recognition method adopting 3D convolution feature fusion network |
| CN114943997A (en) * | 2022-05-18 | 2022-08-26 | 上海大学 | Cerebral apoplexy patient expression classification algorithm and system based on attention and neural network |
| CN116343314A (en) * | 2023-05-30 | 2023-06-27 | 之江实验室 | Expression recognition method and device, storage medium and electronic equipment |
-
2023
- 2023-06-28 CN CN202310773657.0A patent/CN116665281B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107169426A (en) * | 2017-04-27 | 2017-09-15 | 广东工业大学 | A kind of detection of crowd's abnormal feeling and localization method based on deep neural network |
| CN109190487A (en) * | 2018-08-07 | 2019-01-11 | 平安科技（深圳）有限公司 | Face Emotion identification method, apparatus, computer equipment and storage medium |
| WO2020029406A1 (en) * | 2018-08-07 | 2020-02-13 | 平安科技（深圳）有限公司 | Human face emotion identification method and device, computer device and storage medium |
| CN110287895A (en) * | 2019-04-17 | 2019-09-27 | 北京阳光易德科技股份有限公司 | A method of emotional measurement is carried out based on convolutional neural networks |
| CN110807394A (en) * | 2019-10-23 | 2020-02-18 | 上海能塔智能科技有限公司 | Emotion recognition method, test driving experience evaluation method, device, equipment and medium |
| CN110781810A (en) * | 2019-10-24 | 2020-02-11 | 合肥盛东信息科技有限公司 | Face emotion recognition method |
| CN112163459A (en) * | 2020-09-04 | 2021-01-01 | 三峡大学 | Face abnormal emotion recognition method adopting 3D convolution feature fusion network |
| CN114943997A (en) * | 2022-05-18 | 2022-08-26 | 上海大学 | Cerebral apoplexy patient expression classification algorithm and system based on attention and neural network |
| CN116343314A (en) * | 2023-05-30 | 2023-06-27 | 之江实验室 | Expression recognition method and device, storage medium and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| Saman Sarraf.Machine learning applications to recognize autism and Alzheimer's disease.Neurological Disorders and Imaging Physics.2019,第3卷第1-23页. * |
| 基于图像差分的关键帧检测技术;许宏;计算机工程与设计;20100628(第12期);第177-180页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116665281A (en) | 2023-08-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112120716A (en) | Wearable multi-mode emotional state monitoring device | |
| JP4401079B2 (en) | Subject behavior analysis | |
| US20220044821A1 (en) | Systems and methods for diagnosing a stroke condition | |
| CN111326253A (en) | Method for evaluating multi-modal emotional cognitive ability of patients with autism spectrum disorder | |
| CN116665281B (en) | Key emotion extraction method based on doctor-patient interaction | |
| CN109460749A (en) | Patient monitoring method, device, computer equipment and storage medium | |
| Wang et al. | A novel facial thermal feature extraction method for non-contact healthcare system | |
| CN111920420A (en) | Patient behavior multi-modal analysis and prediction system based on statistical learning | |
| Li et al. | An EEG-based multi-modal emotion database with both posed and authentic facial actions for emotion analysis | |
| CN108882853A (en) | Measurement physiological parameter is triggered in time using visual context | |
| CN111222464B (en) | Emotion analysis method and system | |
| CN115299947A (en) | Psychological scale confidence evaluation method and system based on multi-modal physiological data | |
| CN211862821U (en) | Autism auxiliary evaluation system based on deep learning | |
| CN113822164A (en) | Dynamic emotion recognition method and device, computer equipment and storage medium | |
| CN112220455A (en) | Emotion recognition method and device based on video electroencephalogram signals and computer equipment | |
| CN110364260A (en) | Autism earlier evaluations apparatus and system based on indicative language paradigm | |
| CN112741620A (en) | Cervical spondylosis evaluation device based on limb movement | |
| CN113326729B (en) | Multi-mode classroom concentration detection method and device | |
| CN117809354B (en) | Emotion recognition method, medium and device based on head wearable device perception | |
| CN116509419B (en) | Electroencephalogram information processing method and system | |
| CN113887311B (en) | Method, device and storage medium for protecting privacy of ophthalmic patient | |
| CN117224080B (en) | Human body data monitoring method and device for big data | |
| El Morabit | New Artificial Intelligence techniques for Computer vision based medical diagnosis | |
| Kavitha et al. | Implementation of cardiac signal for biometric recognition from facial video | |
| CN117809354A (en) | Emotion recognition method, medium and device based on head wearable device perception |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |