CN114974254A - Child depression emotion recognition method based on multi-mode artificial intelligence technology - Google Patents
Child depression emotion recognition method based on multi-mode artificial intelligence technology Download PDFInfo
- Publication number
- CN114974254A CN114974254A CN202210571017.7A CN202210571017A CN114974254A CN 114974254 A CN114974254 A CN 114974254A CN 202210571017 A CN202210571017 A CN 202210571017A CN 114974254 A CN114974254 A CN 114974254A
- Authority
- CN
- China
- Prior art keywords
- micro
- voice
- emotion recognition
- expression
- text
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 54
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000013473 artificial intelligence Methods 0.000 title claims abstract description 19
- 238000005516 engineering process Methods 0.000 title claims abstract description 19
- 239000013598 vector Substances 0.000 claims abstract description 65
- 238000010195 expression analysis Methods 0.000 claims abstract description 16
- 230000003993 interaction Effects 0.000 claims abstract description 12
- 230000000994 depressogenic effect Effects 0.000 claims description 24
- 230000008451 emotion Effects 0.000 claims description 16
- 238000013528 artificial neural network Methods 0.000 claims description 14
- 238000012549 training Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 11
- 206010012374 Depressed mood Diseases 0.000 claims description 10
- 230000003287 optical effect Effects 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 4
- 238000004422 calculation algorithm Methods 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 claims description 3
- 238000003066 decision tree Methods 0.000 claims description 3
- 210000004709 eyebrow Anatomy 0.000 claims description 3
- 230000007787 long-term memory Effects 0.000 claims description 3
- 238000007637 random forest analysis Methods 0.000 claims description 3
- 230000006403 short-term memory Effects 0.000 claims description 3
- 230000003595 spectral effect Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 7
- 239000000463 material Substances 0.000 abstract description 4
- 230000007547 defect Effects 0.000 abstract description 3
- 230000001815 facial effect Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 2
- 208000020401 Depressive disease Diseases 0.000 description 1
- 208000019022 Mood disease Diseases 0.000 description 1
- 208000013738 Sleep Initiation and Maintenance disease Diseases 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006854 communication Effects 0.000 description 1
- 238000009223 counseling Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 230000004630 mental health Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 201000009240 nasopharyngitis Diseases 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 208000019116 sleep disease Diseases 0.000 description 1
- 208000022925 sleep disturbance Diseases 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/26—Speech to text systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
- G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
- G10L25/63—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for estimating an emotional state
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Acoustics & Sound (AREA)
- Computing Systems (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Psychiatry (AREA)
- Hospice & Palliative Care (AREA)
- Child & Adolescent Psychology (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to emotion recognition, in particular to a child depression emotion recognition method based on a multi-mode artificial intelligence technology, which comprises the steps of acquiring voice data and facial video data of a tested child in a man-machine intelligent voice interaction environment; converting the voice data into text information, extracting the text information, text characteristic vectors and voice characteristic vectors corresponding to the voice data, and respectively obtaining a depression emotion recognition result based on the text characteristic vectors and the voice characteristic vectors; extracting a micro-expression area from the face video data, positioning the micro-expression in the face video data, and obtaining a depression emotion recognition result based on micro-expression analysis according to a positioning result; comprehensively evaluating depression emotion recognition results obtained based on text feature vectors, voice feature vectors and micro-expression analysis; the technical scheme provided by the invention can effectively overcome the defects of large consumption of manpower and material resources, complex identification process and low accuracy in the prior art.
Description
Technical Field
The invention relates to emotion recognition, in particular to a method for recognizing depressed emotions of children based on a multi-mode artificial intelligence technology.
Background
Depression is figuratively referred to as "the heart cold," meaning that depression, like the common cold, is a common mood disorder. At present, the diagnosis of depression mainly comprises two parts of self-cognition identification, diagnosis of hospitals or psychological counseling institutions.
The key of the self-cognition identification of the depression is whether the patient has obvious depressed mood, worry, depression and little feeling, frown the face and long-term sighing; whether the people are lack of interest or not can not feel strenuous for any things, and people feel oppressed feeling in the heart and are not happy; whether the people have poor vigor and fatigue, whether the people do mental labor or physical labor feel fatigue, and the people can not recover even if the people have full rest; whether there is significant sleep disturbance, especially early awakening; typical depressed patients have a depressed mood with a change of light morning and light night, i.e. a severe mood depression in the morning and a decrease in the evening. Meanwhile, self-assessment can be performed by means of self-assessment tables, such as "beck depression questionnaire" and "Zung self-assessment scale", which are commonly used in clinical studies to assess disease severity, such as hamilton depression scale (HAMD) and montgomery depression scale (MADS).
There are various reasons for the low recognition rate of depression, including the "pubic sensation" of the patient himself, the various and complicated symptoms accompanying depression, the diagnostic ability of the doctor, etc., and as the social demand for mental health services increases greatly, the problem of lack of medical staff in the psychiatric department will become more prominent. At present, a large amount of manpower and material resources are required to be invested in traditional children depression emotion recognition, the recognition process is complex, and meanwhile, the recognition accuracy is low, so that a children depression emotion recognition method based on a multi-mode artificial intelligence technology is required.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects in the prior art, the invention provides a method for identifying the depressed emotion of the child based on a multi-mode artificial intelligence technology, which can effectively overcome the defects of large consumption of manpower and material resources, complex identification process and low accuracy in the prior art.
(II) technical scheme
In order to achieve the purpose, the invention is realized by the following technical scheme:
a method for identifying depressed emotion of a child based on a multi-modal artificial intelligence technology comprises the following steps:
s1, acquiring voice data and face video data of the tested child in a man-machine intelligent voice interaction environment;
s2, converting the voice data into text information, extracting the text information, text feature vectors and voice feature vectors corresponding to the voice data, and respectively obtaining a depression emotion recognition result based on the text feature vectors and the voice feature vectors;
s3, extracting a micro-expression area from the face video data, positioning the micro-expression in the face video data, and obtaining a depression emotion recognition result based on micro-expression analysis according to the positioning result;
and S4, comprehensively evaluating the depression emotion recognition result obtained based on the text feature vector, the voice feature vector and the micro-expression analysis to obtain the final depression emotion recognition result of the tested child.
Preferably, the converting the voice data into text information in S2, and extracting a text feature vector corresponding to the text information includes:
and converting a plurality of tested child answer sentences arranged according to the time sequence into a text embedding model by utilizing a Bert model to obtain a plurality of text characteristic vectors arranged according to the time sequence.
Preferably, the obtaining of the recognition result of the depressed mood based on the text feature vector in S2 includes:
carrying out model training on the long-term and short-term memory neural network LSTM by using a training set to obtain a depressed emotion recognition model related to the text;
and inputting the text feature vector into a depression emotion recognition model related to the text to obtain a depression emotion recognition result based on the text feature vector.
Preferably, the extracting of the voice feature vector corresponding to the voice data in S2 and obtaining the depression emotion recognition result based on the voice feature vector include:
performing feature extraction and feature selection on the voice data, and recombining the selected features to obtain a voice feature vector corresponding to the voice data;
and recognizing the voice feature vectors by using a random forest algorithm to obtain a depression emotion recognition result based on the voice feature vectors.
Preferably, the performing feature extraction and feature selection on the voice data, and recombining the selected features to obtain a voice feature vector corresponding to the voice data includes:
carrying out feature extraction on voice data through frame windowing processing, and carrying out feature selection on the extracted features according to a decision tree;
discretizing the time domain features, recombining the discretized time domain features in a common occurrence mode, and counting frequency of feature occurrence in a frame of voice data to generate a corresponding voice feature vector.
Preferably, after the voice data is subjected to feature extraction through frame division and windowing processing, time domain features and frequency domain features are obtained, wherein the time domain features comprise short-time energy, energy entropy and zero crossing rate, and the frequency domain features comprise spectral entropy and fundamental frequency.
Preferably, the extracting the micro-expression regions in the face video data and locating the micro-expressions in the face video data in S3 includes:
carrying out face recognition on the face video data by using a face recognition model, and dividing a face region to obtain a micro-expression region;
performing model training on the double-flow neural network by using a training set to obtain a micro-expression positioning model;
and extracting optical flow information from the micro expression area, inputting the optical flow information and the micro expression area into a micro expression positioning model together, and positioning a starting frame, an intermediate frame and an ending frame of the micro expression.
Preferably, the dividing the face region to obtain the micro-expression region includes:
the face region is divided into eyebrows, eyes, a nose, a mouth, a chin, a left cheek and a right cheek to obtain seven micro-expression regions.
Preferably, the dual-flow neural network is a space-time cascaded dual-flow neural network, the dual-flow neural network adopts three attention-based CNN + BLSTM models to extract temporal features and spatial features of micro-expression regions and optical flow information, two models extract frame features of a flow, and the other model determines weights of the frame features.
Preferably, the obtaining of the depression emotion recognition result based on the micro expression analysis according to the positioning result in S3 includes:
and inputting the positioned start frame, intermediate frame and end frame of the micro expression into a depression emotion recognition model related to the micro expression, and obtaining a depression emotion recognition result based on micro expression analysis by combining the corresponding human-computer intelligent voice interaction situation.
(III) advantageous effects
Compared with the prior art, the method for identifying the depressed emotion of the child based on the multi-mode artificial intelligence technology provided by the invention has the advantages that in the process that the tested child carries out man-machine intelligent voice interaction on the camera, the depressed emotion of the child is accurately captured through the meaning expressed and the used voice of the tested child in the communication process and the exposed micro-expression, the whole identification process is very simple, and a large amount of manpower and material resources are not required to be invested.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is a schematic diagram of a process for obtaining a text feature vector-based recognition result of a depressed mood in the present invention;
FIG. 3 is a schematic diagram of a process for obtaining a recognition result of a depressed emotion based on a speech feature vector according to the present invention;
fig. 4 is a schematic flow chart of obtaining a depression emotion recognition result based on micro-expression analysis in the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
A method for identifying a depressed emotion of a child based on a multi-modal artificial intelligence technology is disclosed, as shown in figure 1, and firstly, voice data and facial video data of a tested child in a man-machine intelligent voice interaction environment are obtained.
In the technical scheme, the man-machine intelligent voice interaction can be carried out on the man-machine intelligent voice interaction device and the children, question-answering sentences designed by research experts aiming at identifying the depressed emotion of the children are stored in the man-machine intelligent voice interaction device, and the voice data answered by the tested children can be converted into text information.
Secondly, converting the voice data into text information, extracting text feature vectors and voice feature vectors corresponding to the text information and the voice data, and respectively obtaining a depression emotion recognition result based on the text feature vectors and the voice feature vectors.
As shown in fig. 1 and fig. 2, converting voice data into text information, and extracting a text feature vector corresponding to the text information includes:
and converting a plurality of tested child answer sentences arranged according to the time sequence into a text embedding model by utilizing a Bert model to obtain a plurality of text characteristic vectors arranged according to the time sequence.
Obtaining a depression emotion recognition result based on the text feature vector, comprising:
carrying out model training on the long-term and short-term memory neural network LSTM by using a training set to obtain a depressed emotion recognition model related to the text;
and inputting the text feature vector into a depressed emotion recognition model related to the text to obtain a depressed emotion recognition result based on the text feature vector.
As shown in fig. 1 and 3, extracting a voice feature vector corresponding to the voice data, and obtaining a depression emotion recognition result based on the voice feature vector, includes:
carrying out feature extraction and feature selection on the voice data, and recombining the selected features to obtain a voice feature vector corresponding to the voice data;
and recognizing the voice feature vectors by using a random forest algorithm to obtain a depression emotion recognition result based on the voice feature vectors.
The method comprises the following steps of carrying out feature extraction and feature selection on voice data, and recombining the selected features to obtain a voice feature vector corresponding to the voice data, wherein the method comprises the following steps:
performing feature extraction on voice data through frame windowing, and performing feature selection on extracted features according to a decision tree;
discretizing the time domain features, recombining the discretized time domain features in a co-occurrence mode, and counting the frequency of the feature occurrences in a frame of voice data to generate a corresponding voice feature vector (each feature value in the voice feature vector represents the frequency of the co-occurrence of the specific discrete value of each time domain feature in a frame of voice data).
After the voice data is subjected to feature extraction through frame division and windowing processing, time domain features and frequency domain features are obtained, wherein the time domain features comprise short-time energy, energy entropy and zero crossing rate, and the frequency domain features comprise spectral entropy and fundamental frequency.
As shown in fig. 1 and 4, extracting a micro-expression area in the face video data, locating the micro-expressions in the face video data, and obtaining a depression emotion recognition result based on micro-expression analysis according to the locating result.
Extracting micro-expression areas in the face video data and positioning micro-expressions in the face video data, wherein the micro-expressions are extracted from the face video data, and the method comprises the following steps:
carrying out face recognition on the face video data by using a face recognition model, and dividing a face region to obtain a micro expression region;
performing model training on the double-flow neural network by using a training set to obtain a micro-expression positioning model;
and extracting optical flow information from the micro expression area, inputting the optical flow information and the micro expression area into a micro expression positioning model together, and positioning a starting frame, an intermediate frame and an ending frame of the micro expression.
Obtaining a depression emotion recognition result based on micro expression analysis according to the positioning result, wherein the depression emotion recognition result comprises:
and inputting the positioned start frame, intermediate frame and end frame of the micro expression into a depression emotion recognition model related to the micro expression, and obtaining a depression emotion recognition result based on micro expression analysis by combining the corresponding human-computer intelligent voice interaction situation.
Wherein, divide the face region, obtain the micro expression region, include:
the face region is divided into eyebrows, eyes, a nose, a mouth, a chin, a left cheek and a right cheek to obtain seven micro-expression regions.
The double-flow neural network is a space-time cascaded double-flow neural network, three CNN + BLSTM models based on an attention system are adopted by the double-flow neural network to extract time characteristics and space characteristics of micro-expression areas and optical flow information, two models extract frame characteristics of flows, and the other model determines the weight of the frame characteristics.
In the technical scheme of the application, the output result of the depression emotion recognition model related to the micro expression comprises sadness, neutrality and pleasure, and because a depression patient usually hides his sadness emotion, under the condition that the human-computer intelligent voice interaction situation is biased to be negative, compared with an ordinary child, the child with depression emotion is more likely to show neutral or even pleasure micro expression under the negative situation.
And fourthly, comprehensively evaluating the depression emotion recognition result obtained based on the text feature vector, the voice feature vector and the micro-expression analysis to obtain the final depression emotion recognition result of the tested child, as shown in the figure 1.
The comprehensive evaluation is carried out by presetting the weight parameters respectively corresponding to the depressed emotion recognition results obtained based on the text characteristic vectors, the voice characteristic vectors and the micro-expression analysis, so that more accurate depressed emotion recognition results can be obtained, and the depressed emotion of the tested child can be accurately captured.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.
Claims (10)
1. A method for identifying depressed emotion of children based on a multi-modal artificial intelligence technology is characterized by comprising the following steps: the method comprises the following steps:
s1, acquiring voice data and face video data of the tested child in a man-machine intelligent voice interaction environment;
s2, converting the voice data into text information, extracting the text information, text feature vectors and voice feature vectors corresponding to the voice data, and respectively obtaining a depression emotion recognition result based on the text feature vectors and the voice feature vectors;
s3, extracting a micro-expression area from the face video data, positioning the micro-expression in the face video data, and obtaining a depression emotion recognition result based on micro-expression analysis according to the positioning result;
and S4, comprehensively evaluating the depression emotion recognition result obtained based on the text feature vector, the voice feature vector and the micro-expression analysis to obtain the final depression emotion recognition result of the tested child.
2. The method for identifying depressed emotion of child based on multi-modal artificial intelligence technology as recited in claim 1, wherein: in S2, converting the speech data into text information, and extracting text feature vectors corresponding to the text information, including:
and converting a plurality of tested child answer sentences arranged according to the time sequence into a text embedding model by utilizing a Bert model to obtain a plurality of text characteristic vectors arranged according to the time sequence.
3. The method for identifying depressed emotion of child based on multi-modal artificial intelligence technology as recited in claim 2, wherein: obtaining a depression emotion recognition result based on the text feature vector in S2, including:
carrying out model training on the long-term and short-term memory neural network LSTM by using a training set to obtain a depressed emotion recognition model related to the text;
and inputting the text feature vector into a depressed emotion recognition model related to the text to obtain a depressed emotion recognition result based on the text feature vector.
4. The method for identifying depressed emotion of child based on multi-modal artificial intelligence technology as recited in claim 1, wherein: the step S2 of extracting the voice feature vector corresponding to the voice data, and obtaining a depression emotion recognition result based on the voice feature vector includes:
carrying out feature extraction and feature selection on the voice data, and recombining the selected features to obtain a voice feature vector corresponding to the voice data;
and recognizing the voice feature vectors by using a random forest algorithm to obtain a depression emotion recognition result based on the voice feature vectors.
5. The method for identifying depressed mood of children based on multi-modal artificial intelligence technology as recited in claim 4, wherein: the method for extracting and selecting the features of the voice data and recombining the selected features to obtain the voice feature vector corresponding to the voice data comprises the following steps:
carrying out feature extraction on voice data through frame windowing processing, and carrying out feature selection on the extracted features according to a decision tree;
discretizing the time domain features, recombining the discretized time domain features in a common occurrence mode, and counting frequency of feature occurrence in a frame of voice data to generate a corresponding voice feature vector.
6. The method for identifying the depressed mood of the child based on the multi-modal artificial intelligence technology as recited in claim 5, wherein: after the voice data is subjected to feature extraction through frame windowing, time domain features and frequency domain features are obtained, wherein the time domain features comprise short-time energy, energy entropy and zero crossing rate, and the frequency domain features comprise spectral entropy and fundamental frequency.
7. The method for identifying depressed emotion of child based on multi-modal artificial intelligence technology as recited in claim 1, wherein: in S3, extracting micro-expression regions from the face video data, and locating micro-expressions in the face video data include:
carrying out face recognition on the face video data by using a face recognition model, and dividing a face region to obtain a micro-expression region;
performing model training on the double-flow neural network by using a training set to obtain a micro-expression positioning model;
and extracting optical flow information from the micro expression area, inputting the optical flow information and the micro expression area into a micro expression positioning model together, and positioning a starting frame, an intermediate frame and an ending frame of the micro expression.
8. The method for identifying depressed mood of children based on multi-modal artificial intelligence technology as recited in claim 7, wherein: the dividing of the face area to obtain the micro expression area comprises the following steps:
the face region is divided into eyebrows, eyes, a nose, a mouth, a chin, a left cheek and a right cheek to obtain seven micro-expression regions.
9. The method for identifying depressed mood of children based on multi-modal artificial intelligence technology as recited in claim 7, wherein: the double-flow neural network is a space-time cascaded double-flow neural network, three attention-based CNN + BLSTM models are adopted for extracting the time characteristics and the space characteristics of micro-expression areas and optical flow information, two models extract the frame characteristics of flows, and the other model determines the weight of the frame characteristics.
10. The method for identifying depressed mood of children based on multi-modal artificial intelligence technology as recited in claim 7, wherein: and S3, obtaining a depression emotion recognition result based on the micro expression analysis according to the positioning result, wherein the depression emotion recognition result comprises:
and inputting the positioned start frame, intermediate frame and end frame of the micro expression into a depression emotion recognition model related to the micro expression, and obtaining a depression emotion recognition result based on micro expression analysis by combining the corresponding human-computer intelligent voice interaction situation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210571017.7A CN114974254A (en) | 2022-05-24 | 2022-05-24 | Child depression emotion recognition method based on multi-mode artificial intelligence technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210571017.7A CN114974254A (en) | 2022-05-24 | 2022-05-24 | Child depression emotion recognition method based on multi-mode artificial intelligence technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114974254A true CN114974254A (en) | 2022-08-30 |
Family
ID=82955045
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210571017.7A Pending CN114974254A (en) | 2022-05-24 | 2022-05-24 | Child depression emotion recognition method based on multi-mode artificial intelligence technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114974254A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116049743A (en) * | 2022-12-14 | 2023-05-02 | 深圳市仰和技术有限公司 | Cognitive recognition method based on multi-modal data, computer equipment and storage medium |
-
2022
- 2022-05-24 CN CN202210571017.7A patent/CN114974254A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116049743A (en) * | 2022-12-14 | 2023-05-02 | 深圳市仰和技术有限公司 | Cognitive recognition method based on multi-modal data, computer equipment and storage medium |
| CN116049743B (en) * | 2022-12-14 | 2023-10-31 | 深圳市仰和技术有限公司 | Cognitive recognition method based on multi-modal data, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110507335B (en) | Multi-mode information based criminal psychological health state assessment method and system | |
| Chen et al. | A hierarchical bidirectional GRU model with attention for EEG-based emotion classification | |
| CN110556129B (en) | Bimodal emotion recognition model training method and bimodal emotion recognition method | |
| Rodrigues Makiuchi et al. | Multimodal fusion of bert-cnn and gated cnn representations for depression detection | |
| CN111461176B (en) | Multi-mode fusion method, device, medium and equipment based on normalized mutual information | |
| CN112863630A (en) | Personalized accurate medical question-answering system based on data and knowledge | |
| CN108877801A (en) | More wheel dialog semantics based on multi-modal Emotion identification system understand subsystem | |
| CN108899050A (en) | Speech signal analysis subsystem based on multi-modal Emotion identification system | |
| CN108805088A (en) | Physiological signal analyzing subsystem based on multi-modal Emotion identification system | |
| CN108805089A (en) | Based on multi-modal Emotion identification method | |
| CN114898861A (en) | Multi-modal depression detection method and system based on full attention mechanism | |
| CN109998570A (en) | Inmate's psychological condition appraisal procedure, terminal, equipment and system | |
| CN111462841A (en) | Depression intelligent diagnosis device and system based on knowledge graph | |
| CN113035232B (en) | Psychological state prediction system, method and device based on voice recognition | |
| CN111000556A (en) | Emotion recognition method based on deep fuzzy forest | |
| CN115064246A (en) | Depression evaluation system and equipment based on multi-mode information fusion | |
| Das et al. | Interpretable self-supervised facial micro-expression learning to predict cognitive state and neurological disorders | |
| Li et al. | Improvement on speech depression recognition based on deep networks | |
| CN114974254A (en) | Child depression emotion recognition method based on multi-mode artificial intelligence technology | |
| Du et al. | A novel emotion-aware method based on the fusion of textual description of speech, body movements, and facial expressions | |
| An et al. | Mental health detection from speech signal: A convolution neural networks approach | |
| Li et al. | Multimodal emotion recognition and state analysis of classroom video and audio based on deep neural network | |
| Guo et al. | Deep neural networks for depression recognition based on facial expressions caused by stimulus tasks | |
| Jo et al. | Diagnosis of depression based on four-stream model of bi-LSTM and CNN from audio and text information | |
| Liu | Using convolutional neural networks for the assessment research of mental health |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |