CN113392918A - Depressive disorder related factor identification method based on multi-source information fusion - Google Patents
Depressive disorder related factor identification method based on multi-source information fusion Download PDFInfo
- Publication number
- CN113392918A CN113392918A CN202110701988.4A CN202110701988A CN113392918A CN 113392918 A CN113392918 A CN 113392918A CN 202110701988 A CN202110701988 A CN 202110701988A CN 113392918 A CN113392918 A CN 113392918A
- Authority
- CN
- China
- Prior art keywords
- preprocessing
- time length
- disorder related
- source information
- distance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 29
- 238000000034 method Methods 0.000 title claims abstract description 28
- 208000020401 Depressive disease Diseases 0.000 title claims abstract description 26
- 230000002996 emotional effect Effects 0.000 claims abstract description 22
- 238000007781 pre-processing Methods 0.000 claims abstract description 21
- 238000000605 extraction Methods 0.000 claims abstract description 19
- 230000008451 emotion Effects 0.000 claims abstract description 15
- 230000008909 emotion recognition Effects 0.000 claims abstract description 15
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 claims abstract description 9
- 238000010801 machine learning Methods 0.000 claims abstract description 7
- 230000006854 communication Effects 0.000 claims abstract description 5
- 238000013528 artificial neural network Methods 0.000 claims abstract description 4
- 230000035484 reaction time Effects 0.000 claims abstract description 4
- 238000012360 testing method Methods 0.000 claims description 16
- 230000008921 facial expression Effects 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 9
- 238000003066 decision tree Methods 0.000 claims description 7
- 230000011218 segmentation Effects 0.000 claims description 6
- 230000001965 increasing effect Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 3
- 238000005259 measurement Methods 0.000 claims description 3
- 238000007637 random forest analysis Methods 0.000 claims description 3
- 238000012706 support-vector machine Methods 0.000 claims description 3
- 230000007547 defect Effects 0.000 abstract description 4
- 238000004422 calculation algorithm Methods 0.000 description 10
- 230000014509 gene expression Effects 0.000 description 9
- 238000003745 diagnosis Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 7
- 238000007635 classification algorithm Methods 0.000 description 5
- 230000002085 persistent effect Effects 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000013145 classification model Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- 208000035473 Communicable disease Diseases 0.000 description 1
- 206010012374 Depressed mood Diseases 0.000 description 1
- 206010020710 Hyperphagia Diseases 0.000 description 1
- 206010028980 Neoplasm Diseases 0.000 description 1
- 206010071299 Slow speech Diseases 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 201000011510 cancer Diseases 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 230000000994 depressogenic effect Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 230000004064 dysfunction Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000001939 inductive effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 208000024714 major depressive disease Diseases 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 201000003995 melancholia Diseases 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 230000006996 mental state Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000008450 motivation Effects 0.000 description 1
- 238000011369 optimal treatment Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 235000020830 overeating Nutrition 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000013138 pruning Methods 0.000 description 1
- 208000020016 psychiatric disease Diseases 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 208000019116 sleep disease Diseases 0.000 description 1
- 208000020685 sleep-wake disease Diseases 0.000 description 1
- 230000004037 social stress Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
- G06F18/256—Fusion techniques of classification results, e.g. of results related to same input data of results relating to different input data, e.g. multimodal recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Signal Processing (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Medical Informatics (AREA)
- Image Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A depression disorder related factor identification method based on multi-source information fusion has many defects, such as error identification, slow identification speed and the like. A depressive disorder related factor identification method based on multi-source information fusion. Preprocessing a voice signal and extracting characteristics; video frame preprocessing and feature extraction; preprocessing emotion text features and extracting the features; collecting the reaction time length in the steps, including the time length between questions and answers in the language communication process, the time length of the subject and the other party looking at the eyes and the time length required by combing self opinions; fusing the emotional characteristics of the different modes; classifying the fused emotional features; wherein, the emotion recognition classifier used for classification adopts a machine learning classifier or utilizes an artificial neural network to directly perform classification recognition; outputting the identification result of the relevant factors of the depression. The invention has the advantage of high identification accuracy.
Description
Technical Field
The invention relates to a depressive disorder related factor identification method based on multi-source information fusion.
Background
Emotion recognition has been a research area of great interest. The expression of human emotion is performed simultaneously in various forms, such as intrinsic information that is difficult to observe through pulse, heart rate, and the like, and extrinsic information that is easy to observe naturally, such as voice, semantics, facial expression, and the like. The emotion calculation theory is put forward to endow the computer with the ability of perceiving, understanding and expressing emotion so as to promote the further development of the human-computer interaction technology. The emotion recognition algorithm is used as a key ring in an emotion recognition system and plays a significant role in the whole system.
Depressive disorders, also commonly referred to as melancholia or depression, are characterized clinically by persistent significant sadness as the principal manifestation, as well as by a decline in motivation or pleasurable feedback for participation in daily activities. Persistent depressed mood for a long time can seriously affect the social ability of the patient, such as lack of confidence, active avoidance of the population, even crime, and also cause physical dysfunction of the patient, such as sleep disorder, overeating, and the like. Along with the continuous development of society, the pace of work and life of people is obviously accelerated, the competitive pressure of daily life is increased continuously, the spirit of people is highly stressed, and the incidence of depressive disorder in people is increased.
A professional clinician makes a preliminary diagnosis based on clinically common symptoms, while a clinician or psychiatrist can also diagnose and assess the severity of depression in a patient by means of a standardized questionnaire of some profession. Unlike the detection of physiological diseases such as cancer and infectious diseases, structured questionnaires do not allow accurate quantification of the patient's condition through clear physiological index data or medical images, but rather rely on the professional level of clinicians and psychological consultants with higher observation and questionnaires, requiring relatively long time for diagnosis. In addition, the domestic professional mental medical and health service organization is less than two thousand. The industry is still in a severe supply and demand phase. And due to the difference of depression diagnosis modes, subjective influences brought by clinicians cannot be avoided in the diagnosis process. The difference in performance of the same patient at different times of diagnosis can also cause different diagnosis results for the doctor. In addition, the bias of society against patients with mental diseases leads to the patients hiding the conditions intentionally, which results in the inability of doctors to make timely and accurate assessment of the mental states of patients and thus to miss the optimal treatment period. Therefore, the search for an objective and efficient new method for diagnosing the depressive disorder becomes a hot research point for the early detection of the depressive disorder in the world.
In recent years, many researchers at home and abroad find that the persistent emotional state of the depression patient obviously affects the voice, semantic expression and facial expression of the patient through research, wherein compared with other clinical expressions, the changes in the voice, semantic and expression have obvious characteristics, such as slow speech speed during speaking, frequent appearance of negative words, obvious monotonous and rigid expression and the like. Clinicians often make judgments in conjunction with changes in the patient's voice, language, and expression.
Just because of the complexity and the diversity of manifestations of depressive disorders, considered from either perspective alone, there are limitations, such as the patient deliberately masking his or her true emotional state due to social stress, or deliberately giving some expression, sound or answer different from the true emotion. This makes the models obtained from these modalities naturally deficient, meaning that models constructed from monomodal physiological signals are not reliable enough. With the development of related research, researchers realize that multi-modal physiological signal data can reflect the change of the emotion of a human body from different angles, and a more objective and effective depressive disorder recognition model can be obtained by fusing the multi-modal physiological data.
Although multi-modal emotion recognition can theoretically overcome the defects of single information and one-sided emotion characteristics of single-modal emotion recognition, how to effectively fuse multi-modal information is a core problem of multi-modal emotion recognition. The effect of multi-modal emotion recognition can be better improved only by effectively fusing the information of all the modalities, otherwise, the effect of the multi-modal information cannot be exerted. The traditional multi-modal information fusion mode mainly comprises data layer fusion, feature layer fusion and decision layer fusion, wherein the most used modes are feature layer fusion and decision layer fusion. In the multi-modal emotion recognition task, a feature layer-based fusion mode is a relatively common mode used by people. Because the fusion mode really fuses the information of different modes together, the information of different modes can be mutually influenced and complemented. But the approach based on feature-layer fusion also presents a number of problems. For example, when combining features of different modalities, it is common and most straightforward to concatenate feature vectors of different modalities to form a longer feature vector. Therefore, the direct cascade connection causes the formed characteristic dimension to be very high, and the dimension disaster problem exists.
In the multi-modal emotion recognition task, a mode based on decision layer fusion is generally applied to the first. When merging the classification results of different modalities, a summation method or a voting method is generally adopted. Although this approach has achieved some effect, it does not really fuse the data of different modalities, but simply combines the results of different modalities. Therefore, the fusion mode is greatly influenced by the result of a single modality, and is easy to cause wrong recognition result.
Combining the three fusion modes described above, each of the fusion modes has certain advantages, but also has certain disadvantages. In the actual task, factors in all aspects need to be comprehensively considered, and a proper fusion mode can be selected.
Disclosure of Invention
The invention aims to solve the defects of the existing multi-source information fusion method, such as error identification, low identification speed and the like, and provides a depressive disorder related factor identification method based on multi-source information fusion.
A method for identifying depression disorder related factors based on multi-source information fusion is realized by the following steps:
step one, voice signal preprocessing and feature extraction;
firstly, converting a collected voice signal in an analog signal form into a digital signal;
secondly, preprocessing the voice signal in the form of a digital signal, and then extracting and analyzing the characteristic parameters of the depression disorder related factors in the preprocessed voice signal to obtain voice signal characteristics; wherein, full feature extraction is carried out through open SMILE;
secondly, preprocessing a video frame and extracting characteristics;
firstly, preprocessing a video frame;
secondly, extracting depression disorder related factors in the facial expression characteristics of the preprocessed video frames; extracting facial expression features of the preprocessed video frame by adopting a deep convolutional neural network (VGGNet) to extract image features, and setting a smaller convolution kernel operation and an operation of increasing the network depth;
thirdly, emotion text feature preprocessing and feature extraction;
the emotion text preprocessing operation comprises document segmentation, text word segmentation, stop word removal, word frequency statistics and text vectorization operation;
performing feature extraction of relevant factors of depressive disorder aiming at the preprocessed emotional text, wherein the feature extraction method adopts one of Word frequency-reverse file frequency, Word2Vec or countvectorer;
step four, collecting the reaction time length in the steps, including the time length between questions and answers related in the language communication process, the time length of the subject and the other party looking at the eyes and the time length required by combing self opinions;
step five, the emotional characteristics of the different modes are fused;
step six, classifying the fused emotional characteristics; wherein, the emotion recognition classifier used for classification adopts a machine learning classifier or utilizes an artificial neural network to directly perform classification recognition;
and seventhly, outputting the identification result of the relevant factors of the depression.
Preferably, the machine learning classifier comprises a support vector machine and a random forest.
Preferably, in the step five, the step of classifying the fused emotional features uses a K nearest neighbor classifier based on statistics to perform classification, and the calculating step includes the following steps:
calculating the distance: given a test object, calculating its distance to each object in the training set;
and finding neighbors: defining k objects with the nearest distance as the neighbors of the test objects;
classification is carried out: classifying the test object according to the main category to which the k neighbors belong;
the K-nearest neighbor classifier has three basic elements:
(4) distance measurement:
when p is 2, the Euclidean distance is obtained; when p is 1, manhattan distance;
(5) selection of the value of K:
the smaller the K value is, the more complex the whole model is, and overfitting is easy to occur; the larger the value of K, the simpler the overall model, and the larger the approximation error (misclassification);
(6) determination of test object class:
c) majority voting:
wherein v is a class number, yiIs a nearest neighbor class label, I (.) is an indication function, if its parameter is true, returns 1, otherwise returns 0;
d) distance weighted voting:
wherein d is the distance between the two objects;
preferably, the step of classifying the fused emotional features in the step five includes:
a decision tree classifier based on discrimination is adopted for classification,
assuming the sample set is T, the gini coefficient value for T can be calculated by:
wherein p isjRefers to the probability that the category j appears in the sample set T.
The invention has the beneficial effects that:
the voice semantic expression and the communication duration are respectively used as modal emotion information based on the characteristics that the voice, the semantic expression and the facial expression of a patient are obviously changed by the persistent emotional state shown by a depressed patient. And then, carrying out feature layer fusion processing to obtain an identification method of relevant factors of the depressive disorder, thereby improving the accuracy and the identification rate of the multi-modal emotion identification.
Specifically, the invention starts from the overall framework of the multi-source information joint judgment algorithm in depression diagnosis, and then refines the detail parts of the framework one by one, optimizes the algorithm in multiple aspects, improves the network precision, reduces the network parameters and achieves the effect of accuracy and high speed.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
The first embodiment is as follows:
in this embodiment, as shown in fig. 1, the method for identifying a relevant factor of depressive disorder based on multi-source information fusion is implemented by the following steps:
step one, voice signal preprocessing and feature extraction;
firstly, because the voice signal collected by the equipment is an analog signal, the voice signal collected in the form of the analog signal is converted into a digital signal before feature extraction, so that subsequent processing can be continued;
secondly, acquiring irrelevant noise brought by mixed environment and recording equipment in the acquired voice signal, reducing emotion recognition accuracy, and performing preprocessing operation on the voice signal in a digital signal form before extracting emotional characteristics, and then extracting and analyzing characteristic parameters of depression disorder related factors in the preprocessed voice signal to obtain voice signal characteristics in order to improve emotion recognition accuracy and model generalization capability; wherein, full feature extraction is carried out through open SMILE;
secondly, preprocessing a video frame and extracting characteristics;
because the human face shielding phenomenon exists in a part of images in the video, certain interference can be generated on model training, and secondly, because the rotation amplitude of the part of human face images in the video is large, the human face feature extraction speed is slowed down. Aiming at the problems, before the extraction of emotional features, firstly, preprocessing a video frame;
secondly, extracting depression disorder related factors in the facial expression characteristics of the preprocessed video frames; in the step of extracting the facial expression features of the preprocessed video frame, a deep convolutional neural network VGGNet with good generalization is adopted to extract image features, and a smaller convolution kernel operation and an operation of increasing the network depth are set to improve the characteristics of classification and recognition effects;
thirdly, emotion text feature preprocessing and feature extraction;
the emotion text preprocessing operation comprises document segmentation, text word segmentation, stop word removal (including punctuation, numbers, single words and other nonsense words), word frequency statistics and text vectorization operation;
performing feature extraction of relevant factors of depressive disorder aiming at the preprocessed emotional text, wherein the feature extraction method adopts one of Word frequency-inverse file frequency (TF-IDF), Word2Vec or countvectorer;
step four, collecting the reaction time length in the steps, including the time length between questions and answers related in the language communication process, the time length of the subject and the other party looking at the eyes and the time length required by combing self opinions;
step five, the emotional characteristics of the different modes are fused;
step six, classifying the fused emotional characteristics, and taking the reaction duration as the same time; wherein, the emotion recognition classifier used for classification adopts a machine learning classifier or utilizes an artificial neural network to directly perform classification recognition;
and seventhly, outputting the identification result of the relevant factors of the depression.
The second embodiment is as follows:
different from the specific embodiment, in the method for identifying the relevant factors of the depressive disorder based on the multi-source information fusion, the traditional machine learning classifier comprises a support vector machine, a random forest and the like.
The third concrete implementation mode:
different from the first specific embodiment, in the method for identifying relevant factors of depressive disorder based on multi-source information fusion of the present embodiment, in the fifth step, the step of classifying the fused emotional features is performed by using a K-nearest neighbor classifier based on statistics, where the K-nearest neighbor classifier is designed to find all training examples with relatively close attributes to the test examples. The calculation steps comprise the following steps:
calculating the distance: given a test object, calculating its distance to each object in the training set;
and finding neighbors: defining k objects with the nearest distance as the neighbors of the test objects;
classification is carried out: classifying the test object according to the main category to which the k neighbors belong;
the K-nearest neighbor classifier has three basic elements:
(7) distance measurement:
when p is 2, the Euclidean distance is obtained; when p is 1, manhattan distance;
(8) selection of the value of K:
the smaller the K value is, the more complex the whole model is, and overfitting is easy to occur; the larger the value of K, the simpler the overall model, and the larger the approximation error (misclassification);
(9) determination of test object class:
e) majority voting:
wherein v is a class number, yiIs a nearest neighbor class label, I (.) is an indication function, if its parameter is true, returns 1, otherwise returns 0;
f) distance weighted voting:
wherein d is the distance between the two objects;
the K nearest neighbor algorithm has the advantages that: the method is simple, easy to understand, easy to realize, free of estimating parameters and training (free of spending any time for constructing a model), particularly suitable for the problem of multi-classification, suitable for classifying rare events, high in precision, insensitive to abnormal values and free of data input hypothesis. The disadvantages are: the passive learning method requires a large amount of storage space (the entire data set needs to be saved), is time-consuming (the values of the target sample and each sample in the training set need to be calculated), and has poor interpretability.
The fourth concrete implementation mode:
different from the first specific embodiment, in the method for identifying relevant factors for depressive disorder based on multi-source information fusion of the present embodiment, the step of classifying the fused emotional features in the fifth step specifically includes:
the decision tree classification algorithm is an example-based inductive learning method and can extract a tree type classification model from given unordered training samples. Each non-leaf node in the tree records which feature is used for category determination, and each leaf node represents the last determined category. And forming a classified path rule from the root node to each leaf node. When a new sample is tested, the test is performed on each branch node only from the root node, the branch node recursively enters the subtree along the corresponding branch and is retested until reaching the leaf node, and the category represented by the leaf node is the prediction category of the current test sample.
The CART classification algorithm is a commonly used algorithm in a decision tree classification algorithm, and the basic idea of the CART classification algorithm is as follows: carrying out recursive division on the training sample set to obtain independent variable spaces, sequentially establishing decision tree models, and then carrying out branch pruning by adopting a data verification method to obtain a decision tree classification model meeting requirements. The algorithm can process either discrete data or continuous data. The CART classification algorithm selects test attributes according to a Gini coefficient, and the smaller the value of the Gini coefficient, the better the partitioning effect. Assuming the sample set is T, the gini coefficient value for T can be calculated by:
wherein p isjRefers to the probability that the category j appears in the sample set T.
The CART algorithm has the advantages that: the method has the characteristics of high accuracy, high efficiency, simple mode and the like of the general decision tree. The CART algorithm has the following defects: the CART algorithm is a large-capacity sample set mining algorithm, and is not stable enough when the sample set is small; requiring that the selected attribute only produce two child nodes, the error may increase faster when there are too many categories.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes may be made to the present invention by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (4)
1. A depressive disorder related factor identification method based on multi-source information fusion is characterized in that: the method is realized by the following steps:
step one, voice signal preprocessing and feature extraction;
firstly, converting a collected voice signal in an analog signal form into a digital signal;
secondly, preprocessing the voice signal in the form of a digital signal, and then extracting and analyzing the characteristic parameters of the depression disorder related factors in the preprocessed voice signal to obtain voice signal characteristics; wherein, full feature extraction is carried out through open SMILE;
secondly, preprocessing a video frame and extracting characteristics;
firstly, preprocessing a video frame;
secondly, extracting depression disorder related factors in the facial expression characteristics of the preprocessed video frames; extracting facial expression features of the preprocessed video frame by adopting a deep convolutional neural network (VGGNet) to extract image features, and setting a smaller convolution kernel operation and an operation of increasing the network depth;
thirdly, emotion text feature preprocessing and feature extraction;
the emotion text preprocessing operation comprises document segmentation, text word segmentation, stop word removal, word frequency statistics and text vectorization operation;
performing feature extraction of relevant factors of depressive disorder aiming at the preprocessed emotional text, wherein the feature extraction method adopts one of Word frequency-reverse file frequency, Word2Vec or countvectorer;
step four, collecting the reaction time length in the steps, including the time length between questions and answers related in the language communication process, the time length of the subject and the other party looking at the eyes and the time length required by combing self opinions;
step five, the emotional characteristics of the different modes are fused;
step six, classifying the fused emotional characteristics; wherein, the emotion recognition classifier used for classification adopts a machine learning classifier or utilizes an artificial neural network to directly perform classification recognition;
and seventhly, outputting the identification result of the relevant factors of the depression.
2. The method for identifying the depressive disorder related factors based on multi-source information fusion according to claim 1, characterized in that: the machine learning classifier comprises a support vector machine and a random forest.
3. The method for identifying the depressive disorder related factors based on multi-source information fusion according to claim 1 or 2, wherein: step six, the step of classifying the fused emotional features adopts a K nearest neighbor classifier based on statistics to classify, and the calculation step comprises the following steps:
calculating the distance: given a test object, calculating its distance to each object in the training set;
and finding neighbors: defining k objects with the nearest distance as the neighbors of the test objects;
classification is carried out: classifying the test object according to the main category to which the k neighbors belong;
the K-nearest neighbor classifier has three basic elements:
(1) distance measurement:
when p is 2, the Euclidean distance is obtained; when p is 1, manhattan distance;
(2) selection of the value of K:
the smaller the K value is, the more complex the whole model is, and overfitting is easy to occur; the larger the value of K, the simpler the overall model, and the larger the approximation error (misclassification);
(3) determination of test object class:
a) majority voting:
wherein v is a class number, yiIs a nearest neighbor class label, I (.) is an indication function, if its parameter is true, returns 1, otherwise returns 0;
b) distance weighted voting:
where d is the distance between the two objects.
4. The method for identifying the depressive disorder related factors based on multi-source information fusion according to claim 1 or 2, wherein: step six, the step of classifying the fused emotional features specifically comprises the following steps:
and (3) classifying by adopting a decision tree classifier based on discrimination, and setting the sample set as T, wherein the gini coefficient value of T can be calculated by the following formula:
wherein p isjMeans that the category j is in the sample set TThe probability of occurrence.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110701988.4A CN113392918A (en) | 2021-06-24 | 2021-06-24 | Depressive disorder related factor identification method based on multi-source information fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110701988.4A CN113392918A (en) | 2021-06-24 | 2021-06-24 | Depressive disorder related factor identification method based on multi-source information fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113392918A true CN113392918A (en) | 2021-09-14 |
Family
ID=77623682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110701988.4A Pending CN113392918A (en) | 2021-06-24 | 2021-06-24 | Depressive disorder related factor identification method based on multi-source information fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113392918A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114093025A (en) * | 2021-10-29 | 2022-02-25 | 济南大学 | Man-machine cooperation method and system for multi-mode intention reverse active fusion |
| CN115631772A (en) * | 2022-10-27 | 2023-01-20 | 四川大学华西医院 | Method and device for evaluating risk of suicide injury, electronic equipment and storage medium |
| CN116204851A (en) * | 2023-03-21 | 2023-06-02 | 中关村科学城城市大脑股份有限公司 | Event recognition method and system based on multi-mode recognition technology |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109171769A (en) * | 2018-07-12 | 2019-01-11 | 西北师范大学 | It is a kind of applied to depression detection voice, facial feature extraction method and system |
| CN109614895A (en) * | 2018-10-29 | 2019-04-12 | 山东大学 | A method of the multi-modal emotion recognition based on attention Fusion Features |
| CN111199205A (en) * | 2019-12-30 | 2020-05-26 | 科大讯飞股份有限公司 | Vehicle-mounted voice interaction experience evaluation method, device, equipment and storage medium |
| CN111553295A (en) * | 2020-05-01 | 2020-08-18 | 北京邮电大学 | Multi-mode emotion recognition method based on self-attention mechanism |
| US20210015416A1 (en) * | 2018-12-14 | 2021-01-21 | Shenzhen Institutes Of Advanced Technology | Method for evaluating multi-modal emotion cognition capability of patient with autism spectrum disorder |
| CN112768070A (en) * | 2021-01-06 | 2021-05-07 | 万佳安智慧生活技术(深圳)有限公司 | Mental health evaluation method and system based on dialogue communication |
-
2021
- 2021-06-24 CN CN202110701988.4A patent/CN113392918A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109171769A (en) * | 2018-07-12 | 2019-01-11 | 西北师范大学 | It is a kind of applied to depression detection voice, facial feature extraction method and system |
| CN109614895A (en) * | 2018-10-29 | 2019-04-12 | 山东大学 | A method of the multi-modal emotion recognition based on attention Fusion Features |
| CN110751208A (en) * | 2018-10-29 | 2020-02-04 | 山东大学 | Criminal emotion recognition method for multi-mode feature fusion based on self-weight differential encoder |
| US20210015416A1 (en) * | 2018-12-14 | 2021-01-21 | Shenzhen Institutes Of Advanced Technology | Method for evaluating multi-modal emotion cognition capability of patient with autism spectrum disorder |
| CN111199205A (en) * | 2019-12-30 | 2020-05-26 | 科大讯飞股份有限公司 | Vehicle-mounted voice interaction experience evaluation method, device, equipment and storage medium |
| CN111553295A (en) * | 2020-05-01 | 2020-08-18 | 北京邮电大学 | Multi-mode emotion recognition method based on self-attention mechanism |
| CN112768070A (en) * | 2021-01-06 | 2021-05-07 | 万佳安智慧生活技术(深圳)有限公司 | Mental health evaluation method and system based on dialogue communication |
Non-Patent Citations (5)
| Title |
|---|
| ZEYU PAN等: "DEPRESSION DETECTION BASED ON REACTION TIME AND EYE MOVEMENT", 《ICIP》 * |
| 宋冠军等: "音视频双模态情感识别融合框架研究", 《计算机工程与应用》 * |
| 潘莹: "情感识别综述", 《电脑知识与技术》 * |
| 王璐等: "缓解期抑郁障碍患者面孔情绪识别特征的研究", 《临床精神医学杂志》 * |
| 黄越: "抑郁倾向的辅助识别研究——基于多模态特征的探索", 《中国优秀硕士学位论文全文数据库 医药卫生科技辑》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114093025A (en) * | 2021-10-29 | 2022-02-25 | 济南大学 | Man-machine cooperation method and system for multi-mode intention reverse active fusion |
| CN115631772A (en) * | 2022-10-27 | 2023-01-20 | 四川大学华西医院 | Method and device for evaluating risk of suicide injury, electronic equipment and storage medium |
| CN116204851A (en) * | 2023-03-21 | 2023-06-02 | 中关村科学城城市大脑股份有限公司 | Event recognition method and system based on multi-mode recognition technology |
| CN116204851B (en) * | 2023-03-21 | 2023-08-22 | 中关村科学城城市大脑股份有限公司 | Event recognition method and system based on multi-mode recognition technology |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Alghowinem et al. | Interpretation of depression detection models via feature selection methods | |
| CN108805087B (en) | Time sequence semantic fusion association judgment subsystem based on multi-modal emotion recognition system | |
| CN108805089B (en) | Multi-modal-based emotion recognition method | |
| CN108877801B (en) | Multi-turn dialogue semantic understanding subsystem based on multi-modal emotion recognition system | |
| CN108899050B (en) | Voice signal analysis subsystem based on multi-modal emotion recognition system | |
| CN108805088B (en) | Physiological signal analysis subsystem based on multi-modal emotion recognition system | |
| Xiao et al. | Follow the sound of children’s heart: a deep-learning-based computer-aided pediatric CHDs diagnosis system | |
| CN113392918A (en) | Depressive disorder related factor identification method based on multi-source information fusion | |
| US11950020B2 (en) | Methods and apparatus for displaying, compressing and/or indexing information relating to a meeting | |
| US20220037022A1 (en) | Ensemble machine-learning models to detect respiratory syndromes | |
| Chen et al. | A robust deep learning framework based on spectrograms for heart sound classification | |
| Zhao et al. | Multi-head attention-based long short-term memory for depression detection from speech | |
| Alqaysi et al. | Hybrid diagnosis models for autism patients based on medical and sociodemographic features using machine learning and multicriteria decision-making (MCDM) techniques: an evaluation and benchmarking framework | |
| Lu et al. | Speech depression recognition based on attentional residual network | |
| Zhou et al. | Tamfn: Time-aware attention multimodal fusion network for depression detection | |
| CN113868374B (en) | Graph convolution network biomedical information extraction method based on multi-head attention mechanism | |
| CN114191665A (en) | Method and device for classifying man-machine asynchronous phenomena in mechanical ventilation process | |
| Beriwal et al. | Techniques for suicidal ideation prediction: a qualitative systematic review | |
| CN107085655B (en) | Traditional Chinese medicine data processing method and system based on attribute constraint concept lattice | |
| Abhishek et al. | Diagnosis of Parkinson’s disorder through speech data using machine learning algorithms | |
| Bi et al. | Detecting depression on Sina microblog using depressing domain lexicon | |
| Sunsirikul et al. | Associative classification mining in the behavior study of autism spectrum disorder | |
| Touahria et al. | Feature selection algorithms highlight the importance of the systolic segment for normal/murmur PCG beat classification | |
| Begum et al. | Effective Techniques for Depression Detection on Social Media: A Comprehensive Review | |
| Rao et al. | Parkinson's Disease Detection Using Voice and Spiral Drawing Dataset |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210914 |