CN112618911A - Music feedback adjusting system based on signal processing - Google Patents
Music feedback adjusting system based on signal processing Download PDFInfo
- Publication number
- CN112618911A CN112618911A CN202011630898.2A CN202011630898A CN112618911A CN 112618911 A CN112618911 A CN 112618911A CN 202011630898 A CN202011630898 A CN 202011630898A CN 112618911 A CN112618911 A CN 112618911A
- Authority
- CN
- China
- Prior art keywords
- music
- emotion
- signal
- unit
- signals
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 18
- 230000008451 emotion Effects 0.000 claims abstract description 120
- 238000000034 method Methods 0.000 claims abstract description 45
- 230000002996 emotional effect Effects 0.000 claims abstract description 39
- 230000008921 facial expression Effects 0.000 claims abstract description 38
- 238000012544 monitoring process Methods 0.000 claims abstract description 26
- 238000004458 analytical method Methods 0.000 claims abstract description 8
- 230000004044 response Effects 0.000 claims abstract description 7
- 238000001514 detection method Methods 0.000 claims description 23
- 230000001815 facial effect Effects 0.000 claims description 18
- 108090000623 proteins and genes Proteins 0.000 claims description 17
- 210000004709 eyebrow Anatomy 0.000 claims description 15
- 230000008909 emotion recognition Effects 0.000 claims description 13
- 238000012549 training Methods 0.000 claims description 12
- 230000005236 sound signal Effects 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000012417 linear regression Methods 0.000 claims description 7
- 230000007704 transition Effects 0.000 claims description 7
- 230000033764 rhythmic process Effects 0.000 claims description 6
- 238000005311 autocorrelation function Methods 0.000 claims description 4
- 238000012216 screening Methods 0.000 claims description 4
- 238000013528 artificial neural network Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000001755 vocal effect Effects 0.000 claims description 3
- 230000003750 conditioning effect Effects 0.000 claims 5
- 230000009123 feedback regulation Effects 0.000 claims 2
- 230000007246 mechanism Effects 0.000 abstract description 9
- 238000011156 evaluation Methods 0.000 abstract description 4
- 238000000554 physical therapy Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 230000000694 effects Effects 0.000 description 5
- 238000000605 extraction Methods 0.000 description 5
- 238000007637 random forest analysis Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 210000004556 brain Anatomy 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000009432 framing Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 208000027534 Emotional disease Diseases 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000747 cardiac effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 229940079593 drug Drugs 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000006397 emotional response Effects 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M21/00—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
- A61M21/02—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis for inducing sleep or relaxation, e.g. by direct nerve stimulation, hypnosis, analgesia
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
- G06F16/65—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
- G10L25/63—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for estimating an emotional state
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/70—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mental therapies, e.g. psychological therapy or autogenous training
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
- A61M21/00—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
- A61M2021/0005—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis by the use of a particular sense, or stimulus
- A61M2021/0027—Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis by the use of a particular sense, or stimulus by the hearing sense
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Psychiatry (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Public Health (AREA)
- Multimedia (AREA)
- Veterinary Medicine (AREA)
- Psychology (AREA)
- Heart & Thoracic Surgery (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Biomedical Technology (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Child & Adolescent Psychology (AREA)
- Animal Behavior & Ethology (AREA)
- Hospice & Palliative Care (AREA)
- Data Mining & Analysis (AREA)
- Medical Informatics (AREA)
- Social Psychology (AREA)
- Anesthesiology (AREA)
- Surgery (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Developmental Disabilities (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Acoustics & Sound (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Educational Technology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Physiology (AREA)
Abstract
The invention discloses a music feedback adjusting system based on signal processing, which comprises a music library, a music library unit, an emotion analyzing unit, an emotion real-time monitoring unit and a music playing unit. The music library unit classifies the music in the music library according to the acoustic characteristics and lyric semantics of the music; the emotion analysis unit judges the initial emotional state of the subject according to the audio features; the emotion real-time monitoring unit realizes real-time monitoring of the emotion of the testee through physiological signals and facial expression signals of the testee; the music playing unit plays corresponding music according to the initial emotion state and switches the music in real time according to the result of the emotion real-time monitoring unit. The method realizes the accurate matching of the music played by the testee based on the accurate evaluation of the emotional state of the testee and the accurate classification of the music; the music switching device can automatically realize switching of played music in response to changes of the physical reaction mechanism of a subject, has a self-learning self-adjusting function, and has a function of suppressing emotional pseudo-variation in switching of music.
Description
Technical Field
The invention relates to the field of artificial intelligence, in particular to a music playing system based on a feedback evaluation mechanism for speech emotion evaluation and response to a body reaction mechanism.
Background
In the face of the ever-changing society and the increasingly accelerated pace of life, the pressure born by professional people is increasing day by day, and the phenomenon of psychological sub-health is more and more common. In addition to relieving the condition of the patient through medicines, a music physiotherapy method, which is a physiotherapy mechanism through physiological actions, is provided for relieving the physical and mental stress of the patient by playing music in a quiet environment. Practice proves that the mode can act on the neurons of the patient to relieve the tension of the patient.
However, although a systematic physiotherapy mechanism is established in the existing music physiotherapy method, appropriate music cannot be accurately and pertinently provided for a patient, the emotional state of the patient is determined by randomly playing music to observe the emotional response of the patient, and after a preset time interval, the state of the patient is re-evaluated by the method, and the physiotherapy effect is realized by repeated iteration. In the mode, firstly, the patient is required to be completely matched with the physical therapy process, and the physical therapy process can be interrupted for many times, so that the user experience is seriously influenced; furthermore, this method may also have the counter effect of selecting music for playback that increases the stress on the patient.
Disclosure of Invention
The invention aims to: in view of the above problems, a music feedback adjustment system based on signal processing is provided to provide a matched music piece for a subject on the basis of accurately evaluating the emotional state of the subject, and automatically adjust the played music piece in response to the physical reaction mechanism of the subject.
The technical scheme adopted by the invention is as follows:
a music feedback adjusting system based on signal processing comprises a music library, a music library unit, an emotion analyzing unit, an emotion real-time monitoring unit and a music playing unit;
the music library unit classifies the music in the music library according to the acoustic characteristics and the lyric semantics of the music;
the emotion analysis unit extracts the audio features of the acquired audio signals and judges the initial emotion state of the subject according to the audio features;
the emotion real-time monitoring unit is used for realizing real-time monitoring of the emotion of the testee through the physiological signal and the facial expression signal of the testee, and comprises the steps of taking the detection results of the physiological signal and the facial expression signal as items to be weighted, taking the emotion state at the previous moment as items to be weighted, respectively calculating the weight values of the physiological signal detection result, the facial expression signal detection result and the emotion state at the previous moment by using an entropy weight weighting method, and calculating the emotion monitoring result according to the weight values;
the music playing unit selects corresponding music from the music library classified by the music library unit to play according to the initial emotion state of the subject judged by the emotion analysis unit; and selecting corresponding music from the music library according to the real-time monitoring result of the emotion monitoring unit on the emotion of the subject, and switching the currently played music to the selected music.
The system can accurately evaluate the emotional state and the change condition of the testee, automatically adjust the played music in response to the emotional state and the fluctuation of the testee, and has the effect of self-learning and self-adjustment.
Further, the physiological signals comprise electrocardiosignals and electroencephalogram signals; the method for respectively calculating the physiological signal detection result, the facial expression signal detection result and the weight value of the emotional state at the previous moment by using the entropy weight weighting method and taking the detection results of the physiological signal and the facial expression signal as the items to be weighted, and the method comprises the following steps:
extracting electrocardiosignal characteristics of electrocardiosignals of a testee, and carrying out emotion classification identification according to the electrocardiosignal characteristics;
extracting electroencephalogram signal characteristics of electroencephalograms of the subjects, and performing emotion classification identification according to the electroencephalogram signal characteristics;
extracting facial features of facial expression signals of the testee, and performing emotion classification and identification according to the facial features;
and integrating emotion classification results obtained based on electrocardiosignals, electroencephalogram signals and facial expression signals and the emotion state at the previous moment as to-be-weighted items, and performing weight distribution by adopting an entropy weight method weighting mode.
Further, the extracting the electrocardiosignal characteristics of the electrocardiosignals of the testee and performing emotion classification recognition according to the electrocardiosignal characteristics comprises:
extracting the electrocardiosignal characteristics of the sample:
3/4 of the maximum value of the electrocardiosignal is set as a threshold value, and the local maximum value is positioned in a continuous limited signal segment exceeding the threshold value to obtain an electrocardio R peak;
calculating the average value and standard deviation of RR intervals in unit time;
calculating the root mean square of the difference values of adjacent RR intervals;
calculating the ratio of the total number of RR intervals to the height of the R peak;
calculating the difference between the maximum value and the minimum value of the RR interval;
counting the number of RR intervals exceeding a preset time length;
training the electrocardiosignal characteristics of the sample by using a classifier model to obtain an emotion classifier based on the electrocardiosignals; and identifying the electrocardiosignals of the testee by using the emotion classifier based on the electrocardiosignals.
Further, the extraction of the electroencephalogram signal characteristics of the electroencephalogram signal of the subject and the emotion classification and identification according to the electroencephalogram signal characteristics comprise:
respectively carrying out the following operations on the sample electroencephalogram signals:
denoising the sample electroencephalogram signal;
respectively calculating power values of five wave bands of alpha, beta, gamma, theta and delta of the sample electroencephalogram signals as characteristics of the electroencephalogram signals;
training the electroencephalogram signal characteristics by using a classifier model to obtain an emotion classifier based on the electroencephalogram signal;
and carrying out emotion recognition on the electroencephalogram signals of the subject by using the emotion classifier based on the electroencephalogram signals.
Further, the extracting facial features of the facial expression signals of the subject and performing emotion classification recognition according to the facial features comprises:
the following operations are respectively performed on the sample facial expression signals:
detecting the characteristic points of the human face by using a DLib open source neural network;
respectively calculating the eyebrow raising degree, the eyebrow inclination degree, the eye opening degree and the mouth breaking degree based on the detected face characteristic points as facial expression characteristics;
training the facial expression feature by using a classifier model to obtain an emotion classifier based on facial features;
facial expressions of the subject are identified using a facial feature-based emotion classifier.
Further, the fusion is based on the emotion classification result obtained by the electrocardiosignal, the electroencephalogram signal and the facial expression signal, and the emotion state at the previous moment is taken as a to-be-weighted item, and weight distribution is performed by adopting an entropy weight method weighting mode, and the method comprises the following steps:
respectively carrying out normalization calculation on a result of emotion classification recognition according to electrocardiosignal characteristics, a result of emotion classification recognition according to electroencephalogram signal characteristics, a result of emotion classification recognition according to facial characteristics and an emotion recognition result at the previous moment;
respectively calculating the information entropy of each classification result;
and calculating the weight of each weighted item according to the information entropy of each classification result.
Further, the classifying the music in the music library according to the acoustic characteristics and the lyric semantics of the music comprises:
screening out music with negative emotion characteristic tendency in the music library by adopting emotion characteristics combined with lyric semantics; and then, classifying the rest music by using acoustic features representing the four aspects of tone color, loudness, tone and rhythm.
Further, the acoustic features characterizing the timbre include: mel-frequency cepstral coefficients; the features characterizing loudness include: short-time energy, short-time energy jitter, and short-time energy linear regression coefficients; the intensity of the rhythm acoustic characteristic is represented by using the short-time autocorrelation function peak value; the acoustic features characterizing the pitch include: pitch frequency, first order gene frequency jitter, second order gene frequency jitter.
Further, the audio features of the audio signal include mel-frequency cepstrum coefficients, short-time energy jitter, short-time energy linear regression coefficients, fundamental frequency, first-order gene frequency jitter, second-order gene frequency jitter, and formant frequency and bandwidth; the solving method of the formant frequency and the bandwidth comprises the following steps:
and (3) deconvolving the voice signal by using an LPC method to obtain a holopolar model parameter of the vocal tract response:
solving the complex root of A (z)
Is a root of A (z), then its conjugate complex value
A root also denoted A (z), the formant frequency corresponding to i being denoted FiThe bandwidth is represented as BiAnd then:
where T is the sampling period.
Further, the music playing unit adopts a constant power transition mode to switch the currently played music. Therefore, the phenomenon that the physiotherapy effect is influenced by the excessive hardness of the music during switching can be avoided.
In summary, due to the adoption of the technical scheme, the invention has the beneficial effects that:
1. the method has extremely high rigor, and realizes the accurate matching of the music played by the testee based on the accurate evaluation of the emotional state of the testee and the accurate classification of the music.
2. The method of the invention responds to the change of the physical reaction mechanism of the subject, can automatically realize the switching of the played music and has the self-learning and self-adjusting function. In addition, the switching music adopts a constant power transition mode, so that the harsh condition of the switching process can not be caused, and the experience of a subject can not be interrupted or influenced.
3. The invention judges the emotion of the testee by taking the body reaction mechanism of the testee into consideration and combining the emotional state of the testee at the previous moment by using a weighting mode, and can inhibit the emotional pseudo-variation caused by noise signals and the like.
Drawings
The invention will now be described, by way of example, with reference to the accompanying drawings, in which:
fig. 1 is a flow chart of the operation of a music feedback adjustment system based on signal processing.
Fig. 2 is a flowchart of classification of music pieces.
FIG. 3 is a flow chart of word semantic emotional feature propensity score calculation.
FIG. 4 is a flow chart of calculating emotion monitoring results by entropy weighting.
Fig. 5 is a face feature point label diagram.
Detailed Description
All of the features disclosed in this specification, or all of the steps in any method or process so disclosed, may be combined in any combination, except combinations of features and/or steps that are mutually exclusive.
Any feature disclosed in this specification (including any accompanying claims, abstract) may be replaced by alternative features serving equivalent or similar purposes, unless expressly stated otherwise. That is, unless expressly stated otherwise, each feature is only an example of a generic series of equivalent or similar features.
Description of the prior art to which embodiments of the present invention may relate:
a method for extracting MFCC is described in the third edition of Speech Signal processing, editions by Zhao Li et al, Press of mechanical industries, ISBN 978-7-111-.
The embodiment discloses a music feedback adjusting system based on signal processing, the operation flow of the system is shown in figure 1, the system comprises a music library, and a plurality of music is stored in the music library; the system further comprises:
and the music library unit classifies the music in the music library according to the acoustic characteristics and the lyric semantics of the music.
In some embodiments, the classification of the musical composition results in classifying the musical compositions in the music library into a class a, B, C and D. As shown in fig. 2, the specific classification process includes:
and screening the music with negative emotion characteristic tendency in the music library by adopting the emotion characteristic combined with the lyric semantics. And then, dividing the rest (emotional feature tendency is positive) music into A type, B type, C type and D type by using acoustic features representing the four aspects of tone color, loudness, tone and rhythm.
And aiming at the emotional characteristics of the lyric semantics, adopting an emotion classification method based on a dictionary. Firstly, the hidden Markov word segmentation model obtained by training the open source database is used for analyzing the word segmentation and the part of speech of the lyric piece of the music. And then calculating the total emotional feature tendency scores of the nouns and the adjectives based on the open-source BosonNLP emotional dictionary. And if the total score is negative, judging that the emotional feature tendency of the corresponding music is negative, and if the total score is positive, judging that the emotional feature tendency of the corresponding music is positive. And (5) screening out music with negative emotional characteristic tendency from the music library.
The method for calculating the lyric semantic emotional feature tendency score, as shown in fig. 3, comprises the following steps:
traversing each noun and adjective in the word segmentation result, and if the word is a positive word (defined in the field), taking the score of the word as the weight of the word; if it is a negative word, its score is the negative of its weight. Detecting whether two words in the preamble of the word are degree adverbs or negative words, if the degree adverbs exist, multiplying the score of the word by the weight of the degree adverbs, and if the negative words exist, taking the score of the word as a first order negative number.
The step of classifying the remaining music pieces by their acoustic features comprises:
randomly intercepting a section of the music with the duration T (for example, 10s), and taking the acoustic features of the section as the acoustic features of the music. The method for extracting the characteristics of various aspects of the acoustic features comprises the following steps:
1) acoustic features characterizing tone color: mel-frequency cepstral coefficients (MFCCs).
For the extraction of acoustic features representing timbre, the extraction method of MFCC may be directly used.
2) Acoustic features characterizing loudness: short-term energy, short-term energy jitter, and short-term energy linear regression coefficients.
Windowing and framing preprocessing is carried out on the music, and the nth frame voice signal obtained after windowing and framing preprocessing is recorded as xn(m), then its short-time energy is expressed as:
where N is the frame length.
The short-time energy jitter is:
where M represents the total number of frames.
The short-time energy linear regression coefficient is:
3) acoustic features that characterize tempo:
and characterizing the rhythm intensity by using the short-time autocorrelation function peak value.
The short-time autocorrelation function of a speech signal is:
by Rn(k) Gene period values were extracted from the peak positions and were designated as T0iTaking the total frame number of the audio signal as M, the tempo strength can be represented by the following equation:
4) acoustic features characterizing pitch: pitch frequency, first order gene frequency jitter, second order gene frequency jitter.
The reciprocal of the period value of the gene is the gene frequency, i.e.
F0i＝1/T0i
Record the voiced frame number of the voice signal as M*Then the first order gene frequency is dithered F0s1And second order gene frequency dithering F0s2The following equation is used to calculate:
and training the acoustic features and corresponding classification of the music serving as the sample by using a support vector machine to obtain a four-classifier based on the acoustic features of the music, and classifying the music into corresponding classes based on the acoustic features of the music by using the four-classifier.
And the emotion analysis unit analyzes the acquired audio signal of the subject, extracts audio features and judges the initial emotional state of the subject according to the audio features.
The emotion analysis unit divides the audio signals into four types with large differences according to a Thayer two-dimensional emotion model, wherein the four types are a type, b type, c type and d type.
And judging the emotional state, firstly, filtering and denoising the acquired audio signal, and then extracting audio features, wherein the audio features comprise a Mel frequency cepstrum coefficient, short-time energy jitter, a short-time energy linear regression coefficient, a fundamental tone frequency, first-order gene frequency jitter, second-order gene frequency jitter, a formant frequency and a bandwidth.
Solving formant frequency and bandwidth, firstly, deconvolving a speech signal by using an LPC method to obtain a holopolar model parameter of vocal tract response:
then, the complex root of A (z) is obtained, and the formant parameter can be obtained. Is provided with
Is a root of A (z), then its conjugate complex value
A root also denoted A (z), the formant frequency corresponding to i being denoted FiThe 3dB bandwidth is denoted as BiAnd then:
where T is the sampling period.
The audio characteristics and the corresponding emotional states of the audio signals serving as the samples are trained by using a support vector machine to obtain an emotional state recognition model based on the audio signals, and the initial emotional state of the subject (the audio signals of the subject) can be recognized by using the emotional state recognition model.
The real-time emotion monitoring unit is used for realizing multi-mode real-time emotion monitoring on the testee through physiological signals (electrocardiosignals and electroencephalogram signals) and facial expression signals of the testee.
The emotional states of the subject are classified into a class a, b class c class and d class corresponding to the classification of the emotional state by the emotion analyzing unit. And respectively carrying out classification training on the characteristics corresponding to the physiological signals and the facial expression signals of the samples to obtain corresponding emotion recognition classifiers. In order to improve the stability of the emotion recognition result, in this embodiment, the detection result of the physiological signal and the facial expression signal represented by the probability is used as the item to be weighted, the emotion state at the previous moment is also used as the item to be weighted, and the entropy weighting method is used to obtain the physiological signal, the detection mode of the facial expression signal and the weight value of the emotion state at the previous moment, so as to obtain the final (monitored) emotion recognition result, which is shown in fig. 4.
In some embodiments, the physiological signals include cardiac electrical signals and brain electrical signals.
1) Emotion detection based on electrocardiosignals
The electrocardiosignals are human physiological signals, contain rich physiological information and can reflect the physiological characteristics of human bodies to a certain extent. The heart rate variability refers to the phenomenon of slight difference or slight fluctuation between RR intervals (instantaneous heart rate) of successive heartbeats. Studies have shown that individual emotional changes are reflected in heart rate variability. Therefore, the real-time emotion monitoring unit adds electrocardiosignal characteristics to the multi-modal emotion real-time monitoring of the testee. The emotion detection process based on the electrocardiosignals comprises the steps of extracting electrocardiosignal characteristics of the electrocardiosignals of the testee and carrying out emotion classification and identification according to the electrocardiosignal characteristics.
The extraction of the electrocardiosignal characteristics of the electrocardiosignals comprises the following steps:
by setting 3/4 of the maximum value of the electrocardiosignal as a threshold value, the local maximum value is positioned in a continuous limited signal segment exceeding the threshold value to obtain the electrocardio R peak. The mean and standard deviation of the RR intervals per unit time (in minutes) are calculated as follows:
wherein N is the total number of RR intervals in unit time.
The root mean square of the difference between adjacent RR intervals is calculated, the formula is as follows,
calculating the ratio of the total number of RR intervals to the height of the R peak:
wherein y isRR(i) The peak value of the R wave.
The electrocardiosignal characteristics also comprise the difference between the maximum value and the minimum value of the RR interval:
ΔRR＝max(RR)-min(RR)
and a number N of RR intervals exceeding 50ms50(time adjustable).
The five characteristics (average value and standard deviation of RR intervals in unit time, root mean square of difference values of adjacent RR intervals, ratio of total number of RR intervals to height of R peak, difference between maximum value and minimum value of RR intervals, and number N of RR intervals exceeding 50ms50) As the electrocardiosignal characteristics, a random forest classifier (or other classifier models, the same applies below) is used for training to obtain a corresponding emotion recognition classifier based on the electrocardiosignals. The classifier is used for identifying the electrocardiosignals of the testee to obtain a corresponding emotion identification result.
2) Emotion detection based on electroencephalogram signals
The electroencephalogram signals can reflect the real emotion of a person, the electroencephalogram signal characteristics of the electroencephalogram signals of the subject can be extracted, and emotion classification recognition is carried out according to the electroencephalogram signal characteristics. Different wave band powers of the brain electricity reflect different emotions. In the embodiment, the power values of five wave bands of alpha, beta, gamma, theta and delta are used as the characteristics of the electroencephalogram signals.
The electroencephalogram signal is weak, other noise interference exists in the acquisition process, and before feature extraction, a band-pass filter is used for denoising the electroencephalogram signal. Marking the EEG original signal as ee (n), and marking the filtered signal as e (n), then
e(n)＝ee(n)*f(n)
Wherein f (n) is a band-pass filter, and the upper and lower cut-off frequencies are 50Hz and 0.3Hz, respectively.
Calculating the power spectrum of the electroencephalogram signal according to the following calculation principle:
wherein
And respectively calculating the signal power of the five wave bands by using the power spectral density:
wherein f isHi、fLiRespectively, the upper limit value and the lower limit value of the wave band frequency.
And taking the power values of the five wave bands of the sample as electroencephalogram signal characteristics, and training by using a random forest classifier to obtain an emotion classifier based on the electroencephalogram signal. The classifier is used for identifying the electroencephalogram signals of the testee to obtain corresponding emotion identification results.
3) Emotion detection based on facial expressions
The human facial expression has a great reference value for real emotional reaction, so that facial expression detection based on human face characteristic points is integrated into a multi-mode real-time emotion monitoring mechanism, namely facial characteristics of facial expression signals of a subject are extracted, and emotion classification recognition is carried out according to the facial characteristics. The method mainly calculates the eyebrow picking or frowning state, the eye opening degree and the mouth opening degree through the face characteristic points, and takes the eyebrow picking or frowning state, the eye opening degree and the mouth opening degree as characteristic values to carry out emotion recognition, and the specific method comprises the following steps:
and for the obtained real-time face image, carrying out face characteristic point detection by using a DLib open source neural network, and obtaining a face detection frame and characteristic point coordinates. The number of the human face characteristic points is 68, wherein the characteristic points 18-22 and 23-27 represent the left and right eyebrows, the characteristic points 37-42 and 43-48 represent the eyes and the characteristic points 49-55 represent the upper and lower lips respectively. The feature point labels are shown in fig. 5.
The lift on the eyebrows and the inclination of the eyebrows can reflect the emotion of a person to a certain extent, such as happy or angry. Therefore, the average Euclidean distance from the eyebrow feature points to the upper frame of the face detection frame is calculated to serve as the eyebrow raising degree. The calculation formula is as follows,
wherein y is the vertical coordinate of the upper frame of the face frame.
Four slopes can be obtained from the end points (No. 18, No. 22, No. 23, No. 27) of the left and right eyebrows and the corresponding eyebrow center points (No. 20, No. 25), and the average of the four slopes is used as the inclination degree of the eyebrows. The calculation formula is as follows,
when excited, a person can open the eyes. The average area of the eyes of both eyes is calculated as a measure of the degree of opening of the eyes. Taking the left eye as an example, the area calculation formula is as follows:
SLeye＝(d1+d2)l/3
wherein d is1The euclidean distances of feature points No. 44 and No. 48. l is the Euclidean distance between feature points No. 43 and No. 46.
The greater the degree of grin indicates to some extent that the emotion is more excited, and therefore, the degree of grin is also taken as one feature value of facial expression recognition. The characterization mode is the ratio of the distance between the feature points No. 49 and No. 55 to the transverse length of the face frame, the calculation formula is as follows,
wherein x is the transverse width of the face frame, and x and y with subscripts are the horizontal and vertical coordinates of the corresponding characteristic points.
And (3) performing classification training by using a random forest by taking the four values (the eyebrow raising degree, the eyebrow inclination degree, the eye opening degree and the mouth breaking degree) of the sample as characteristic values to obtain the emotion classifier based on the facial features. And identifying the facial expression signals (real-time images of human faces) of the testee by using an emotion classifier based on the facial features to obtain corresponding emotion identification results.
4) Entropy weight-weighting method fusion
In order to improve the accuracy of the emotion recognition result, the emotion classification result obtained based on the electrocardiosignals, the electroencephalogram signals and the facial expression signals and the emotion state at the previous moment are fused to serve as the items to be weighted, and weight distribution is carried out in an entropy weight weighting mode.
The entropy weight weighting method firstly normalizes the data of the to-be-weighted items.
Respectively recording the classification results of the three different emotion classifiers and the emotion recognition result of the previous moment as P1、P2、P3、P4For example, emotional states are classified into 4 categories, where Pi＝{p1,p2,p3,p4And represents probability values in 4 emotional states. The normalization calculation is carried out, and the formula is as follows:
Pijrepresenting the probability result calculated by the ith classifier for the jth emotion. And solving the information entropy of each classification result according to the information entropy definition, wherein the formula is as follows:
wherein
And calculating the weight of each weighted item according to the information entropy of each classification result:
therefore, the emotion recognition probability and the weight coefficient of the emotion state result at the previous moment which are respectively obtained by the three detection modes are obtained. And calculating the final emotion recognition result according to the weight to obtain:
the emotion classification indicated by the maximum value is used as the final real-time monitoring result of the emotion.
And the music playing unit (randomly) selects corresponding music from the music library classified by the music library unit to play according to the initial emotional state of the testee judged by the emotion analysis unit. And according to the real-time monitoring result of the emotion monitoring unit on the emotion of the subject, selecting corresponding music from the music library (randomly), and switching the currently played music to the selected music. Generally, the music playing unit stops playing the music when the real-time emotion monitoring unit detects that the emotional state of the subject is at a low stress level (e.g., a normal or relaxed state). Certainly, in order to avoid influencing the physiotherapy effect, the playing can be finished by gradually reducing the volume, finishing the playing after the current music is played, and the like.
There is a correspondence between the classification of the music and the classification of the emotional state, for example, there is the following correspondence:
| emotional state of a person | Class a | Class b | class c | Class d |
| Playing music classification | Class A | Class B | Class C | Class D |
In some embodiments, the music is divided into four categories of relaxation, happiness, relaxation and delicacy, the emotional state is divided into four categories of anger, difficulty, excitement and normal, and the correspondence between the music category and the emotional state category is shown in the following table:
| emotional state of a person | Generating qi | Difficult to pass | Excitement | Is normal |
| Playing music classification | Relief and relieve | Happy music | Easy to use | Soft and beautiful |
In order to avoid the harsh transition process during the music switching, a constant power transition mode is adopted for the music switching. I.e. slowly lowering the audio being played and then quickly approaching the end of the transition. For audio to be played, the audio is first increased quickly and then more slowly towards the end of the transition. In another switching manner, for switching or ending of music, it is preferable to switch to the next music or end playing when the currently played music is played.
The invention is not limited to the foregoing embodiments. The invention extends to any novel feature or any novel combination of features disclosed in this specification and any novel method or process steps or any novel combination of features disclosed.
Claims (10)
1. A music feedback adjusting system based on signal processing is characterized by comprising a music library, a music library unit, an emotion analyzing unit, an emotion real-time monitoring unit and a music playing unit;
the music library unit classifies the music in the music library according to the acoustic characteristics and the lyric semantics of the music;
the emotion analysis unit extracts the audio features of the acquired audio signals and judges the initial emotion state of the subject according to the audio features;
the emotion real-time monitoring unit is used for realizing real-time monitoring of the emotion of the testee through the physiological signal and the facial expression signal of the testee, and comprises the steps of taking the detection results of the physiological signal and the facial expression signal as items to be weighted, taking the emotion state at the previous moment as items to be weighted, respectively calculating the weight values of the physiological signal detection result, the facial expression signal detection result and the emotion state at the previous moment by using an entropy weight weighting method, and calculating the emotion monitoring result according to the weight values;
the music playing unit selects corresponding music from the music library classified by the music library unit to play according to the initial emotion state of the subject judged by the emotion analysis unit; and selecting corresponding music from the music library according to the real-time monitoring result of the emotion monitoring unit on the emotion of the subject, and switching the currently played music to the selected music.
2. The signal processing based music feedback conditioning system of claim 1 wherein said physiological signals comprise electrocardiosignals and electroencephalographic signals; the method for respectively calculating the physiological signal detection result, the facial expression signal detection result and the weight value of the emotional state at the previous moment by using the entropy weight weighting method and taking the detection results of the physiological signal and the facial expression signal as the items to be weighted, and the method comprises the following steps:
extracting electrocardiosignal characteristics of electrocardiosignals of a testee, and carrying out emotion classification identification according to the electrocardiosignal characteristics;
extracting electroencephalogram signal characteristics of electroencephalograms of the subjects, and performing emotion classification identification according to the electroencephalogram signal characteristics;
extracting facial features of facial expression signals of the testee, and performing emotion classification and identification according to the facial features;
and integrating emotion classification results obtained based on electrocardiosignals, electroencephalogram signals and facial expression signals and the emotion state at the previous moment as to-be-weighted items, and performing weight distribution by adopting an entropy weight method weighting mode.
3. The music feedback regulation system based on signal processing as claimed in claim 2, wherein the extracting of the electrocardiosignal characteristics of the electrocardiosignal of the subject and the emotion classification recognition based on the electrocardiosignal characteristics comprises:
extracting the electrocardiosignal characteristics of the sample:
3/4 of the maximum value of the electrocardiosignal is set as a threshold value, and the local maximum value is positioned in a continuous limited signal segment exceeding the threshold value to obtain an electrocardio R peak;
calculating the average value and standard deviation of RR intervals in unit time;
calculating the root mean square of the difference values of adjacent RR intervals;
calculating the ratio of the total number of RR intervals to the height of the R peak;
calculating the difference between the maximum value and the minimum value of the RR interval;
counting the number of RR intervals exceeding a preset time length;
training the electrocardiosignal characteristics of the sample by using a classifier model to obtain an emotion classifier based on the electrocardiosignals; and identifying the electrocardiosignals of the testee by using the emotion classifier based on the electrocardiosignals.
4. The music feedback adjustment system based on signal processing as claimed in claim 2, wherein said extracting the electroencephalogram signal characteristics of the subject electroencephalogram signal, and performing emotion classification recognition according to the electroencephalogram signal characteristics comprises:
respectively carrying out the following operations on the sample electroencephalogram signals:
denoising the sample electroencephalogram signal;
respectively calculating power values of five wave bands of alpha, beta, gamma, theta and delta of the sample electroencephalogram signals as characteristics of the electroencephalogram signals;
training the electroencephalogram signal characteristics by using a classifier model to obtain an emotion classifier based on the electroencephalogram signal;
and carrying out emotion recognition on the electroencephalogram signals of the subject by using the emotion classifier based on the electroencephalogram signals.
5. The signal processing based music feedback adjustment system of claim 2, wherein said extracting facial features of the facial expression signal of the subject, performing emotion classification recognition based on the facial features, comprises:
the following operations are respectively performed on the sample facial expression signals:
detecting the characteristic points of the human face by using a DLib open source neural network;
respectively calculating the eyebrow raising degree, the eyebrow inclination degree, the eye opening degree and the mouth breaking degree based on the detected face characteristic points as facial expression characteristics;
training the facial expression feature by using a classifier model to obtain an emotion classifier based on facial features;
facial expressions of the subject are identified using a facial feature-based emotion classifier.
6. The music feedback regulation system based on signal processing as claimed in any one of claims 3 to 5, wherein the fusing of emotion classification results obtained based on electrocardiosignals, electroencephalogram signals and facial expression signals and emotion states at previous time as items to be weighted and the weighting distribution is performed by weighting with entropy weighting method, comprising:
respectively carrying out normalization calculation on a result of emotion classification recognition according to electrocardiosignal characteristics, a result of emotion classification recognition according to electroencephalogram signal characteristics, a result of emotion classification recognition according to facial characteristics and an emotion recognition result at the previous moment;
respectively calculating the information entropy of each classification result;
and calculating the weight of each weighted item according to the information entropy of each classification result.
7. The signal processing based music feedback conditioning system of claim 1 wherein said classifying the music tracks in the library according to their acoustic characteristics and lyric semantics comprises:
screening out music with negative emotion characteristic tendency in the music library by adopting emotion characteristics combined with lyric semantics; and then, classifying the rest music by using acoustic features representing the four aspects of tone color, loudness, tone and rhythm.
8. The signal processing based music feedback conditioning system of claim 7 wherein the acoustic features characterizing timbre comprise: mel-frequency cepstral coefficients; the features characterizing loudness include: short-time energy, short-time energy jitter, and short-time energy linear regression coefficients; the intensity of the rhythm acoustic characteristic is represented by using the short-time autocorrelation function peak value; the acoustic features characterizing the pitch include: pitch frequency, first order gene frequency jitter, second order gene frequency jitter.
9. The signal processing based music feedback conditioning system of claim 1, wherein the audio features of the audio signal comprise mel-frequency cepstral coefficients, short-time energy jitter, short-time energy linear regression coefficients, pitch frequency, first order gene frequency jitter, second order gene frequency jitter, and formant frequencies and bandwidths; the solving method of the formant frequency and the bandwidth comprises the following steps:
and (3) deconvolving the voice signal by using an LPC method to obtain a holopolar model parameter of the vocal tract response:
solving the complex root of A (z)
Is a root of A (z), then its conjugate complex value
A root also denoted A (z), the formant frequency corresponding to i being denoted FiThe bandwidth is represented as BiAnd then:
where T is the sampling period.
10. The signal processing based music feedback conditioning system of claim 1, wherein the music playing unit switches currently playing music tracks in a constant power transition manner.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011630898.2A CN112618911B (en) | 2020-12-31 | 2020-12-31 | Music feedback adjusting system based on signal processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011630898.2A CN112618911B (en) | 2020-12-31 | 2020-12-31 | Music feedback adjusting system based on signal processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112618911A true CN112618911A (en) | 2021-04-09 |
| CN112618911B CN112618911B (en) | 2023-02-03 |
Family
ID=75290623
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011630898.2A Active CN112618911B (en) | 2020-12-31 | 2020-12-31 | Music feedback adjusting system based on signal processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112618911B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113420177A (en) * | 2021-06-30 | 2021-09-21 | 广州酷狗计算机科技有限公司 | Audio data processing method and device, computer equipment and storage medium |
| CN114117116A (en) * | 2022-01-28 | 2022-03-01 | 中国传媒大学 | Music unlocking method based on biological characteristic interaction and electronic equipment |
Citations (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6230111B1 (en) * | 1998-08-06 | 2001-05-08 | Yamaha Hatsudoki Kabushiki Kaisha | Control system for controlling object using pseudo-emotions and pseudo-personality generated in the object |
| CN101822863A (en) * | 2010-01-28 | 2010-09-08 | 深圳先进技术研究院 | Emotion regulating device and method thereof |
| CN102499677A (en) * | 2011-12-16 | 2012-06-20 | 天津大学 | Emotional state identification method based on electroencephalogram nonlinear features |
| CN203075421U (en) * | 2012-07-31 | 2013-07-24 | 深圳市赛格导航科技股份有限公司 | Music playing system based on emotion change |
| CN107123019A (en) * | 2017-03-28 | 2017-09-01 | 华南理工大学 | A kind of VR shopping commending systems and method based on physiological data and Emotion identification |
| CN107145326A (en) * | 2017-03-28 | 2017-09-08 | 浙江大学 | A kind of the music automatic playing system and method for collection of being expressed one's feelings based on target face |
| CN107220591A (en) * | 2017-04-28 | 2017-09-29 | 哈尔滨工业大学深圳研究生院 | Multi-modal intelligent mood sensing system |
| CN107307872A (en) * | 2016-04-26 | 2017-11-03 | 阿里巴巴集团控股有限公司 | A kind of emotion adjustment method, mood sharing method and device |
| CN108682432A (en) * | 2018-05-11 | 2018-10-19 | 南京邮电大学 | Speech emotion recognition device |
| CN108899050A (en) * | 2018-06-14 | 2018-11-27 | 南京云思创智信息科技有限公司 | Speech signal analysis subsystem based on multi-modal Emotion identification system |
| CN109157231A (en) * | 2018-10-24 | 2019-01-08 | 阿呆科技（北京）有限公司 | Portable multi-channel Depression trend assessment system based on emotional distress task |
| CN109472290A (en) * | 2018-10-11 | 2019-03-15 | 南京邮电大学 | Mood swing model analysis method based on finite state machine |
| CN109620262A (en) * | 2018-12-12 | 2019-04-16 | 华南理工大学 | A kind of Emotion identification system and method based on wearable bracelet |
| CN109961776A (en) * | 2017-12-18 | 2019-07-02 | 上海智臻智能网络科技股份有限公司 | Speech information processing apparatus |
| US20190347476A1 (en) * | 2018-05-09 | 2019-11-14 | Korea Advanced Institute Of Science And Technology | Method for estimating human emotions using deep psychological affect network and system therefor |
| CN110569869A (en) * | 2019-07-23 | 2019-12-13 | 浙江工业大学 | feature level fusion method for multi-modal emotion detection |
| CN111068159A (en) * | 2019-12-27 | 2020-04-28 | 兰州大学 | Music feedback depression mood adjusting system based on electroencephalogram signals |
| CN111956219A (en) * | 2020-08-27 | 2020-11-20 | 济南大学 | Electroencephalogram signal-based emotion feature identification method and system and emotion feature identification and adjustment system |
| CN112120716A (en) * | 2020-09-02 | 2020-12-25 | 中国人民解放军军事科学院国防科技创新研究院 | Wearable multi-mode emotional state monitoring device |
-
2020
- 2020-12-31 CN CN202011630898.2A patent/CN112618911B/en active Active
Patent Citations (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6230111B1 (en) * | 1998-08-06 | 2001-05-08 | Yamaha Hatsudoki Kabushiki Kaisha | Control system for controlling object using pseudo-emotions and pseudo-personality generated in the object |
| CN101822863A (en) * | 2010-01-28 | 2010-09-08 | 深圳先进技术研究院 | Emotion regulating device and method thereof |
| CN102499677A (en) * | 2011-12-16 | 2012-06-20 | 天津大学 | Emotional state identification method based on electroencephalogram nonlinear features |
| CN203075421U (en) * | 2012-07-31 | 2013-07-24 | 深圳市赛格导航科技股份有限公司 | Music playing system based on emotion change |
| CN107307872A (en) * | 2016-04-26 | 2017-11-03 | 阿里巴巴集团控股有限公司 | A kind of emotion adjustment method, mood sharing method and device |
| CN107123019A (en) * | 2017-03-28 | 2017-09-01 | 华南理工大学 | A kind of VR shopping commending systems and method based on physiological data and Emotion identification |
| CN107145326A (en) * | 2017-03-28 | 2017-09-08 | 浙江大学 | A kind of the music automatic playing system and method for collection of being expressed one's feelings based on target face |
| CN107220591A (en) * | 2017-04-28 | 2017-09-29 | 哈尔滨工业大学深圳研究生院 | Multi-modal intelligent mood sensing system |
| CN109961776A (en) * | 2017-12-18 | 2019-07-02 | 上海智臻智能网络科技股份有限公司 | Speech information processing apparatus |
| US20190347476A1 (en) * | 2018-05-09 | 2019-11-14 | Korea Advanced Institute Of Science And Technology | Method for estimating human emotions using deep psychological affect network and system therefor |
| CN108682432A (en) * | 2018-05-11 | 2018-10-19 | 南京邮电大学 | Speech emotion recognition device |
| CN108899050A (en) * | 2018-06-14 | 2018-11-27 | 南京云思创智信息科技有限公司 | Speech signal analysis subsystem based on multi-modal Emotion identification system |
| CN109472290A (en) * | 2018-10-11 | 2019-03-15 | 南京邮电大学 | Mood swing model analysis method based on finite state machine |
| CN109157231A (en) * | 2018-10-24 | 2019-01-08 | 阿呆科技（北京）有限公司 | Portable multi-channel Depression trend assessment system based on emotional distress task |
| CN109620262A (en) * | 2018-12-12 | 2019-04-16 | 华南理工大学 | A kind of Emotion identification system and method based on wearable bracelet |
| CN110569869A (en) * | 2019-07-23 | 2019-12-13 | 浙江工业大学 | feature level fusion method for multi-modal emotion detection |
| CN111068159A (en) * | 2019-12-27 | 2020-04-28 | 兰州大学 | Music feedback depression mood adjusting system based on electroencephalogram signals |
| CN111956219A (en) * | 2020-08-27 | 2020-11-20 | 济南大学 | Electroencephalogram signal-based emotion feature identification method and system and emotion feature identification and adjustment system |
| CN112120716A (en) * | 2020-09-02 | 2020-12-25 | 中国人民解放军军事科学院国防科技创新研究院 | Wearable multi-mode emotional state monitoring device |
Non-Patent Citations (1)
| Title |
|---|
| 宋静，张雪英，孙颖，畅江: "基于模糊综合评价法的情感语音数据库的建立", 《现代电子技术》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113420177A (en) * | 2021-06-30 | 2021-09-21 | 广州酷狗计算机科技有限公司 | Audio data processing method and device, computer equipment and storage medium |
| CN114117116A (en) * | 2022-01-28 | 2022-03-01 | 中国传媒大学 | Music unlocking method based on biological characteristic interaction and electronic equipment |
| CN114117116B (en) * | 2022-01-28 | 2022-07-01 | 中国传媒大学 | Music unlocking method based on biological characteristic interaction and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112618911B (en) | 2023-02-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Leinonen et al. | Expression of emotional–motivational connotations with a one-word utterance | |
| Mesgarani et al. | Phoneme representation and classification in primary auditory cortex | |
| US10223934B2 (en) | Systems and methods for expressive language, developmental disorder, and emotion assessment, and contextual feedback | |
| CN106073706B (en) | A kind of customized information and audio data analysis method and system towards Mini-mental Status Examination | |
| Schuller et al. | The Munich Biovoice Corpus: Effects of Physical Exercising, Heart Rate, and Skin Conductance on Human Speech Production. | |
| CN107657964A (en) | Depression aided detection method and grader based on acoustic feature and sparse mathematics | |
| Kamińska et al. | Recognition of human emotion from a speech signal based on Plutchik's model | |
| CN112618911B (en) | Music feedback adjusting system based on signal processing | |
| Arias-Vergara et al. | Unobtrusive monitoring of speech impairments of Parkinson's disease patients through mobile devices | |
| Seneviratne et al. | Extended Study on the Use of Vocal Tract Variables to Quantify Neuromotor Coordination in Depression. | |
| CN113496696A (en) | Speech function automatic evaluation system and method based on voice recognition | |
| Sztahó et al. | Estimating the severity of parkinson's disease from speech using linear regression and database partitioning. | |
| Deng et al. | Disordered speech recognition using acoustic and sEMG signals. | |
| Egorow et al. | Employing Bottleneck and Convolutional Features for Speech-Based Physical Load Detection on Limited Data Amounts. | |
| Vaysse et al. | Automatic extraction of speech rhythm descriptors for speech intelligibility assessment in the context of Head and Neck Cancers | |
| Usman | On the performance degradation of speaker recognition system due to variation in speech characteristics caused by physiological changes | |
| CN108175426A (en) | A kind of lie detecting method that Boltzmann machine is limited based on depth recursion type condition | |
| Sharma et al. | Processing and analysis of human voice for assessment of Parkinson disease | |
| Tao et al. | Speech deception detection algorithm based on SVM and acoustic features | |
| Khan | Parkinson’s disease assessment using speech anomalies: a review | |
| Akkaralaertsest et al. | Classification of depressed speech samples with spectral energy ratios as depression indicator | |
| Julia et al. | Detection of emotional expressions in speech | |
| Sun | Classification of frequency following responses to English vowels in a biometric application | |
| Deshpande et al. | Automatic Breathing Pattern Analysis from Reading-Speech Signals | |
| CN112002342A (en) | Traditional Chinese medicine auscultation diagnosis automatic system supported by intelligent voice technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |