WO2014042599A1 - Système et procédé de développement d'un modèle révélant l'état émotionnel d'un sujet lorsqu'il écoute des morceaux de musique - Google Patents

Système et procédé de développement d'un modèle révélant l'état émotionnel d'un sujet lorsqu'il écoute des morceaux de musique Download PDF

Info

Publication number
WO2014042599A1
WO2014042599A1 PCT/SG2013/000404 SG2013000404W WO2014042599A1 WO 2014042599 A1 WO2014042599 A1 WO 2014042599A1 SG 2013000404 W SG2013000404 W SG 2013000404W WO 2014042599 A1 WO2014042599 A1 WO 2014042599A1
Authority
WO
WIPO (PCT)
Prior art keywords
subject
musical
features
pieces
state
Prior art date
Application number
PCT/SG2013/000404
Other languages
English (en)
Inventor
Cuntai Guan
Juanhong Yu
Yaozhang PAN
Original Assignee
Agency For Science, Technology And Research
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Agency For Science, Technology And Research filed Critical Agency For Science, Technology And Research
Priority to US14/428,852 priority Critical patent/US9557957B2/en
Priority to SG11201502063RA priority patent/SG11201502063RA/en
Publication of WO2014042599A1 publication Critical patent/WO2014042599A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/372—Analysis of electroencephalograms
    • A61B5/374—Detecting the frequency distribution of signals, e.g. detecting delta, theta, alpha, beta or gamma waves
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/377—Electroencephalography [EEG] using evoked responses
    • A61B5/38—Acoustic or auditory stimuli
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
    • A61M2230/00—Measuring parameters of the user
    • A61M2230/08—Other bio-electrical signals
    • A61M2230/10—Electroencephalographic signals

Definitions

  • the invention relates generally to brain computer interface, and more particularly to musical scoring in response to a subject's measured emotional state.
  • a method for deriving optimal discriminating features indicative of a subject state when the subject listens to one of a set of musical pieces comprising: extracting frequency features from the subject's EEG signal when the subject is in a first subject state and a second subject state, the frequency features being extracted from more than one frequency band in one set of time segments; and identifying optimal discriminating features from the extracted frequency features, the optimal discriminating features indicative of characteristics of the EEG signal when the subject is in the first subject state and the second subject state, wherein one of the first subject state and the second subject state indicates that the subject likes a musical piece while the other state indicates that the subject does not like the musical piece.
  • a method for developing a model indicative of a subject state when the subject listens to one of a set of musical pieces comprising extracting frequency features from the subject's EEG signal when the subject is in a first subject state and a second subject state, the frequency features being extracted from more than one frequency band in one set of time frames; identifying optimal discriminating features from the extracted frequency features, the optimal discriminating features indicative of similar characteristics of the EEG signal when subject is in the first subject state and the second subject state; extracting musical features associated with each of a set of the musical pieces; comparing the extracted musical features with the optimal discriminating features to determine which of the extracted musical features are indicative of characteristics of musical pieces that the subject prefers; and developing a model in response to the optimal discriminating features of the subject when listening to one of the set of the musical pieces and the musical features for the one of the set of the musical pieces, wherein one of the first subject state and the second subject state indicates that the subject likes a musical piece while the other state indicates that the
  • a system for developing a model indicative of a subject state when the subject listens to one of a set of musical pieces comprising: an input device for the system operable to receive electroencephalography (EEG) signal; an EEG discriminative feature generator operable to extract frequency features from the received EEG signal when the subject is in a first subject state and a second subject state and identify optimal discriminating features from the extracted frequency features, the optimal discriminating features indicative of similar characteristics of the EEG signal when subject is in the first subject state and the second subject state; a model builder operable to develop a model in response to the optimal discriminating features of the subject when listening to one of the set of the musical pieces and the musical features for the one of the set of the musical pieces; a music scorer operable to generate a musical preference score for each of the set of the musical pieces, the musical preference score indicating the subject's preference of the musical piece; and a music controller operable to control the set of the musical pieces, wherein the musical pieces is controlled
  • Figure 1 shows a front planar view of an apparatus in accordance with an embodiment when used by a subject
  • Figure 2 shows a flowchart that illustrates a method for deriving optimal discriminating features in accordance with the embodiment
  • FIGS 3 and 4 show flowcharts that illustrate methods for using the optimal discriminating features in accordance with the embodiment
  • Figure 5 shows an apparatus in accordance with the embodiment
  • FIG. 6 shows a diagram depicting how the EEG signal is decomposed in accordance with the embodiment
  • Figure 7 shows a graph depicting how the online music playback controller of Figure 5 may be configured to control the set of musical pieces in accordance with the embodiment
  • Figure 8 shows the musical features retrieved by the music feature generation of Figure 5 in accordance with the embodiment.
  • Figure 9 shows a computer system for performing a method for deriving optimal discriminating features in accordance with an embodiment.
  • a musical piece may be a song, a composition of vocal and/ or instrumental sounds or the like.
  • the musical piece may include a set of features such as pitch or rhythm and when the piece is played, the set of features make up a melody.
  • a musical piece is played to a subject 108 for his enjoyment.
  • the musical piece may be part of a set of musical pieces.
  • the set of musical pieces may be of different genres or subgenres.
  • the artistic nature of a musical piece may mean that the classification (e.g. into genres) of a musical piece may differ based on the person or entity who classifies.
  • the set of musical pieces may be played by the subject 108 through a music player 102.
  • the music player 102 may be a MPEG-1 Audio Layer-3 Player (or MP3 Player), CD player or cassette player or any other suitable devices as would be known to a person skilled in the art. Examples of suitable devices are a personal computer, computer tablets or mobile phones.
  • At least one electroencephalography (“EEG”) electrode 106 may be used to detect the EEG signal of the subject as he listens to the musical pieces.
  • the EEG electrodes are attached at a suitable location of the subject's body (e.g. forehead). Accordingly, the EEG signal will be analyzed to derive features of the EEG signal as the subject listens to at least one of the set of musical pieces.
  • the EEG electrode measures brainwaves that are indicative of the activities of the subject's brain as he listens to the musical piece. In clinical context, the EEG could also refer to the brain's spontaneous electrical activity over a short period of time.
  • the results from the analysis of the EEG signal and the musical pieces may be used to build a training model.
  • This training model may be used for automatically classifying the existing set of musical pieces or a new set of musical pieces.
  • This model may be used to organize the musical pieces in order of the subject's preference or states (e.g., emotional states).
  • the model may also be used to enhance musical parameters of an acoustic signal of the musical pieces in response of the subject's preference.
  • the software that is implemented is stored onto a device 110. It is to be understood that the analysis of the EEG signal and musical pieces may be done by the same or separate devices. In other words, both the analysis of the EEG signal and musical pieces may be carried out within device 110 or any one of the analysis of the EEG signal and musical pieces may be carried out within device 110. In any case, the EEG signal and the musical pieces will be subject to an analysis process during which the model is built. The same or separate device may use the model for other applications pertaining to the musical pieces.
  • FIG. 2 shows a flowchart 200 that illustrates a method for deriving optimal discriminating features indicative of a subject state when the subject listens to one of a set of musical pieces, in accordance with the embodiment.
  • This method aims to provide an effective way to analyze the EEG signal so as to derive optimal discriminating features.
  • a computer system may be used to implement the method shown in the flowchart 200.
  • the computer system may be present in the music player 102 of Figure 1.
  • the computer system may be present in the device 110.
  • frequency features are extracted from the subject's EEG signal when the subject is in a first subject state and a second subject state.
  • the frequency features includes characteristics of the EEG signal at one frequency band.
  • the step of extracting the frequency features includes extracting the frequency features from more than one frequency band in one set of time segments.
  • the EEG signal is decomposed into a plurality of filter bands and time segments.
  • the plurality of filter bands represents the frequency bands from which the frequency features are extracted.
  • optimal discriminating features are identified from the extracted frequency features.
  • the identified optimal discriminating features are those that capture the most discrimination frequency components between the first subject state and the second subject state.
  • one of the first subject state and the second subject state indicates that the subject likes the musical piece while the other state indicates that the subject does not like the musical piece.
  • the subject indicates via a separate device (e.g. device 110 in Figure 1 ) whether he likes the musical piece that he is listening to.
  • Figure 3 shows a flowchart 300 that illustrates a method for further steps included in identifying optimal discriminating features in accordance with the embodiment.
  • the step of identifying optimal discriminating features includes tabulating a matrix including the extracted frequency features in each corresponding frequency band.
  • the step of identifying optimal discriminating features includes deriving optimal spectral filters.
  • Each of the optimal spectral filters is the frequency band associated with the optimal discriminating features.
  • the optimal spectral filters may maximize the difference in the variance of the EEG signal in the first and second subject state. Additionally or alternatively, the optimal spectral filters may be obtained from the matrix obtained in step 202.
  • the step of identifying optimal discriminating features includes obtaining an optimal discriminating signal after passing each EEG signal through the optimal spectral filters.
  • Each EEG signal is obtained when the subject listens to a corresponding one of the set of musical pieces.
  • the optimal discriminating signal is used to calculate a form feature vector for each time segment.
  • the optimal discriminating signal that maximizes the difference in the variance of the EEG signal in the two subject states is associated with the largest eigenvalues associated with the matrix in 302.
  • the form feature vector is used to generate a musical preference score for each of the set of the musical pieces.
  • the musical preference score indicates the subject's preference of the musical piece.
  • the musical preference score may be generated as the subject listens to the musical , piece. Alternatively, the musical preference score may be generated after the subject has finished listening to the musical piece.
  • FIG. 4 shows a flowchart 400 that illustrates a method for developing a model indicative of a subject state when the subject listens to one of a set of musical pieces, in accordance with the embodiment.
  • This method aims to provide an effective model that can help detect music preference from an EEG signal and an effective model that finds an association between the EEG signal and the musical pieces.
  • Operations 402 and 404 may be analogous to operations 202 and 204 of Figure 2. Accordingly, frequency features are extracted from the subject's EEG signal when listening to one of the list of musical pieces in 402.
  • the step of extracting the frequency features includes extracting the frequency features from more than one frequency band in one set of time segments.
  • Optimal discriminating features are identified from the extracted frequency features in 404.
  • the act of extracting the frequency features extracts the frequency features from more than one frequency band in one set of time segments.
  • the act of identifying optimal discriminating features derives the features indicative of a subject state when the subject listens to one of the set of musical pieces.
  • a computer system may be used to implement operations 406, 408, 410, 412 and 414 and may be different from the computer system used to implement operations 402 and 404.
  • the computer system may be present in the music player.
  • the computer system may be present in a device separate from the music player.
  • musical features associated with each of the set of the musical pieces are extracted.
  • the extracted musical features are compared with the optimal discriminating features to determine which of the extracted musical features are indicative of characteristics of musical pieces that the subject prefers.
  • a set of musical features that could be retrieved is shown in Figure 8 and discussed below.
  • the extracted musical features are associated with the optimal discriminating features so as to determine which of the extracted musical features are the most indicative features of musical pieces that affect the subject's preference.
  • a model is built in response to 408.
  • the model is built using the parameters of the optimal discriminating features identified from the music-evoked EEG signal and of the extracted musical features that are indicative of characteristics of musical pieces that the subject prefers.
  • the model is used as a training model for the same subject.
  • the model is trained using at least one further subject and used for at least two subjects who may be the same or different subjects used for operations 412 and 414.
  • the model is trained using a first plurality of subjects and used for a second plurality of subjects, which may be the same or different from the first plurality of subjects.
  • the musical parameters of an acoustic signal of one or more of the set of the musical pieces are enhanced in response to the model.
  • the musical parameters of an acoustic signal are similar to the characteristics that the subject prefers. For example, if it were determined that the subject prefers musical pieces having bass or singing voices within the lowest range, such musical parameters (e.g., bass or singing voices within the lowest range) will be enhanced or made more prominent.
  • the order of the set of musical pieces is organized in response to the model. In accordance with the embodiment, a model organizes the order of the musical pieces based on the parameters used to build the model in operation 410.
  • musical pieces with musical parameters indicative of such musical features will be ranked higher.
  • the musical pieces that are ranked higher will be placed higher in a new order so as to have a greater positive effect on the subject.
  • FIG. 5 shows an apparatus 500, according to the embodiment.
  • Sensor data is acquired from a subject via an input device 502 (an example is the electrode 106 in Figure 1) as the subject listens to a musical piece.
  • An example of the input device 502 is a human-machine system or device, for example a subject 108 may listen to music pieces via the music player 102 which may also process the received EEG data and analyze the musical information.
  • the subject 108 attaches the at least one EEG electrode 106 which is the input device onto the forehead.
  • the at least one EEG electrode 106 may send the EEG data wirelessly to the music player 102.
  • the subject indicates his preference of the musical piece after listening to the musical piece by simply selecting "like” or “dislike” (e.g. using mouse to click when 102 is a computer, using touch screen when 102 is a tablet or smart phone).
  • the input device 502 is operatively coupled to an EEG discriminative feature generator 504.
  • the EEG discriminative feature generator 504 is configured to analyze the EEG signal of the subject in response to the selection of "like" or "dislike".
  • the EEG signal and the selection would be transmitted to the EEG discriminative feature generator 504 in order to find the most discriminative features of EEG for differentiating two different mental states “like” and “dislike", and the transmitted "selection" is viewed as the true label of the input data EEG for training the model.
  • the EEG discriminative feature generator 504 In the embodiment of online application phase 500(b), only the EEG signal would be transmitted to the EEG discriminative feature generator 504, which processes the EEG signal to observe the subject's brain activity to understand whether or not he enjoys or likes the musical piece that is playing.
  • the EEG signal received from the input device 502 will be decomposed into a plurality of filter banks and a plurality of time segments by the EEG discriminative feature generator 504.
  • the frequency banks may comprise 13 frequency bands and may cover the delta band to the gamma band (0.5Hz - 48 Hz) with a bandwidth of 4 Hz.
  • the EEG discriminative feature generator 504 may perform filter bank transformation so as to identify the discriminative features that capture the most discriminative features components.
  • Various embodiments employ the common frequency analysis (CFA) which uses the combination of optimal spectral filter to maximize difference in the variance of the EEG measurements in the first subject state and the second subject state.
  • CFA common frequency analysis
  • X a or ⁇ represents a single channel signal trial EEG for the first subject state and the second subject state.
  • the first subject state indicates that a subject likes or prefers the musical piece and the second subject state indicates that a subject does not like or prefer the musical piece.
  • the method employed by the CFA algorithm is based on the simultaneous diagonalization of two covariance matrices:
  • ⁇ is a diagonal matrix and the largest and smallest diagonal elements in ⁇ are corresponding to the optimized spectral filters which keep most discriminative information.
  • the discriminative spectral features are then derived by using variances of only a small number m of the spatial filtered signal Z by the CFA algorithm.
  • the signal Z p pe ⁇ 1..2m ⁇ that maximizes the difference in the variance of the two classes of EEG are associated with the largest eigenvalues ⁇ and ⁇ - ⁇ . These signals are used to form feature vector f p for each time segment.
  • the EEG discriminative feature vector f p is then used to compute the music preference score:
  • the EEG discriminative feature generator 504 is coupled to a music preference scorer (or music scorer) 506.
  • the music preference scorer 506 uses the generated EEG discriminative features to perform real-time detection of the level of music preference.
  • the music preference scorer 506 is configured to detect whether the subject likes the musical piece that is currently playing in response to the generated EEG discriminative features.
  • the music preference scorer 506 uses a classification algorithm to model and classify the EEG discriminative musical features in order to obtain the musical preference score based on the probability estimation.
  • SVM Support Vector Machine
  • the SVM maximizes the separation between the two classes by minimizing the cost function: subject to the constraint:
  • Equation (7) maps the decision function value to the interval [0, 1] , thus acan be viewed as a probability.
  • respective EEG signals of the subject in the first subject state and the second subject state may be collected.
  • the probability of these two states p ⁇ ) + p(z 2 ) 1.
  • the class probability of arbitrary x can be respectively denoted by vi z ⁇ ⁇ x ) for 'like' state and p(z 2 ⁇ x) for 'dislike' state.
  • the ratio which indicates EEG-based music preference level is:
  • the outputs of the music preference scorer 506 are the parameters that build the music preference model 580. In the embodiment, the outputs of music preference scorer 506 are tabulated for one set of musical pieces for one subject. In another embodiment, the outputs of music preference scorer 506 are tabulated for one set of musical pieces for at least two subjects.
  • the music player 514 could be coupled to a music feature representation 516.
  • the music player 514 is configured to play the musical pieces for training.
  • An example of the music player 514 is the music player 102 in Figure 1.
  • a controller (not shown) is coupled to the music player 514 to control the music pieces.
  • the music feature representation 5 6 could retrieve a set of musical features from an acoustical signal in order to make the further investigation of association between the musical features and music- evoked EEG signal. A set of musical features that could be retrieved is shown in Figure 8.
  • the music feature representation 516 is coupled to a music discriminative feature generator 517.
  • the music discriminative feature generator 517 is configured to perform a correlation analysis for the set of musical features retrieved by music feature representation 516. The analysis is performed to investigate the relationship between musical features and the music preference level in order to perform musical feature selection.
  • the musical feature selection is conducted to compare the extracted musical features with the optimal discriminating features so as to determine which of the extracted musical features are indicative of characteristics of musical pieces that the subject prefers.
  • the set of musical features could be used to predict the music preference level.
  • the correlation between each feature type and music preference level is computed to assess the validity of the musical features and select the best predictors.
  • the feature type with the highest correlation could be viewed as the best feature type for predicting music preference level.
  • stepwise regression between each musical feature and preference level may be computed for the same purpose. This helps to eliminate or reduce the complexity problem with unnecessary redundancy.
  • the music discriminative feature generator 517 and the EEG discriminative feature generator 504 may be coupled to a Discriminative Canonical Correlation Analyzer (DCCA) 518.
  • the DCCA 518 is configured to find out the association between music acoustic signal and music evoked EEG signal.
  • the generated EEG discriminative features from the EEG discriminative feature generator 504 and the musical features from the music feature representation 516 could be correlated to obtain a new feature that highlights the most important features reflecting music properties.
  • a discriminative CCA (DCCA) 518 that maximizes the correlation of the two groups and the discrimination of two classes at the same time is utilized to improve the music preference scoring performance.
  • the DCCA 5 8 may be a supervised CCA.
  • c w denotes the correlations of from the same classes
  • c passionate denotes the correlations of x i ,y i from the different classes.
  • EEG-Music associated music feature can be obtained from Equation (14).
  • the outputs of the DCCA 518 are the parameters that build the EEG-music association model 590.
  • the above description for Figure 5 relates to a calibration phase 500(a).
  • the calibration phase 500(a) could be carried out with one subject to train a subject specific model. Additionally or alternatively, the calibration phase 500(a) could be carried out with one further subject to train a general model. In another embodiment, the calibration phase 500(a) is carried out with a plurality of subjects.
  • the parameters that build the music preference model 580 are transmitted to a music preference detector 532 in an application phase.
  • a music player 524 (analogous to music player 5 4) plays a set of musical pieces to a subject via an input device 522.
  • An example of the music player 524 is the music player 102 in Figure 1.
  • the set of musical pieces that are played during the application phase may be the same or different from the set of musical pieces played during the calibration phase 500(a).
  • the subject in the application phase 500(b) may be the same or different from the subject in the calibration phase 500(a).
  • the input device 522 is analogous to the input device 502 in the calibration phase.
  • Sensor data is acquired from a subject in a system via the input device 522, as the subject listens to a musical piece.
  • An example of the system is shown in Figure 1 and includes the music player 102, the channels 104, the electrode 106 and the subject 108.
  • the input device 522 is operatively coupled to the music preference detector 532.
  • the music preference detector 532 receives the parameters from the music preference scorer 580 and uses these parameters to determine an indicator of the music preference level based on the evoked EEG signals. In the embodiment, it is determined that a subject likes a musical piece if p ⁇ z x ⁇ x) > p(z 2 ⁇ x).
  • the indicator is a music preference score and is determined using the parameters from the music preference scorer 580.
  • the music preference detector 532 is operatively coupled to an online music playback controller (or music controller) 534.
  • the online music playback controller 534 could organize the order of the set of musical pieces by the musical preference score that is associated with the musical piece. By way of example, a musical piece that is associated with a higher musical preference score than another musical piece would be placed higher in the order of the set of musical pieces.
  • the online music playback controller 534 is configured to receive the ratio tabulated in equation (8) and provide a music preference class information based on the criterion below : x ⁇ 'like' class
  • the EEG signal is tabulated at a regular interval (e.g., 25s).
  • the music preference class information is used to control the musical pieces that would be played in response to the music preference class information provided in online music playback controller 534.
  • the controller 534 is configured to control the musical piece such that it will continue to play.
  • the controller is configured to control the musical piece such that it will skip to the next musical piece.
  • the controller is configured not to take any action.
  • the playlist ranker 530 is configured to receive the music preference score and the music feature from the EEG-music associator 528 to rank the music database and rank the order of the set of the musical pieces.
  • the dissimilarity of two musical pieces is measured through a distance metric.
  • An example is Mahalanobis distance:
  • the current mode will be identified as 'like'. Further, the database of musical pieces will be ranked based on the similarity of each musical piece in the database to the musical piece that is playing. The set of musical pieces will be reorganized based on the descending order of the ranking.
  • the music playlist updating rule for the case is updated for the subject "dislikes" the current musical piece given the music preference score l(x) and a predefined minimum moving order N min and minimum similarity s min .
  • the current musical piece y and its neighborhood with music s(y y) > s min to order N min //(x) of the music playlist may be moved.
  • the online music playback controller 534 is operatively coupled to the music player 524.
  • the online music playback controller 534 controls the order on how the set of musical pieces would be played via a controller (not shown).
  • the music player 524 would play the set of musical pieces in the order determined by the online music playback controller 534.
  • the music player 524 is operatively coupled to a music feature representation 526 which is analogous to music feature representation 516.
  • the music feature representation 516 could retrieve a set of musical features from the acoustical signal in order to make the further investigation of association between the musical features and music-evoked EEG signal.
  • a set of musical features that could be retrieved is shown in Figure 8.
  • the music feature representation 526 is further configured to calculate the form feature vector.
  • the music preference detector 532 is operatively coupled to a playlist ranker 530.
  • the playlist ranker 530 could also rank the set of musical pieces that the subject is listening to by the musical preference score that is associated with the musical piece.
  • the music feature representation 526 is operatively coupled to an EEG-music associator (or model builder) 528.
  • the parameters for a EEG-music association model 590 are sent to the EEG-music assoicator 528.
  • the trained EEG-music association model 590 is then used in the EEG-music associator 528 to give high weight to the musical features that are highly correlated to the subject's preference and give low weight to or even discard the musical features that are weakly correlated to the subject's preference.
  • the EEG-music associator 528 is operatively coupled to the playlist ranker 530.
  • the results of the weighted music features will be used by the playlist ranker 530 to rank the set of musical pieces that the subject is listening to.
  • FIG. 8 shows how the EEG signal is decomposed into filter banks by the EEG discriminative feature generator 504 in accordance with the embodiment.
  • FIG. 7 shows how the online music playback controller 53 is configured to control the set of musical pieces.
  • Lines 702 and 704 represent the musical preference class information to be 1 + ⁇ and 1 , respectively. In other words, lines 702 and 704 represent "like" and "neutral" state for the music preference class information. When it is determined that the music preference state information is like and neutral, the musical piece that is playing will continue to play.
  • Line 706 represents the musical preference class information to be 1 - ⁇ . This means that line 706 represents that the subject dislikes the musical piece that is currently playing. When it is determined that the musical preference state information is "dislike", the musical piece that is currently playing will stop and the music player will skip to the next musical piece.
  • the playlist ranker 530 may be coupled to the online music playback controller 534.
  • Figure 8 shows the features 900 that are held for consideration and the steps to derive these features.
  • the features are extracted by music feature representation 516 or 526.
  • the retrieved musical features include but are not limited to: 1. Root-mean-square (RMS): root average of the square of the amplitude
  • Brightness estimated by measuring the amount of energy above a cut-off frequency.
  • Roughness estimated depending on the frequency ratio of each pair of sinusoids. An estimation of the total roughness is computed by computing the peaks of the spectrum, and taking the average of all the dissonance between all possible pairs of peaks
  • MFCC Mel-frequency cepstral coefficients
  • Pitch is estimated by computing an autocorrelation and then a peak picking is applied to the autocorrelation function.
  • Inharmonicity estimates the inharmocicity, i.e., the amount of partials that are not multiples of the fundamental frequency, as a value between 0 and 1. More precisely, the inharmonicity considered here takes into account the amount of energy outside the ideal harmonic series.
  • Chromagram also called harmonic pitch class profile, which shows the distribution of energy along the pitches or pitch classes.
  • Mode estimates the modality, i.e. major vs. minor
  • H .Tonalcentroid a 6-dimensional tonal centroid vector corresponds to a projection of the chords along circles of fifths, of minor thirds, and of major thirds.
  • Entropy it is the relative Shannon entropy of the music segment.
  • Figure 9 shows an exemplary computer system 800 for use in performing the abovementioned operations, for example, as part of a music feature generation.
  • the computer system performs operations 202 to 204 of Figure 2, operations 302 to 310 of Figure 3 and operations 402 to 414 of Figure 4.
  • the computer system 800 is part of the music player 102 which may be a smartphone or a tablet.
  • the computer system 800 includes a computer module 802, input modules such as a keyboard 804 and mouse 806 and a plurality of output devices such as a display 808, printer 810 and a device 811..
  • the computer module 802 is connected to a computer network 812 via a suitable transceiver device 814, to enable access to, for example, the Internet or other network systems, such as, a Local Area Network (LAN) or a Wide Area Network (WAN).
  • LAN Local Area Network
  • WAN Wide Area Network
  • the computer module 802 in the example includes a processor 818, a Random Access Memory (RAM) 820 and a Read Only Memory (ROM) 822.
  • the computer module 802 also includes a number of Input/ Output (I/O) interfaces, for example I/O interface 824 to the display 808, interface 826 to the keyboard 804 and I/O interface 827 to the device 811.
  • I/O Input/ Output
  • the components of the computer module 802 typically communicate via an interconnected bus 828 and in a manner known to the person skilled in the relevant art.
  • the computer system 800 may function in accordance with software stored on the RAM 820 and/or the ROM 822.
  • the software may include an operating system and one or more application programs.
  • An application program may be supplied to the user of the computer system 800 encoded on a data storage medium, such as, a CD-ROM or flash memory carrier and read utilizing a corresponding data storage medium drive of a data storage device 830.
  • the application program may be read and controlled in its execution by the processor 818. Intermediate storage of program data may be accomplished using RAM 820.
  • the computer system 800 provides a non- limiting example of a suitable computer system.
  • one or more elements may be combined together into the same element.
  • one or more elements may be absent or duplicated. Additional elements may be present in some embodiments.
  • EEG signals were acquired using a NeuroSky-MindBand with a sampling rate of 256 Hz.
  • two EEG electrodes were attached horizontally onto the subjects' foreheads.
  • the left and right electrodes corresponded to FP1 and FP2 (106, FIG. 1 ), respectively.
  • the subjects would also indicate whether they like, dislike or feel neutral about each musical piece to obtain the true labels for the experiment.
  • a total of 120 EEG signals were obtained for each subject/Consequently, a " 5 times 5-fold" cross-validation was performed on each subject's data so as to evaluate the performance.
  • Table 1 shows the classification results of several combinations of feature extraction methods and classifiers that are conventionally known. The same frequency band was used for the EEG signal at 0.5-48Hz. The test accuracies were obtained by "5 times 5-fold" cross validation for each subject. The results shown in Table 1 are the average values and standard deviations across the twenty-one subjects. It can be seen that embodiments of this invention obtained the highest accuracy as compared to other methods.
  • the conventional methods used are principal component analysis (PCA), spectral filter, differential asymmetry of twelve electrode pairs (DASM), logistic regression and binary Fisher discriminant analysis (BFDA).
  • Table 1 Average accuracy of different feature extraction methods and classifiers
  • a subject specific model is a training model used on one subject which is built using the data of the same subject.
  • the music discriminative feature generation was evaluated by comparing the classification results of music preferences using a full feature set to the results of those using selected features by mutual information or based on correlations as described above.
  • the EEG and music signal association were evaluated by comparing the music preference classification results of the three methods (full feature, mutual information, correlation) purely based on music information to the other two methods (CCA, DCCA) associating music with EEG information and extracting the music information based on the music-EEG association.
  • CCA full music Mutual correlation
  • Table 3 Classification Accuracy of Music Preference using Different Music Feature Generation Method
  • PCA principal components analysis
  • Spectral filter utilises Fourier analysis, a commonly used transformation, to look at specific frequency bands of signals recorded from each electrode.
  • the frequency range for the Fourier transform is 0.5-48 Hz.
  • Short-time Fourier transform (STFT) with Hanning window of time interval and with a 50% overlap is used to extract the power spectral density value over time.
  • Spectral powers of EEG components are typically adopted where EEG components are defined according to frequency range, including delta (0.5-3 Hz), theta (4-7 Hz), alpha (8-13Hz), beta (14-30 Hz) and gamma (31-48 Hz). Therefore, after applying STFT, a power value of each EEG components across time over each channel is derived.
  • DASM is the abbreviation for differential asymmetry of 12 electrode pairs and is conventionally known. It is possible to derive the DASM12 after obtaining the spectral powers, which is a power difference at each symmetric electrode pair. For the purposes of the experiment, the power difference between the two electrodes was obtained.
  • Logistic regression is a type of regression analysis used for predicting the outcome of dependent variables which may take on one of a limited and usually fixed number of possible values. This means that logistic regression is typically used in estimating empirical values of the parameters in a quantitative response model (involving one or more discrete random variables whose conditional probability distribution given the values of variables is specified up to a finite number of unknown parameters). Logisitc regression is also typically known to be used when there are only two categories of the dependent variables because it measures the relationship between a dependent variable and one or more independent variables, which are usually (but not necessarily) continuous by using probability scores as the predicted values of the dependent variable.
  • FDA aims to find a linear projection of the data such that the ratio of between group variability to within group variability is maximized in order to classify different classes.
  • BFDA is a specific form of FDA that is used to classify two classes.
  • the CFA is used to identify the optimal discriminating features. It is a further advantage that an association model associating the EEG signal and the musical features of the musical pieces is generated. This is advantageous because, as shown above, the CFA can identify the optimal discriminating features of an EEG signal and thereby help to provide a musical preference score, and the association model can enhance musical features of an acoustic signal, which can be used to generate a new order of a list of musical pieces in descending order of a subject's preference level.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Psychiatry (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Veterinary Medicine (AREA)
  • Biophysics (AREA)
  • Animal Behavior & Ethology (AREA)
  • Surgery (AREA)
  • Public Health (AREA)
  • Molecular Biology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Psychology (AREA)
  • Biomedical Technology (AREA)
  • Pathology (AREA)
  • Theoretical Computer Science (AREA)
  • Developmental Disabilities (AREA)
  • Acoustics & Sound (AREA)
  • Social Psychology (AREA)
  • Hospice & Palliative Care (AREA)
  • General Physics & Mathematics (AREA)
  • Educational Technology (AREA)
  • Child & Adolescent Psychology (AREA)
  • General Engineering & Computer Science (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Auxiliary Devices For Music (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Automation & Control Theory (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)

Abstract

La présente invention concerne un procédé permettant de dériver des caractéristiques discriminantes optimales révélatrices de l'état d'un sujet lorsque le sujet écoute un morceau parmi un ensemble de morceaux de musique, comprenant l'étape consistant à extraire les caractéristiques de fréquence d'un signal d'EEG d'un sujet lorsque le sujet se trouve dans un premier état et dans un second état, les caractéristiques de fréquence étant extraites à partir de plus d'une bande de fréquences dans un ensemble de segments temporels ; et identifier les caractéristiques discriminantes optimales issues des caractéristiques de fréquence extraites, les caractéristiques discriminantes optimales révélatrices des caractéristiques du signal d'EEG lorsque le sujet se trouve dans le premier état et dans le second état, l'un des premier et second états du patient indiquant que le sujet aime un morceau de musique et l'autre état indiquant que le sujet n'aime pas le morceau musical.
PCT/SG2013/000404 2012-09-17 2013-09-17 Système et procédé de développement d'un modèle révélant l'état émotionnel d'un sujet lorsqu'il écoute des morceaux de musique WO2014042599A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US14/428,852 US9557957B2 (en) 2012-09-17 2013-09-17 System and method for developing a model indicative of a subject's emotional state when listening to musical pieces
SG11201502063RA SG11201502063RA (en) 2012-09-17 2013-09-17 System and method for developing a model indicative of a subject's emotional state when listening to musical pieces

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SG201206897-9 2012-09-17
SG2012068979 2012-09-17

Publications (1)

Publication Number Publication Date
WO2014042599A1 true WO2014042599A1 (fr) 2014-03-20

Family

ID=50278546

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SG2013/000404 WO2014042599A1 (fr) 2012-09-17 2013-09-17 Système et procédé de développement d'un modèle révélant l'état émotionnel d'un sujet lorsqu'il écoute des morceaux de musique

Country Status (3)

Country Link
US (1) US9557957B2 (fr)
SG (1) SG11201502063RA (fr)
WO (1) WO2014042599A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9557957B2 (en) 2012-09-17 2017-01-31 Agency For Science, Technology And Research System and method for developing a model indicative of a subject's emotional state when listening to musical pieces
CN107106063A (zh) * 2014-11-02 2017-08-29 恩戈格勒公司 智能音频头戴式耳机系统
CN110947076A (zh) * 2019-11-27 2020-04-03 华南理工大学 一种可进行精神状态调节的智能脑波音乐可穿戴设备

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10321842B2 (en) * 2014-04-22 2019-06-18 Interaxon Inc. System and method for associating music with brain-state data
US12029573B2 (en) 2014-04-22 2024-07-09 Interaxon Inc. System and method for associating music with brain-state data
EP3508972A1 (fr) * 2018-01-04 2019-07-10 Harman International Industries, Incorporated Système de traitement audio personnalisé biométrique
US11786694B2 (en) 2019-05-24 2023-10-17 NeuroLight, Inc. Device, method, and app for facilitating sleep
CN112674768A (zh) * 2019-10-18 2021-04-20 中国人民解放军战略支援部队航天工程大学 一种基于智能汗带的情绪分析系统
CN112799505B (zh) * 2020-12-21 2022-08-26 杭州电子科技大学 一种基于脑电信号的音乐音调想象区分方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4883067A (en) * 1987-05-15 1989-11-28 Neurosonics, Inc. Method and apparatus for translating the EEG into music to induce and control various psychological and physiological states and to control a musical instrument
US5667470A (en) * 1994-02-08 1997-09-16 Jiri Janata Apparatus and method for using listener feedback to determine the nature of sound stimuli presented to the listener
WO2005025416A2 (fr) * 2003-09-11 2005-03-24 Regents Of The University Of Minnesota Localisation de sources neuronales dans un cerveau
US8636640B2 (en) * 2008-04-11 2014-01-28 Brain Symphony LLC Method and system for brain entertainment
WO2014042599A1 (fr) 2012-09-17 2014-03-20 Agency For Science, Technology And Research Système et procédé de développement d'un modèle révélant l'état émotionnel d'un sujet lorsqu'il écoute des morceaux de musique

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
SCHMIDT ET AL.: "Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions", COGNITION AND EMOTION, vol. 15, no. 4, 2001, pages 487 - 500 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9557957B2 (en) 2012-09-17 2017-01-31 Agency For Science, Technology And Research System and method for developing a model indicative of a subject's emotional state when listening to musical pieces
CN107106063A (zh) * 2014-11-02 2017-08-29 恩戈格勒公司 智能音频头戴式耳机系统
CN110947076A (zh) * 2019-11-27 2020-04-03 华南理工大学 一种可进行精神状态调节的智能脑波音乐可穿戴设备
CN110947076B (zh) * 2019-11-27 2021-07-16 华南理工大学 一种可进行精神状态调节的智能脑波音乐可穿戴设备

Also Published As

Publication number Publication date
US9557957B2 (en) 2017-01-31
SG11201502063RA (en) 2015-10-29
US20150235134A1 (en) 2015-08-20

Similar Documents

Publication Publication Date Title
US9557957B2 (en) System and method for developing a model indicative of a subject's emotional state when listening to musical pieces
Karan et al. Parkinson disease prediction using intrinsic mode function based features from speech signal
Sakar et al. A comparative analysis of speech signal processing algorithms for Parkinson’s disease classification and the use of the tunable Q-factor wavelet transform
Yogesh et al. A new hybrid PSO assisted biogeography-based optimization for emotion and stress recognition from speech signal
Huang et al. Music genre classification based on local feature selection using a self-adaptive harmony search algorithm
Nalini et al. Music emotion recognition: The combined evidence of MFCC and residual phase
US20120233164A1 (en) Music classification system and method
GB2533654A (en) Analysing audio data
Temko et al. Fuzzy integral based information fusion for classification of highly confusable non-speech sounds
Chang et al. A music recommendation system with consideration of personal emotion
Zhou et al. Gammatone spectral latitude features extraction for pathological voice detection and classification
Panteli et al. A model for rhythm and timbre similarity in electronic dance music
WO2016102738A1 (fr) Détermination de similitude et sélection de musique
AnilKumar et al. Classification of voice pathology using different features and bi-lstm
US20180173400A1 (en) Media Content Selection
Rao Audio signal processing
Wu Musical tempo octave error reducing based on the statistics of tempogram
Ribeiro et al. A whole brain eeg analysis of musicianship
Ansari et al. SpectNet: End-to-end audio signal classification using learnable spectrograms
Er et al. Music emotion recognition with machine learning based on audio features
Gidaye et al. Unified wavelet-based framework for evaluation of voice impairment
Shabber et al. AFM signal model for dysarthric speech classification using speech biomarkers
Kumar et al. Analysis and classification of electroglottography signals for the detection of speech disorders
de Souza et al. Music Tempo Estimation via Neural Networks--A Comparative Analysis
Carvajal-Castaño et al. Articulation analysis in the speech of children with cleft lip and palate

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13836684

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 14428852

Country of ref document: US

122 Ep: pct application non-entry in european phase

Ref document number: 13836684

Country of ref document: EP

Kind code of ref document: A1