KR102381925B1 - Robust speaker adaptation modeling method for personalized emotion recognition and apparatus thereof - Google Patents

Abstract

Disclosed are a robust speaker adaptive modeling method for personalized emotion recognition and an apparatus therefor.
In this method, when the emotional voice data (hereinafter referred to as “target user data”) initially obtained from the target user is less than a preset threshold number, and the emotional data of the target user data is not absent data, in advance Data similar to the emotion-specific data of the target user's data is selected and strengthened from the unlabeled data of the labeled data of the formed initial model (hereinafter referred to as "initial model") for a plurality of users. Or in this method, when the target user's data is less than the preset threshold number and the emotional data of the target user's data is absent data, the user most similar to the target user's emotional data among the data of the initial model Data corresponding to the emotion corresponding to the absence data of the target user is selected and strengthened from among the data. Or, in this method, when the target user's data is equal to or greater than the preset threshold number and the number of data for each emotion of the target user is in an imbalanced state, a virtual data augmentation method using an oversampling algorithm for each emotion of the target user Enrich your data.

Description

ROBUST SPEAKER ADAPTATION MODELING METHOD FOR PERSONALIZED EMOTION RECOGNITION AND APPARATUS THEREOF

The present invention relates to a robust speaker adaptive modeling method and apparatus for personalized emotion recognition.

Traditional emotion recognition technology collects emotional voice data from multiple users, learns it through a machine learning algorithm, and applies the model generated by learning to all users equally.

However, this technique has a disadvantage in that it is difficult to represent an even accuracy to all users and the accuracy deviation is large. Accordingly, in recent years, a personalized emotion recognition technology for generating a personalized model for each user and applying it to the user has emerged.

Existing personalized emotion recognition technology reflects the labeled emotion speech data collected from the target user, centering on the emotion recognition model built from multiple users, and models it through linear parameter change and label refinement work. works by changing the

However, the existing personalized emotion recognition technology requires sufficient data to change to a highly accurate personalized model, and the best performance can be achieved only when the user's emotional voice data is uniformly input. However, in the initial stage of personalized emotion recognition, it is difficult to sufficiently secure the labeled data of the target user, and it is easy to fall into an imbalanced environment, so that it is easy to fall into a cold-start problem in which it is difficult to create a personalized model.

Therefore, a strong personalized emotion recognition model generation technology is required to solve the cold start problem in the existing personalized emotion recognition technology.

The problem to be solved by the present invention is to derive a personalized training dataset that is sufficient and balanced for creating a personalized training model even in small data, absent data, and imbalanced data environments where target user data is not sufficiently collected in the personalized voice-based emotion recognition process It is to provide a robust speaker adaptive modeling method capable of performing and an apparatus for the same.

The characteristic configuration of the present invention for achieving the object of the present invention as described above and for realizing the characteristic effects of the present invention to be described later is as follows.

According to one aspect of the present invention, a speaker adaptive modeling method is provided, the method comprising:

As a robust speaker adaptation modeling method for personalized emotion recognition, emotional voice data (hereinafter referred to as “target user data”) initially obtained from a target user is less than a preset threshold number, and the emotion of the target user data When the data is not absent data, among the data obtained by unlabeling the labeled data of an initial model (hereinafter referred to as an “initial model”) for a plurality of preformed users, emotion-specific data of the data of the target user and Selecting and strengthening similar data, or when the target user's data is less than the preset threshold number and the emotion data of the target user's data is absent data, the target user's emotion among the data of the initial model Selecting and strengthening data corresponding to the emotion corresponding to the absent data of the target user from among the data of the user most similar to the data, or the target user's data is equal to or greater than the preset threshold number, each emotion of the target user and strengthening data for each emotion of the target user by a virtual data augmentation method using an oversampling algorithm when the number of data is in an imbalanced state.

According to another aspect of the present invention, there is provided a speaker adaptive modeling apparatus, the apparatus comprising:

As a robust speaker adaptive modeling device for personalized emotion recognition, it is initially obtained from a target user among unlabeled data of labeled data of an initial model (hereinafter referred to as an “initial model”) for a plurality of preformed users. A first modeling unit that selects and strengthens data similar to emotion-specific data of the acquired emotion voice data (hereinafter referred to as “target user data”), the data of the initial model, the user most similar to the emotion data of the target user A second modeling unit that selects and strengthens data corresponding to emotions corresponding to the absent data of the target user from among the data, and a third modeling unit that strengthens data for each emotion of the target user in a virtual data augmentation method using an oversampling algorithm And, when the target user's data is less than a preset threshold number, and the emotional data of the target user's data is not absent data, data reinforcement is performed through the first modeling unit, or the target user's data is While less than the preset threshold number, if the emotional data of the target user's data is absent data, data reinforcement is performed through the second modeling unit, or the target user's data is greater than or equal to the preset threshold number, and a processing controller configured to enhance data through the third modeling unit when the number of data for each emotion of the target user is in an imbalanced state.

According to the present invention, it is possible to derive a personalized training dataset sufficient and balanced for creating a personalized training model even in a small data, absent data, and imbalanced data environment in which the target user's data is not sufficiently collected in the personalized voice-based emotion recognition process. .

In addition, as more accurate emotion recognition is possible through personalized modeling technology, it can be applied to various IoT intelligent service fields.

In addition, it is possible to provide a personalized emotion recognition service by accumulating and updating the emotional voices of users applied to robots and call centers.

In addition, it can be used as a measurement technique in the market necessary for psychological treatment such as bipolar disorder/depression using this.

1 is a schematic flowchart of a speaker adaptive modeling method for a small amount of emotional voice data of a target user according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating the flow according to FIG. 1 using a distribution diagram of voice data.
FIG. 3 is a diagram illustrating an example of unlabeled data for labeled emotion-specific data of an IM in the speaker adaptive modeling method shown in FIG. 1 .
FIG. 4 shows the average value of the feature vector values for each emotion of Anger, Sadness, and Happiness obtained for a target user according to the speaker adaptive modeling method shown in FIG. 1 is calculated by MLE calculation. It is a drawing showing an example of the
FIG. 5 is a diagram illustrating an example of selecting data by setting a threshold value according to the speaker adaptive modeling method shown in FIG. 1 .
6 is a diagram illustrating an example of target user data (right) enhanced according to the speaker adaptive modeling method shown in FIG. 1 and target user data secured according to the existing method.
7 is a schematic flowchart of a speaker adaptive modeling method for emotional voice data of a target user's absence according to an embodiment of the present invention.
8 is a diagram illustrating the flow according to FIG. 7 using a distribution diagram of voice data.
FIG. 9 is a diagram illustrating an example of calculating a data distribution element for each of target user data and IM data according to the speaker adaptive modeling method shown in FIG. 7 .
10 is a diagram illustrating an example of calculating a similarity based on data distribution factors calculated for each of target user data and IM data according to the speaker adaptive modeling method shown in FIG. 7 .
11 is a schematic flowchart of a speaker adaptation modeling method for unbalanced emotional voice data of a target user according to an embodiment of the present invention.
12 is a schematic flowchart of a robust speaker adaptation modeling method for personalized emotion recognition according to an embodiment of the present invention.
13 is a schematic block diagram of a speaker adaptive modeling apparatus for personalized emotion recognition according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as “…unit”, “…group”, and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. there is.

The devices described in the present invention are composed of hardware including at least one processor, a memory device, a communication device, and the like, and a program to be executed in combination with the hardware is stored in a designated place. The hardware has the configuration and capability to implement the method of the present invention. The program includes instructions for implementing the method of operation of the present invention described with reference to the drawings, and is combined with hardware such as a processor and a memory device to execute the present invention.

Hereinafter, a robust speaker adaptation modeling method for personalized emotion recognition will be described in an embodiment of the present invention.

In an embodiment of the present invention, three methods are provided as robust speaker adaptation modeling for personalized emotion recognition. Specifically, the first method is a method for a case where the emotional voice data of the target user is small, the second method is a method for a case where some emotional voice data of the target user is absent, and the third method is a method for a case where some emotional voice data of the target user is absent. This is a method for a case in which star voice data is in an unbalanced state.

First, the first method will be described. That is, it relates to a robust speaker adaptation modeling method performed when the emotional voice data collected for the target user is small. This method selects a voice similar to the target user from the initial model of a large number of users previously built in a small data environment, and then refines the label to obtain a training dataset, that is, the real-case centered on the target user. A method of selecting data.

1 is a schematic flowchart of a speaker adaptive modeling method for a small amount of emotional voice data of a target user according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating the flow according to FIG. 1 using a distribution diagram of voice data. Before the description, in the embodiment of the present invention, an initial model (IM) 11 for a plurality of users is already built and stored in the IM database 10, and emotional voice data initially acquired for the target user It is assumed that 21 is obtained in advance and stored in the target user DB 20 .

1 and 2 , first, the labeled data of the IM 11 is UN-labeled (S100). That is, the emotional information of other users corresponding to the IM 11 is removed. The reason is that emotions expressed through voice may be expressed differently for each user. This is because, for example, a voice of a specific user's neutral emotion may be similar to a voice of an angry emotion in emotional voice information of another user. Therefore, it is desirable to remove the label from the IM 11 and utilize it when reinforcing the training data set of the target user with a voice similar to that of other users. Referring to FIG. 3 , an example of unlabeled data for the labeled emotion-specific data of the IM 11 on the left is shown on the right.

Next, a feature vector value (TfeatureVector) of data for each emotion is calculated using the acquired target user data 21 stored in the target user database 20 ( S110 ). Here, since the method of calculating the feature vector value of the emotion-specific data is already well known, it will not be described in detail here.

Thereafter, the average value of the feature vector values for each emotion of the target user is calculated ( S120 ). The average value of these feature vector values may be calculated using a Maximum Liklihood Estimation (MLE) technique. For example, the MLE (TMLE) of the feature vector value for each emotion of the target user may be calculated as in [Equation 1].

[Equation 1]

Here, e is an index of a corresponding emotion, I is an index of a feature vector, j is an index of data, N is the number of data, and TfeatureVector is a feature vector value of data for each emotion of the target user.

Referring to FIG. 4 , it can be seen that the average value of the feature vector values for each of the emotions of Anger, Sadness, and Happiness obtained for the target user is calculated by MLE calculation.

Next, the distance between the average values of the feature vector values for each emotion of the target user is calculated, and half of the maximum distance among the calculated distances is set as a threshold value (S130). Here, the distance between the average values of the feature vector values for each emotion may be calculated as the Euclidean distance as shown in Equation 2 below.

[Equation 2]

Here, TMLE is the average value of the feature vector values for each emotion of the target user calculated in [Equation 1], e _i and e _i are different emotion indexes, k is the index of the feature vector, and FN is the number of features .

In this way, when the distance between the average values of the feature vector values for each emotion of the target user is calculated, the distance of half of the maximum value, that is, 1/2 of the distance calculated using the following [Equation 3], is set as the threshold value (Maximum). Threshold Distance, MTD).

[Equation 3]

Then, based on the average value of the feature vector values for each emotion of the target user calculated in the step S120, data of the IM 11 within the threshold value range calculated in the step S130 is selected (S140). .

Here, in order to select the data of the IM 11 within the range of the threshold, the distance between the average value of the feature vector values for each emotion and the data of the IM 11 is the Euclidean distance as in the following [Equation 4] can be calculated.

[Equation 4]

Here, TMLE is the average value of the feature vector values for each emotion of the target user calculated in [Equation 1], e _i is the emotion index, k is the index of the feature vector, and IDS _m is the unlabeled IM (11). Data (Initial Data Set, IDS), where FN is the number of features.

)가 상기 단계(S130)에서 산출된 임계값(

) 내에 속하는 경우 해당 IM(11)의 데이터가 타겟 사용자의 음성 데이터로서 선택되게 된다. 구체적으로, 에 의해 비교한 후, [수학식 4]에 의해 산출된 거리(

)가 임계값(

)보다 큰 경우에는 임계값 범위를 벗어나므로 해당되는 IM(11)의 데이터는 폐기되고, [수학식 4]에 의해 산출된 거리(

)가 임계값(

) 이하인 경우에는 임계값 범위 내에 있는 것이므로 해당되는 IM(11)의 데이터가 선택된다.Therefore, the distance calculated by [Equation 4] (

) is the threshold value calculated in step (S130) (

), the data of the corresponding IM 11 is selected as voice data of the target user. Specifically, after comparing by , the distance calculated by [Equation 4] (

) is the threshold (

), since it is out of the threshold range, the data of the corresponding IM 11 is discarded, and the distance calculated by [Equation 4] (

) is the threshold (

) or less, since it is within the threshold range, the data of the corresponding IM 11 is selected.

Thereafter, the average value of the feature vector values for each emotion of the target user closest to the selected data of the IM 11 is found and the corresponding emotion is checked ( S150 ). This step may be performed simultaneously with the aforementioned step (S140). For example, as shown in FIG. 5 , the distance between the data of the IM 11 and the average value of the feature vector value for each emotion of the target user is calculated by [Equation 4], and When selecting the IM 11 data by comparison, it can be simultaneously performed by confirming the emotion when the difference between the calculated distance and the threshold value is the minimum.

Finally, label refinement is performed on the IM 11 data based on the emotion confirmed in the step (S150) (S160).

As described above, according to an embodiment of the present invention, when the initial emotional voice data obtained for the target user is small, similar data is selected for each emotion of the target user from among the unlabeled data of the IM 11 , which is an initial model of a plurality of users. can be strengthened.

Referring to FIG. 6 , examples of target user data (left) secured according to the existing method for target user data 21 and target user data enhanced according to the present method (right) are illustrated. As can be seen from FIG. 6 , the amount of data secured according to this method is larger than that of the existing method, and as a result, it can be seen that speaker adaptive modeling for personalized emotion recognition is strengthened.

Next, the second method will be described. That is, it relates to a robust speaker adaptation modeling method performed when data on a specific emotion among emotional voice data collected for a target user is absent. In general, people do not express emotions at the same rate in everyday life. If the target user does not express a certain emotion for a long time, a training model must be created without a sample of that emotion expression. In this case, the specific emotion is not recognized by the system and the accuracy is 0%. Therefore, in this method, it is assumed that the voice of a user having a voice pattern similar to that of the target user among the voices of various users in the IM 11 is similar to the absence emotion data, and is replaced with the emotional voice data of the corresponding user and reinforced. .

7 is a schematic flowchart of a speaker adaptive modeling method for voice data of a target user's absence emotion according to an embodiment of the present invention, and FIG. 8 is a diagram illustrating the flow according to FIG. 7 using a distribution diagram of voice data. Similarly, before the description, an initial model (IM) 11 for a plurality of users has already been built and stored in the IM database 10 , and emotional voice data 21 initially acquired for the target user is It is assumed that it is obtained in advance and stored in the target user DB 20 .

Referring to FIGS. 7 and 8 , first, data distribution elements are calculated for each emotional voice data 21 of a target user ( S200 ). For example, median, variance, skewness, and kurtosis are used as data distribution elements, but the present invention is not limited thereto.

Specifically, using [Equation 5], the median value, variance, skewness, and kurtosis are calculated for each emotional voice data of the target user.

[Equation 5]

Here, fi is the index of the feature vector, FeatureVector _fi is a specific feature value, mean is the average value of FeatureVector _fi , that is, the same as the aforementioned TMLE, and N is the number of data.

Next, among the data of the IM 11 , data distribution elements are calculated for each data corresponding to the emotion existing in the data 21 of the target user ( S210 ). That is, the similarity ( to find similiarity).

The data distribution elements here should be the same as the data distribution elements in step S200. That is, the data distribution elements here are median, variance, skewness, and kurtosis. Similarly, the above-mentioned [Equation 5] may be used to calculate data distribution elements for each data of the IM 11 .

Referring to FIG. 9 , only data on emotions of anger and sadness exist in the emotional voice data of the target user, and data on other emotions, for example, happiness, are absent. In this case, data distribution elements for each of the data of the anger and sadness emotions of the target user are calculated, respectively, and among the data of the IM 11 , also data distribution for only the data of the anger and sadness emotions It can be seen that the elements are calculated individually.

Thereafter, the similarity between the data distribution elements of the target user data and the data distribution elements of the data of the IM 11 is calculated for each emotion ( S220 ). Here, the similarity between the data distribution elements for each emotion may be calculated based on the Euclidean distance.

For example, similarity between data of a target user and data of another user may be calculated using the following [Equation 6] and [Equation 7].

[Equation 6]

[Equation 7]

는 IM(11) 데이터의 데이터 분포 요소이며, DFN은 데이터 분포 요소의 개수이다. Here, TDDF _ei is a data distribution element of the target user data,

is a data distribution element of the IM 11 data, and DFN is the number of data distribution elements.

That is, after calculating the Euclidean distance between the emotion-specific data distribution element of the target user data 21 and the corresponding emotion-specific data distribution element of the IM 11 data using [Equation 6], [Equation 7] to calculate the similarity between the emotion-specific data distribution element of the target user data 21 and the corresponding emotion-specific data distribution element of the IM 11 data.

In the example of FIG. 9 , calculated between the data distribution elements calculated for each of the anger and sadness of the target user and the data distribution elements calculated for each of the anger and sadness of User1 and User2 respectively An example of similarity is shown in FIG. 10 . Referring to FIG. 10 , (a) is a similarity calculated between data distribution elements calculated for each of anger and sadness of the target user and data distribution elements calculated for each of anger and sadness of User1. 10.48, (b) is the similarity calculated between the data distribution elements calculated for each of the target user's anger and sadness and the data distribution elements calculated for each of the anger and sadness of User 1 (User2), indicating that it is 12.14. can Accordingly, it can be seen that the user similar to the emotional voice data of the target user is User2.

Thereafter, from among the emotional voice data of the user having the most similar emotional voice to the target user through the calculated similarity, emotion data corresponding to the target user's absence emotion is selected as the target user's absence emotion data (S230).

Referring to the example of FIG. 10 , since the user most similar to the emotional voice of the target user is User2, the emotion data of User2 corresponding to the emotion of absence of the target user is data of the emotion of absence of the target user. is chosen Referring to the example of FIG. 9 , since the target user does not have emotional data on happiness, data corresponding to the happiness emotion of User2 that is most similar to the emotional voice of the target user among the IM 11 data is the target user's emotion data. It is secured as data on happiness emotion, which is an absent emotion. Through this, the data on the happiness emotion of the target user may be non-absent data.

As described above, according to an embodiment of the present invention, when the initial emotional voice data obtained for the target user is absence emotion data, the target user is based on data distribution factors among users of the IM 11 , which is an initial model of a plurality of users. From among the emotional data most similar to the emotional voice of the user, emotion data corresponding to the target user's absence emotion may be selected and strengthened as the target user's absence emotion data.

Next, the third method will be described. That is, it relates to a robust speaker adaptation modeling method performed when voice data for each emotion collected for a target user is in an imbalanced state. In this method, in order to create a balanced state when the amount of speech data for each emotion of the target user is in an unbalanced state that is not suitable for use as a training set for machine learning for speech recognition, a small amount of emotion data is used as virtual data. way to create it.

11 is a schematic flowchart of a speaker adaptive modeling method for unbalanced emotional voice data of a target user according to an embodiment of the present invention. Before the description, in the embodiment of the present invention, an initial model (IM) 11 for a plurality of users is already built and stored in the IM database 10, and emotional voice data initially acquired for the target user It is assumed that 21 is obtained in advance and stored in the target user DB 20 .

Referring to FIG. 11 , first, an Imbalanced Ratio (IR) based on the quantity of data for each emotion of the target user is calculated ( S300 ). Here, the imbalance ratio represents the ratio of the largest number (Major Class) to the smallest number (Minor Class) among emotional data of the target user as shown in Equation 8 below.

[Equation 8]

Imbalanced Ratio = Major Class/Minor Class

Thereafter, it is determined whether the calculated imbalance ratio is smaller than a threshold ratio, for example, 2.0, which is a criterion for determining imbalance ( S310 ). Here, the critical ratio of 2.0 is only an example, and may be set differently depending on an environment such as an emotion recognition method or other use.

If the calculated imbalance ratio is equal to or greater than the threshold ratio of 2.0, the target user's emotion-specific data is determined to be in an imbalanced state, and a small number of emotion data data numbers are generated using a virtual data augmentation method (S320). Here, as the virtual data augmentation method, SMOTE (Synthetic Minority Oversampling Technique), which is well-known as an oversampling algorithm for synthesizing and augmenting data of a minority class, may be used.

If the calculated imbalance ratio is smaller than the threshold ratio of 2.0, the data for each emotion of the target user is determined to be in a balanced state, and the virtual data augmentation method is stopped ( S330 ).

As described above, according to an embodiment of the present invention, when the voice data for each emotion of the target user is in an imbalanced state, the number of emotion data with a small number of data can be increased through the virtual data augmentation method.

Hereinafter, a method for performing robust speaker adaptation modeling for personalized emotion recognition according to an embodiment of the present invention using the three methods described above will be described.

12 is a schematic flowchart of a robust speaker adaptation modeling method for personalized emotion recognition according to an embodiment of the present invention. Similarly, before the description, in the embodiment of the present invention, an initial model (IM) 11 for a plurality of users is already built and stored in the IM database 10, and the emotion initially acquired for the target user It is assumed that the voice data 21 is obtained in advance and stored in the target user DB 20 .

Referring to FIG. 12 , first, the number of data for each emotion with respect to the data 21 of the target user is compared with the threshold number ( S400 ). Here, the threshold number is a sufficient number to perform the speaker adaptive modeling method according to the third method described above, that is, the robust speaker adaptation modeling method performed when the voice data for each emotion collected for the target user is in an imbalanced state. It is the reference value used for That is, if the emotional data is smaller than the threshold number, it is an insufficient number of data to perform the third method, indicating that a larger number of data should be strengthened by performing the first or second method described above. However, when the number of emotion data is equal to or greater than the threshold number, since the number is sufficient to perform the third method without performing the first or second method, data reinforcement is performed by directly performing the third method. In general, since the initial target user's collection data is very small, the number of emotion data is smaller than the threshold number, but there may be cases where this is not the case, so this determination must be performed.

Thereafter, it is determined whether the number of data for each emotion is equal to or greater than the threshold number (S410), and if it is determined that the number of data for each emotion is smaller than the threshold number, it is determined whether the number of emotions is 0 and absent data (S420).

If it is determined that a small amount of data is present instead of absent data, the speaker adaptive modeling method according to the first method described above with reference to FIGS. 1 to 6 , that is, when the emotional voice data collected for the target user is small The data corresponding to the emotion is reinforced by performing the strong speaker adaptive modeling method performed in (S430).

However, if it is determined in step S420 that the corresponding emotion is absent data, the speaker adaptive modeling method according to the second method described above with reference to FIGS. 7 to 10 , that is, among the emotional voice data collected for the target user, A strong speaker adaptive modeling method performed when data on a specific emotion is absent is performed to reinforce data corresponding to the emotion of absence (S440).

On the other hand, if it is determined that the number of data for each emotion is equal to or greater than the threshold number in step S410, it is determined whether the emotion data of the target user is in an imbalanced state (S450). Here, the determination of the imbalance state means that the determination is made using the imbalance ratio as described above.

If the emotion data of the target user is in an imbalanced state, the speaker adaptive modeling method according to the third method, that is, the robust speaker adaptation modeling method performed when the voice data for each emotion collected for the target user is in an imbalanced state is performed. One emotion data is reinforced (S460).

However, when the emotion data of the target user is not in an imbalanced state, the speaker adaptive modeling method is stopped ( S470 ).

On the other hand, the above steps (S410 to S470) are repeatedly performed on the emotional voice data of the target user strengthened through the steps (S430) and (S440), and finally, the emotional voice data of the target user is not small. There is no missing emotion data, and the data state for each emotion is reinforced with a training dataset that is not in an unbalanced state.

As described above, in an embodiment of the present invention, speaker adaptive modeling may be adaptively performed according to the quantity or imbalanced state of emotional voice data initially collected for the target user.

Next, a robust speaker adaptive modeling apparatus for personalized emotion recognition according to an embodiment of the present invention will be described. The present apparatus corresponds to an apparatus for performing the above-described robust speaker adaptation modeling method for individualized emotion recognition.

13 is a schematic block diagram of a speaker adaptive modeling apparatus for personalized emotion recognition according to an embodiment of the present invention.

13 , the speaker adaptive modeling apparatus 100 according to an embodiment of the present invention includes a first modeling unit 110 , a second modeling unit 120 , a third modeling unit 130 , and a processing control unit ( 140).

The first modeling unit 110 performs speaker adaptive modeling according to the first method described above with reference to FIGS. 1 to 6 . That is, when the emotional voice data 21 of the target user is a small amount of data, the first modeling unit 110 selects a voice similar to the target user from the multiple user initial models (IM) 11 and refines the label to Reinforce the target user training dataset, that is, the target user data.

To this end, the first modeling unit 110 includes a labeler 111 , a feature vector value calculator 112 , an average value calculator 113 , a distance calculator 114 , a threshold value calculator 115 , and a data selector. (116) and an emotion determiner (117).

The labeler 111 performs unlabeling for removing emotions from the emotion data of the IM 11 and emotion labeling on the data 21 of the target user.

The feature vector value calculator 112 calculates a feature vector value (TfeatureVector) of data for each emotion by using the data 21 of the target user.

The average value calculator 113 calculates an average value of the feature vector values for each emotion of the target user. Here, the average value of the feature vector values may be calculated using the MLE technique as in [Equation 1] described above.

The distance calculator 114 calculates the distance between average values of the feature vector values for each emotion of the target user as the Euclidean distance as in [Equation 2] above. Also, the distance calculator 114 calculates the distance between the average value of the feature vector values for each emotion and the data of the IM 11 using the above-mentioned [Equation 4].

The threshold value calculator 115 is a half of the maximum value among the distances between average values of the feature vector values for each emotion of the target user calculated by the distance calculator 114 using the above-mentioned [Equation 3], that is, 1 A distance of /2 is calculated as a threshold value.

The data selector 116 selects data of the IM 11 within a threshold range based on the average value of the feature vector values for each emotion of the target user.

The emotion determiner 117 determines the corresponding emotion by finding the average value of the feature vector values for each emotion of the target user closest to the data of the IM 11 selected by the data selector 116 .

The second modeling unit 120 performs speaker adaptive modeling according to the second method described above with reference to FIGS. 7 to 10 . That is, when there is an emotion of the absence of data among the emotions of the target user, the second modeling unit 120 determines from among the emotion voice data of users having a voice pattern similar to that of the target user among the voices of various users in the IM 11 . It is reinforced by replacing it with emotion data corresponding to the target user's absence emotion.

To this end, the second modeling unit 120 includes a distribution factor calculator 121 , a distance calculator 122 , a similarity calculator 123 , and a data selector 124 .

The distribution element calculator 121 calculates the median value, variance, skewness, and kurtosis, which are data distribution elements, for each emotional voice data of the target user by using the above-mentioned [Equation 5]. In addition, the distribution factor calculator 121 calculates the median value, variance, skewness, and kurtosis for each user-specific emotional voice data of the IM 11 using the above-mentioned [Equation 5].

The distance calculator 122 calculates the Euclidean distance between the distribution element for each emotion of the target user and the distribution element for each emotion of the user of the IM 11 by using the above-mentioned [Equation 6].

The similarity calculator 123 calculates the similarity between the data 21 of the target user and the data of other users of the IM 11 using Equation 7 described above.

The data selector 124 selects emotion data corresponding to the target user's absence emotion from among the emotional voice data of the user having the most similar emotional voice to the target user through the calculated similarity, as the target user's absence emotion data.

The third modeling unit 130 performs speaker adaptive modeling according to the third method described above with reference to FIG. 11 . That is, when the voice data for each emotion collected for the target user is in an unbalanced state, the third modeling unit 130 generates data of a small amount of emotion as virtual data to create a balanced state, and performs data reinforcement.

To this end, the third modeling unit 130 includes an imbalance ratio calculator 131 and an SMOTE performer 132 .

The imbalance ratio calculator 131 calculates the imbalance ratio of the target user, that is, the ratio of the largest number (Major Class) and the smallest number (Minor Class) among the emotional data of the target user by using the above-mentioned [Equation 8]. do.

The SMOTE executor 132 determines that the emotional data of the target user is in an imbalanced state by the imbalance ratio of the target user calculated by the imbalance ratio calculator 131, for example, when the imbalance ratio is 2.0 or more. The number of data is generated using a virtual data augmentation method, for example, the SMOTE oversampling algorithm.

Next, when the target user's data is smaller than the threshold number, if the target user's emotional voice data is small, the processing control unit 140 performs data reinforcement by the first modeling unit 110, and the target user's emotional voice data If is absent, data enhancement by the second modeling unit 120 is performed.

In addition, if the target user's data is equal to or greater than the threshold number and the imbalance ratio is equal to or greater than the threshold ratio, the processing control unit 140 performs data enhancement by the third modeling unit 130 .

The embodiment of the present invention described above is not implemented only through the apparatus and method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto. is within the scope of the right.

Claims (15)

A robust speaker adaptive modeling method for personalized emotion recognition, comprising:
When the emotional voice data initially obtained from the target user (hereinafter referred to as “target user data”) is less than a preset threshold number and the emotional data of the target user data is not absent data, a plurality of pre-formed users Among the individual data of the unlabeled dataset in which the label is removed from the labeled training dataset of the initial model (hereinafter referred to as “initial model”) for Selecting and enriching individual unlabeled data having a similar pattern by comparison of
When the target user's data is less than the preset threshold number, and the emotional data of the target user's data is absent data, among user data most similar to the target user's emotional data among the datasets of the initial model Selecting and strengthening data corresponding to the emotion corresponding to the absence data of the target user; or
Reinforcing data for each emotion of the target user by a virtual data augmentation method using an oversampling algorithm when the target user's data is equal to or greater than the preset threshold number and the number of data for each emotion of the target user is in an imbalanced state
A speaker adaptive modeling method comprising a. According to claim 1,
Selecting and strengthening the individual unlabeled data includes:
calculating a feature vector value of the emotion-specific data of the target user;
calculating an average value of feature vector values for each emotion of the target user;
calculating the distance between the average values of the feature vector values for each emotion of the target user;
setting a threshold value based on the calculated distance;
selecting unlabeled data of the initial model that falls within the range within the threshold value based on the average value of the feature vector values for each emotion of the target user;
Determining an emotion corresponding to the unlabeled data of the initial model based on a distance between the unlabeled data of the selected initial model and the average value of the feature vector values for each emotion of the target user, and
performing label refinement with a determined emotion on the unlabeled data of the initial model
Including, speaker adaptive modeling method. 3. The method of claim 2,
The average value of the feature vector values for each emotion of the target user is calculated by MLE (Maximum Liklihood Estimation),
A speaker-adaptive modeling method. 3. The method of claim 2,
The distance between the average values of the feature vector values for each emotion of the target user is calculated as the Euclidean distance,
A speaker-adaptive modeling method. 3. The method of claim 2,
The threshold value is set to half of the maximum distance among the calculated distances,
A speaker-adaptive modeling method. According to claim 1,
The step of selecting and strengthening data corresponding to the emotion corresponding to the absence data of the target user,
Calculating a data distribution element for each emotion-specific data of the target user;
Calculating a data distribution element for each emotion-specific data for each user of the initial model;
calculating a distance between a data distribution element calculated for each emotion-specific data of the target user and a data distribution element calculated for each emotion-specific data for each user of the initial model;
Calculating the user-specific similarity between the target user and the initial model based on the calculated distance; and
From the data of the user of the initial model having the highest calculated similarity, data of the user of the initial model corresponding to the emotion corresponding to the absence data of the target user is selected, and the emotion corresponding to the absence data of the target user is selected. Steps to enrich as data
Including, speaker adaptive modeling method. 7. The method of claim 6,
wherein the data distribution factors include median, variance, skewness and kurtosis,
A speaker-adaptive modeling method. According to claim 1,
Reinforcing data for each emotion of the target user in a virtual data augmentation method using the oversampling algorithm includes:
calculating an imbalance ratio for the emotion-specific data of the target user, wherein the imbalance ratio represents the ratio of the largest number to the smallest number among the emotion-specific data of the target user; and
When the imbalance ratio is greater than or equal to a preset threshold ratio to indicate that the data for each emotion of the target user is in an imbalanced state, virtual data is generated by a virtual data augmentation method for the smallest number of emotion data among the data for each emotion of the target user. step to strengthen
Including, speaker adaptive modeling method. 9. The method of claim 8,
The virtual data augmentation method is a Synthetic Minority Oversampling Technique (SMOTE) method for synthesizing and augmenting data of a minority class,
A speaker-adaptive modeling method. As a strong speaker adaptive modeling device for personalized emotion recognition,
Emotional speech initially acquired from a target user among individual data of an unlabeled dataset in which labels are removed from a labeled training dataset of an initial model (hereinafter referred to as “initial model”) for a plurality of preformed users A first modeling unit that selects and reinforces individual unlabeled data having a similar pattern by comparing the emotion-specific dataset and feature vector values of the data (hereinafter referred to as “target user data”);
a second modeling unit for selecting and strengthening data corresponding to the emotion corresponding to the absence data of the target user from among the data of the user most similar to the emotion data of the target user from among the data of the initial model;
A third modeling unit for reinforcing data for each emotion of the target user in a virtual data augmentation method using an oversampling algorithm, and
When the target user's data is less than a preset threshold number and the emotional data of the target user's data is not absent data, data reinforcement is performed through the first modeling unit, or the target user's data is While less than the set threshold number, if the emotional data of the target user's data is absent data, data reinforcement is performed through the second modeling unit, or the target user's data is greater than or equal to the preset threshold number, the target When the number of data for each emotion of the user is in an imbalanced state, a processing control unit for performing data reinforcement through the third modeling unit
A speaker adaptive modeling device comprising a. 11. The method of claim 10,
The first modeling unit,
A labeler that unlabels the labeled data of the initial model or performs an emotion label on the selected data when selected as the target user's data among the unlabeled data of the initial model;
a feature vector value calculator for calculating a feature vector value of data for each emotion of the target user;
an average value calculator for calculating an average value of the feature vector values for each emotion of the target user;
A distance calculator for calculating the distance between the average values of the feature vector values for each emotion of the target user, or for calculating the distance between the unlabeled data of the initial model and the average value of the feature vector values for each emotion of the target user;
a threshold value calculator for calculating a threshold value based on the distance calculated by the distance calculator;
a data selector for selecting unlabeled data of the initial model that falls within the range within the threshold value based on the average value calculated by the average value calculator; and
An emotion determiner for determining an emotion corresponding to the unlabeled data of the selected initial model based on a distance between the unlabeled data of the initial model selected by the data selector and the average value calculated by the average value calculator
Including, speaker adaptive modeling device. 12. The method of claim 11,
The threshold value calculator calculates, as the threshold value, half of the maximum distance among the distances calculated by the distance calculator.
Speaker adaptive modeling device. 11. The method of claim 10,
The second modeling unit,
A distribution element calculator for calculating a data distribution element for each emotion-specific data of the target user and a data distribution element for each emotion-specific data for each user of the initial model;
A distance calculator for calculating a distance between a data distribution element calculated for each emotion-specific data of the target user and a data distribution element calculated for each emotion-specific data for each user of the initial model;
a similarity calculator for calculating the similarity for each user of the target user and the initial model based on the distance calculated by the distance calculator;
From the data of the user of the initial model having the highest calculated similarity, data of the user of the initial model corresponding to the emotion corresponding to the absence data of the target user is selected, and the emotion corresponding to the absence data of the target user is selected. Data selector to enrich as data
Including, speaker adaptive modeling device. 14. The method of claim 13,
wherein the data distribution factors include median, variance, skewness and kurtosis,
Speaker adaptive modeling device. 11. The method of claim 10,
The third modeling unit,
An imbalance ratio calculator for calculating an imbalance ratio for the emotion-specific data of the target user, wherein the imbalance ratio represents the ratio of the largest number to the smallest number among the emotion-specific data of the target user; and
When the imbalance ratio is greater than or equal to a preset threshold ratio to indicate that the data for each emotion of the target user is in an imbalanced state, virtual data is generated by a virtual data augmentation method for the smallest number of emotion data among the data for each emotion of the target user. data augmentation performer that enhances
Including, speaker adaptive modeling device.

Images