JP7141641B2 - Paralinguistic information estimation device, learning device, method thereof, and program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Law (1) Date of posting on the website February 19, 2019 Website address Website of the 2019 Spring Research Presentation Meeting of the Acoustical Society of Japan http://www. asj. gr. jp/annualmeeting/index. html (2) Date March 5th to March 7th, 2019 (Publication date: March 6th, 2019) Assembly name The Acoustical Society of Japan 2019 Spring Research Presentation Venue The University of Electro-Communications

The present invention relates to technology for estimating paralinguistic information from speech.

Techniques for estimating paralinguistic information from uttered speech are known. Paralinguistic information means peripheral information other than the language among the linguistic information given to the listener by the speaker. For example, paralinguistic information includes the speaker's emotions such as joy, sadness, anger, and calmness, the speaker's attitude (polite, high-handed, etc.), and the speaker's intention (positive, negative, etc.).

For example, Non-Patent Document 1 discloses a paralinguistic information estimation technique based on expression learning. Representation learning refers to a method of automatically extracting factors representing data and solving estimation problems based on the extracted factors. For example, in estimating paralinguistic information based on expression learning, a plurality of factors expressing speech are extracted from speech (speech data), and these factors are used as inputs to estimate paralinguistic information. A factor expressing speech (speech data) is an element expressing each feature of the speech. Examples of factors are elements of the spoken word (phonological), elements of the speaker (speakerness), elements of speaking style (speech style), characteristics of background noise, characteristics of reverberation, and the like.

Semi-supervised learning is possible in representation learning, and highly accurate models can be obtained when there is a small amount of supervised labeled data but a large amount of unsupervised unlabeled data. In representation learning using conventional semi-supervised learning, a factor extraction model for extracting factors is trained with a large amount of unsupervised unlabeled data, and then a reconstruction model that reconstructs the original data from the factors is trained with a small amount of data. Train with supervised labeled data. Since the learning of the conventional factor extraction model is based on whether or not data can be reconstructed from the factors extracted by the factor extraction model, unsupervised label data can be used for learning. In addition, since the reconstruction model learns on the basis of minimizing the data reconstruction error, it is possible to obtain a highly accurate reconstruction model even from a small amount of teacher labels. Collecting speech for paralinguistic information is easy, and a large amount of unsupervised unlabeled data can be collected. On the other hand, creating supervised labels for paralinguistic information requires multiple listeners, and often yields only a small amount of supervised labeled data. Therefore, it can be said that expression learning is a technique suitable for paralinguistic information estimation.

S. E. Eskimez, Z. Duan and W. Heinzelman, “Unsupervised Learning Approach to Feature Analysis for Automatic Speech Emotion Recognition,” in Proc. of ICASSP, 2018, pp. 5099 - 5103.

However, the factor series extracted by the conventional factor extraction model may contain unnecessary factors for estimating paralinguistic information. For example, even if the spoken words are the same, the utterance style may differ depending on whether the person is angry or happy. Therefore, when estimating emotions as paralinguistic information, there is a possibility that the elements of utterance patterns are highly necessary, but the elements of phonology are less necessary. However, in the conventional method, paralinguistic information is estimated using all the factors extracted from the speech, so there is a high possibility that the paralinguistic estimation is performed using factors that are essentially unnecessary for estimating the paralinguistic information. Since unnecessary information for estimation often works as noise, there is a risk that the accuracy of estimating paralinguistic information will decrease in the conventional method.

SUMMARY OF THE INVENTION The present invention has been made in view of these points, and an object of the present invention is to improve the accuracy of paralinguistic information estimation in paralinguistic information estimation based on expression learning.

In order to solve the above problems, we take the acoustic features of speech as an input, extract the factors necessary for estimating the paralinguistic information of the speech from among the factors representing the speech, and use the extracted factors as input to estimate the paralinguistic information of the speech. is estimated and output.

In the present invention, since the paralinguistic information is estimated using factors necessary for estimating the paralinguistic information, the estimation accuracy of the paralinguistic information can be improved.

FIG. 1 is a block diagram illustrating the functional configuration of the learning device of the first embodiment. FIG. 2 is a block diagram illustrating the functional configuration of the paralinguistic information estimation device of the first and second embodiments. FIG. 3 is a conceptual diagram for explaining the processing of the factor extraction model learning unit of the first embodiment. FIG. 4 is a block diagram illustrating the functional configuration of the learning device of the second embodiment. FIG. 5 is a conceptual diagram for explaining the processing of the factor extraction model learning unit of the second embodiment.

Embodiments of the present invention will be described below.
[principle]
Explain the principle. In the paralinguistic information estimation based on expression learning of each embodiment, the factors necessary for estimating the paralinguistic information of the speech are extracted from the acoustic features of the speech, from among the factors representing the speech. Estimate and output the paralinguistic information of As a result, the factors unnecessary for estimating the paralinguistic information are removed, and preferably only the factors required for estimating the paralinguistic information are used to estimate the paralinguistic information, thereby improving the estimation accuracy of the paralinguistic information.

Here, to extract the factors necessary for estimating the paralinguistic information of the speech, the acoustic features of the speech are input, and the factors necessary for estimating the paralinguistic information of the speech are extracted and output from the factors representing the speech. Use a factor extraction model that A paralinguistic information estimation model is used for estimating paralinguistic information, which inputs one or more factors and estimates and outputs the paralinguistic information of speech. These factor extraction models and paralinguistic information estimation models are obtained by machine learning. The point is a factor extraction model that removes elements such as phonology, speaker characteristics, background noise characteristics, and reverberation characteristics that are considered unnecessary for paralinguistic information estimation based on expression learning among the factors representing speech. is to learn

However, when a factor sequence representing speech is extracted from acoustic features by a conventional technique such as Non-Patent Document 1, the extracted factor sequence is expressed as a single factor vector. It is not possible to determine which dimension value corresponds to which factor value (for example, which dimension element of the factor vector represents the phonological factor). Therefore, it is not possible to simply select necessary or unnecessary factors for estimating paralinguistic information from the factor series extracted by the conventional technique.

On the other hand, in each embodiment, when it becomes impossible to estimate a specific factor from the factor series extracted by the factor extraction model, the specific factor is completely extracted from the factor series extracted by the factor extraction model. can be considered to have been removed by For example, when it becomes impossible to estimate the phonology from the factor sequence estimated by the factor extraction model, it is assumed that the phonology can be considered to be removed from the factor sequence estimated by the factor extraction model. Under this assumption, in each embodiment, it is difficult to estimate a specific element (hereinafter "removed element") unnecessary for estimating paralinguistic information from the factor series extracted by the factor extraction model (preferably learns a factor extraction model so that the estimation of the removed component is not possible). That is, the factor extraction model learning unit of the embodiment corrects the acoustic feature amount of the training speech and the removal element, which is a single factor unnecessary for estimating the paralinguistic information of the training speech among the factors representing the training speech. (1) Acoustic features of speech are input. and input the factor extraction model that extracts and outputs the factors necessary for estimating the paralinguistic information of speech from among the factors representing speech, and (2) the correct labels of the factors and removed elements output from the factor extraction model. , a reconstruction model that reconstructs the acoustic features of speech, and (3) a single factor that is unnecessary for estimating paralinguistic information among the factors that represent speech, with the factors output from the factor extraction model as input. or a removal factor estimation model for estimating a plurality of factors that are unnecessary for estimating paralinguistic information of speech among factors representing speech. At that time, the factor extraction model learning unit increases the error between the removed element estimated by the removed element estimation model and the correct label when the acoustic feature amount of the training speech is input to the factor extraction model. Train a factor extraction model. Preferably, the factor extraction model learning unit maximizes the error between the removed element estimated by the removed element estimation model and the correct label when the acoustic feature amount of the training speech is input to the factor extraction model. to learn a factor extraction model. A large error means that it is difficult to estimate the removed elements from the factors extracted by the factor extraction model. Difficult (e.g., impossible). Therefore, the factor extraction model obtained by such learning is presumed to extract the factors necessary for estimating the paralinguistic information of speech from among the factors representing speech.

By inputting the acoustic features of the speech into the factor extraction model obtained in this way, the factors necessary for estimating the paralinguistic information of the speech are extracted from among the factors representing the speech. The paralinguistic information of the speech is estimated by inputting the factors necessary for estimating the linguistic information into the paralinguistic information estimation model.

[First embodiment]
Next, a first embodiment will be described. In the present embodiment, a single factor (removal element) unnecessary for estimating paralinguistic information among the factors representing speech is exemplified.
<Configuration>
As illustrated in FIG. 1, the learning device 11 of this embodiment includes acoustic feature extraction units 111-1 and 111-2, a factor extraction model learning unit 112, a factor extraction model storage unit 113, a factor extraction unit 114, a para language It has an information estimation model learning unit 115 and a paralinguistic information estimation model storage unit 116 . As illustrated in FIG. 2, the paralinguistic information estimation device 12 of this embodiment includes an acoustic feature extraction unit 121, a factor extraction model storage unit 123, a factor extraction unit 124, a paralinguistic information estimation unit 125, and a paralinguistic information estimation unit. It has a model storage unit 126 .

<Learning processing>
Next, the learning process of this embodiment will be described with reference to FIGS. 1 and 3. FIG.
≪Learning data≫
As a premise of the learning process, a learning voice V _t1 , a correct label LA _ta of a removal element, a learning voice V _t2 , and a correct label LA _tp of paralinguistic information are prepared. The training speech _Vt1 is time-series data of uttered speech, and is unsupervised label data for learning the factor extraction model _Mf described above. The correct label LA _ta of the removed element is the correct label of the removed element, which is a single factor unnecessary for estimating the paralinguistic information of the training speech V _{t1 among the factors FA t1 representing} the training speech V _t1 . Examples of removal factors are phonology, speaker characteristics, background noise characteristics, reverberation characteristics, and the like. When the element to be removed is phonological, for example, information representing the phoneme string corresponding to each utterance of the learning speech V _t1 is the correct label LA _ta . When the removal element is speaker characteristics, for example, a speaker ID or index corresponding to each utterance of the training speech V _t1 , a continuous representation of speaker information estimated using a pre-trained model (for example, i- vector (reference document 1, etc.) is the correct label LA _ta . If the element to be removed is the background noise characteristic, for example, the background noise ID or index representing the correct answer for the type of background noise of the learning speech _Vt1 (for example, 0 for no noise, 1 for noise in the car, 2 for crowds, other If 3, etc.) is the correct label LA _ta . If the removal element is the characteristic of reverberation, for example, the correct value of the reverberation time (RT60) (for example, 0 for short reverberation with a reverberation time of less than 500 ms, 1 for general reverberation with a reverberation time of 500 ms or more and less than 1000 ms, long reverberation of 1000 ms or more If 2, etc.) is the correct label LA _ta . The training speech V _t2 is time-series data of uttered speech, and the correct label LA _tp of the paralinguistic information is the correct label representing the paralinguistic information of each utterance of the training speech V _t2 . The learning voice V _t2 may be the same as the learning voice V _t1 or may be different from the learning voice V _t1 .
Reference 1: N. Dehak, PJ Kenny, R. Dehak, P. Dumouchel, P. Ouellet, “Front-End Factor Analysis for Speaker Verification,” in IEEE Trans. on Audio, Speech, and Language Processing, vol. 19 , no. 4, pp. 788-798, 2011.

<<Processing of Acoustic Feature Extraction Unit 111-1>>
Acoustic feature extraction unit 111-1 receives training speech V _t1 as input, extracts and outputs acoustic feature vector sequence F _t1 , which is a sequence (for example, time series) of acoustic features of learning speech V _t1 ( Step S111-1). The elements that make up the acoustic feature vector sequence F _t1 are not limited _. It is a sequence of vectors whose elements include one or more acoustic features of output, fundamental frequency, and short-time power. For example, the acoustic feature vector sequence F _t1 has the same sequence length as the correct label LA _ta of the removed element.

<<Processing of Factor Extraction Model Learning Unit 112>>
In this embodiment, processing using a framework based on deep learning is exemplified. However, this is not a limitation of the invention. The factor extraction model learning unit 112 selects specific removal elements (for example, phonological characteristics, speaker characteristics, background noise characteristics, or one of the reverberation characteristics) is removed, and the factor extraction model _Mf is learned to output the remaining factors (low-dimensional vectors) (step S112).

As illustrated in FIG. 3, the factor extraction model learning unit 112 _acquires the acoustic feature vector sequence F _t1 (acoustic feature amount) of the learning voice V _t1 and the learning voice V t1 among the factors representing the learning voice V _t1 . (1) Acoustic feature vector sequence (acoustic feature quantity) of _speech is input, and among the factors representing speech, Input the factor extraction model M _f that extracts and outputs the factor FA _t1 necessary for estimating the paralinguistic information of speech, and (2) the factor FA _t1 output from the factor extraction model M _f and the correct label of the removed element. A reconstruction model _Mr for reconstructing an acoustic feature vector sequence (acoustic feature amount) of speech and a factor FA _t1 output from (3) a factor extraction model _Mf are input, and speech A removed element estimation model M _a for estimating a removed element LA _a , which is a single factor unnecessary for estimating the paralinguistic information, is learned at the same time. For example, the _{factor extraction} model _learning unit 112 _uses the factor FA When _t1 and the correct label LA _ta of the removed element are input to the reconstruction model _Mr , the acoustic feature vector sequence (acoustic feature quantity) reconstructed by the reconstruction model _Mr and the acoustic feature vector sequence _Ft1 Output from the factor extraction model M _f when the loss function value L _r of the reconstruction model M _r corresponding to the error and the acoustic feature vector sequence F _t1 of the learning speech V _t1 are input to the factor extraction model M _f The removed element estimation model M _a corresponding to the error between the removed element LA _a estimated by the removed element estimation model M _a when the factor FA _t1 is input to the removed element estimation model M _a and the correct label LA _ta of the removed element and the weighted sum of the total loss function value L, the factor extraction model M _f , the reconstruction model M _r , and the removed _element estimation model M _a are learned.

For learning these models, for example, the error backpropagation method is used. That is, the factor extraction model learning unit 112 forward propagates the acoustic feature vector sequence _Ft1 to the factor extraction model _Mf and the reconstruction model _Mr , and transfers the correct label LA _ta of the removed element to the reconstruction model _Mr. A value corresponding to the error between the reconstructed acoustic feature vector sequence obtained by forward propagation and the acoustic feature vector sequence _Ft1 is set as the reconstruction loss function value _{Lr, and the acoustic feature vector sequence Ft1 is} the factor extraction model. The removed element LA _a and the correct label LA of the removed element obtained by forward propagation to _Mf and the removed element estimation model M _a , and forward propagating the correct label LA _ta of the removed element to the removed element estimation model M _a The value corresponding to the error from _ta is set as the loss function value L _a of the removed element, and the weighted sum of these two loss function values L _r and L _a is set as the overall loss function value L, and three models are learned. The correct label LA _ta of the removed element is forward propagated to the reconstructed model M _r because, ideally, the factor FA _t1 output from the factor extraction model M _f does not include the removed element, and only the factor FA _t1 This is because the acoustic feature vector sequence cannot be reconstructed.

Examples of errors between the acoustic feature vector sequence reconstructed by the reconstruction model M _r and the acoustic feature vector sequence F _t1 are these squared errors. An example of the loss function value _Lr is the squared error between the acoustic feature vector sequence reconstructed by the reconstruction model _Mr and the acoustic feature vector sequence _Ft1 . An example of the error between the removed element LA _a estimated by the removed element estimation model M _a and the correct label LA _ta of the removed element is these cross-entropies. An example of the loss function value L _a is the cross entropy between the removed element LA _a estimated by the removed element estimation model M _a and the correct label LA _ta of the removed element. An example of the overall loss function value L is expressed as follows.
L=(1−α)L _r +αL _a (1)
where α is a loss weight and is a constant that satisfies 0≦α≦1.

However, the factor extraction model learning unit 112 performs the removal estimated by the removed element estimation model M _a when the acoustic feature vector sequence F _t1 (acoustic feature quantity) of the learning speech V _t1 is input to the factor extraction model M _f . The factor extraction model M _f is used so that the error between the element LA _a and the correct label LA _ta (for example, the cross entropy between the removed element LA _a estimated by the removed element estimation model M _a and the correct label LA _ta ) becomes large. to learn. For example, when learning is performed by the error backpropagation method, _a gradient reversal layer (GRL) is arranged between the factor extraction model _Mf and the removed element estimation model Ma. The gradient inversion layer performs identity transformation during forward propagation. That is, the gradient inversion layer directly outputs the forward propagated input value as the output value. That is, the gradient inversion layer does nothing during forward propagation. However, the gradient inversion layer inverts the gradient of the input value during backpropagation in the error backpropagation method. That is, the gradient inversion layer outputs a value obtained by multiplying the partial differential value corresponding to the input error by a negative constant. That is, during back propagation from the removed element estimation model M _a to the factor extraction model M _f in the error back propagation method, between the removed element LA _a output from the removed element estimation model M _a and the correct label LA _ta The partial differential value corresponding to the error (the partial differential value at the weight or bias to be updated of the error) is back-propagated from the removed element estimation model _Ma to the gradient inversion layer, and the gradient inversion layer negatively affects this partial differential value. The value multiplied by the constant is propagated back to the factor extraction model _Mf . Details of gradient reversal layers are described, for example, in Reference 2. By the function of this gradient inversion layer, learning is performed so that it becomes difficult to estimate the removed element from the factor extracted by the factor extraction model _Mf (preferably, the learning is performed so that the removed element cannot be estimated from the factor). Therefore, as the learning of the three models progresses, the acoustic feature vector sequence can be correctly reconstructed by the reconstruction model M _r , but the factors that make it difficult to estimate the removed elements by the removed element estimation model M _a A factor extraction model M _f can be obtained that extracts That is, the factor extraction model learning unit 112 can learn the factor extraction model _Mf capable of extracting factors from which removed elements unnecessary for paralinguistic information estimation are removed.
Reference 2: Yaroslav Ganin, Victor Lempitsky, “Unsupervised Domain Adaptation by Backpropagation,” Skolkovo Institute of Science and Technology (Skoltech), Moscow Region, Russia

Learning by the factor extraction model learning unit 112 is repeated until a predetermined end condition is satisfied. For example, when the input epoch number (the number of times the acoustic feature vector sequence F _t and the correct label LA _ta are all used for learning) reaches a certain value (for example, 100 times), Assuming that the learning is completed, the factor extraction model _Mf at that time is output. The factor extraction model _Mf is stored in the factor extraction model storage unit 113 (step S113). The reconstructed model M _r and the removed element estimation model M _a are not used in subsequent processing.

<<Processing of Acoustic Feature Extraction Unit 111-2>>
Acoustic feature extraction unit 111-2 receives training speech V _t2 as input, extracts and outputs acoustic feature vector sequence F _t2 , which is a sequence (for example, time series) of acoustic feature amounts of learning speech V _t2 ( step S111-2). The type of computation for the acoustic feature extraction unit 111-2 to extract the acoustic feature vector sequence F _t2 from the training speech V _t2 is as _{follows .} is the same as the type of operation for extracting In other words, the type of acoustic feature quantity that is the element that configures the acoustic feature vector sequence _{Ft2 is the same as the type of the acoustic feature quantity that is the element that configures the acoustic feature vector sequence Ft1 .} For example, the acoustic feature vector sequence F _t2 has the same sequence length as the correct label LA _ta of the removed element. The acoustic feature vector sequence F _t2 and the acoustic feature vector sequence F _t1 may or may not have the same length.

<<Processing of Factor Extraction Unit 114>>
The acoustic feature vector sequence Ft2 output from the acoustic feature extraction unit _111-2 and the factor extraction model _Mf read from the factor extraction model storage unit 113 are input to the factor extraction unit 114. FIG. The factor extraction unit 114 inputs the acoustic feature vector sequence F _t2 to the factor extraction model M _f , _{and selects} the factor sequence (a plurality of (for example, the time series of factor vectors whose elements are the factors at each point in time) FA _t2 is extracted and output (step S114). For example, the factor sequence FA _t2 has the same length as the acoustic feature vector sequence F _t2 and the correct label LA _tp of the paralinguistic information.

<<Processing of Paralinguistic Information Estimation Model Learning Unit 115>>
The correct label LA _tp of the paralinguistic information and the factor sequence FA _t2 output from the factor extracting unit 114 are input to the paralinguistic information estimation model learning unit 115 . The paralinguistic information estimation model learning unit 115 learns the paralinguistic information estimation model M _p using a set of the factor sequence FA _t2 corresponding to each utterance and the correct label LA _tp of the paralinguistic information (step S115). The paralinguistic information estimation model _Mp is a model that can handle multi-class classification problems. An example of the paralinguistic information estimation model M _p is a model based on deep learning. However, this is not a limitation of the invention and the paralinguistic information estimation model M _p may be another multi-class classification model such as multi-class logistic regression. When the paralinguistic information estimation model M _p uses a model based on deep learning, for example, a factor sequence FA _t2 is input to a model based on a recurrent neural network, and the final output of the model and the correct label LA _tp of the paralinguistic information Model learning is performed by the error backpropagation method with the cross-entropy as the loss function. However, this is not a limitation of the invention. The paralinguistic information estimation model learning unit 115 outputs the paralinguistic information estimation model _Mp obtained by learning, and stores it in the paralinguistic information estimation model storage unit 116 (step S116).

<Paralinguistic information estimation processing>
Paralinguistic information estimation processing according to this embodiment will be described with reference to FIG.
As preprocessing, the factor extraction model M _f stored in the factor extraction model storage unit 113 of the learning device 11 is stored in the factor extraction model storage unit 123 of the paralinguistic information estimation device 12, and is stored in the paralinguistic information estimation model storage unit 116. The stored paralinguistic information estimation model M _p is stored in the paralinguistic information estimation model storage unit 126 .

<<Processing of Acoustic Feature Extraction Unit 121>>
Acoustic feature extraction unit 121 receives as input speech _Vin , which is an object for estimating _{paralinguistic} information, and extracts and outputs acoustic feature vector series _Fin , which is a series (for example, time series) of acoustic features of speech Vin. (step S121). Voice _Vin is time-series data of uttered voice. Further, the type of computation for the acoustic feature extraction unit 121 to extract the acoustic feature vector sequence F _in from the voice V _in is as follows: the acoustic feature extraction unit 111-2 extracts the acoustic feature vector sequence F _t2 from the learning voice V _t2 ; It is the same as the type of operation for That is, the type of the acoustic feature quantity that is the element forming the acoustic feature vector sequence F _in is the same as the type of the acoustic feature quantity that is the element forming the acoustic feature vector sequence F _t2 .

<<Processing of factor extraction unit 124>>
The acoustic feature vector series F _in output from the acoustic feature extraction unit 121 and the factor extraction model M _f read from the factor extraction model storage unit 123 are input to the factor extraction unit 124 . The factor extraction unit 124 inputs the acoustic feature vector sequence F _in to the factor extraction model M _f , and _{extracts the factor sequence FA in necessary} for estimating the _{paralinguistic} information of the speech Vin from among the factors representing the training speech Vin. and output (step S124).

<<Processing of Paralinguistic Information Estimating Unit 125>>
Paralinguistic information estimation unit 125 receives factor sequence FA _in output from factor extraction unit 124 and paralinguistic information estimation model M _p read from paralinguistic information estimation model storage unit 126 . The paralinguistic information estimation unit 125 inputs the factor sequence FA _in to the paralinguistic information estimation model M _p , estimates the paralinguistic information of the speech Vin, and outputs it as a _{paralinguistic} information estimation result P (step S125). Since the paralinguistic information estimation model M _p of the present embodiment is a model based on deep learning, the paralinguistic information estimation unit 125 inputs the factor sequence FA _in to the paralinguistic information estimation model M _p and propagates it forward. A _{paralinguistic} information estimation result P of the voice Vin is obtained and output. When the output of the paralinguistic information estimation model M _p is obtained as the probability of each paralinguistic information class, the paralinguistic information estimation unit 125 selects, for example, the paralinguistic information class that maximizes the output of the paralinguistic information estimation model M _p . Output as a paralinguistic information estimation result P. However, this does not limit the present invention. For example, the paralinguistic information estimation unit 125 may output the probability of each paralinguistic information class output from the paralinguistic information estimation model _Mp .

[Second embodiment]
Next, a second embodiment will be described. This embodiment exemplifies a case where there are a plurality of factors (removal elements) that are unnecessary for estimating paralinguistic information among factors representing speech. Note that the following description will focus on differences from the items that have been described so far, and the same reference numbers will be used for the items that have already been described to simplify the description.

<Configuration>
As illustrated in FIG. 4, the learning device 21 of this embodiment includes acoustic feature extraction units 111-1 and 111-2, a factor extraction model learning unit 212, a factor extraction model storage unit 113, a factor extraction unit 114, a para language It has an information estimation model learning unit 115 and a paralinguistic information estimation model storage unit 116 . The paralinguistic information estimation device 12 of this embodiment is the same as that of the first embodiment.

<Learning processing>
Next, the learning process of this embodiment will be described with reference to FIGS. 4 and 5. FIG.
≪Learning data≫
In this embodiment, as a premise of the learning process, the learning speech V _t1 , the correct labels LA _{ta ,1} , _{. tp} is prepared. However, N is an integer of 2 or more and represents the number of elements to be removed. Also, let n=1, . . . , N. The correct label LA _ta,n of each removed element is the correct label of the removed element, which is a single factor unnecessary for estimating the paralinguistic information of the training speech V _{t1 among the factors FA t1 representing} the training speech V _t1 . is. Specific examples of removal elements are as illustrated in the first embodiment. In the present embodiment, for example, two or more factors out of phonological characteristics, speaker characteristics, background noise characteristics, and reverberation characteristics are used as removal factors. LA _{ta ,1} , . For example, LA _ta,1 , . . . , LA _ta,N are different from each other.

<<Processing of Factor Extraction Model Learning Unit 212>>
The processing of the acoustic feature extraction unit 111-1, the processing of the acoustic feature extraction unit 111-2, the processing of the factor extraction unit 114, and the processing of the paralinguistic information estimation model learning unit 115 are as described in the first embodiment. Only the processing of the factor extraction model learning unit 212, which is different from the first embodiment, will be described below.

This embodiment also exemplifies processing using a framework based on deep learning. However, this is not a limitation of the invention. The factor extraction model learning unit 212 selects specific removal elements (for example, phonological characteristics, speaker characteristics, background noise characteristics, or reverberation characteristics) is learned (step _S212 ).

As illustrated in FIG. 5, the factor extraction model learning unit 212 _acquires the acoustic feature vector sequence F _t1 (acoustic feature amount) of the learning voice V _t1 and the learning voice V t1 among the factors representing the learning voice V _t1 . , LA _{ta ,N} of a plurality of removed elements, which are a plurality of factors unnecessary for estimating paralinguistic information, are input, and (1) a factor extraction model M _f and (2) The reconstruction model M _r and (3) a plurality of removal element estimation models M _a,1 , . . . , M _a,N are learned. However, each removed element _estimation model M _{a,n (} where n=1, . It is a model for estimating the removal factor LA _a,n , which is a single factor unnecessary for estimating information. For example, the _{factor extraction} model _learning unit 112 _uses the factor FA Acoustic feature vector sequences (acoustic features) reconstructed by the _{reconstruction} model _Mr when _t1 and the correct labels LA _{ta ,1} , . and the _{acoustic feature vector sequence Ft1 .} When the factor FA _t1 output from the factor extraction model _Mf is input to the removal element estimation model M _{a ,n} , the removal element LA _a,n estimated by the removal element estimation model M a,n and the correct answer of the removal element Using the weighted sum of the loss function value L _a,n of the removed element estimation model M _a,n corresponding to the error with the label LA _ta,n as the overall loss function value L, the factor extraction model M _f is reconstructed. , _{M a ,N} are learned.

As in the first embodiment, for example, error backpropagation is used for learning these models. That is, the factor extraction model learning unit 212 forward propagates the acoustic feature vector sequence _Ft1 to the factor extraction model _Mf and the reconstruction model _Mr , and transfers the correct label LA _ta of the removed element to the reconstruction model _Mr. A value corresponding to the error between the reconstructed acoustic feature vector sequence obtained by forward propagation and the acoustic feature vector sequence _Ft1 is set as the reconstruction loss function value _{Lr, and the acoustic feature vector sequence Ft1 is} the factor extraction model. Each removal element LA _a, obtained by forward propagation to _Mf and each removal element estimation model M _a,n , and forward propagation of the correct label LA _ta,n of each removal element to the removal element estimation model M _a,n A value corresponding to the error between n and the correct label LA _{ta,n of} each elimination element is defined as the loss function value L _a,n of each elimination element, and these loss function values L _r , L _{a, 1} , . . . , L _{a , N} is the overall loss function value L, and the factor extraction model M _f , the reconstruction model M _r , and the removed element estimation model M _a,1 , . . . , M _a,N are learned.

ただし、αは損失重みであり、０≦α≦１を満たす定数である。β_ｎはｎ番目の除去要素ＬＡ_ａ，ｎの除去要素重みであり、０≦β_ｎ≦１および

を満たす定数である。ｃは正定数であり、例えばｃ＝１である。すなわち、因子抽出モデル学習部２１２は、例えば、再構成モデルＭ_ｒの損失関数値Ｌ_ｒと、各除去要素ＬＡ_ａ，ｎに対応する除去要素重みβ_ｎと各除去要素ＬＡ_ａ，ｎに対応する除去要素モデルの損失関数値Ｌ_ａ，ｎの積β_ｎＬ_ａ，ｎと、の重み付き和を全体の損失関数値Ｌ_ａ，ｎとして、因子抽出モデルＭ_ｆと、再構成モデルＭ_ｒと、複数の除去要素ＬＡ_ａ，１，…，ＬＡ_ａ，Ｎを推定する除去要素推定モデルＭ_ａ，１，…，Ｍ_ａ，Ｎとを学習する。 An example of the error between each removed element LA _a,n estimated by each removed element estimation model M _a,n and the correct label LA _ta,n of each removed element is these cross-entropies. An example of each loss function value L _a,n is the cross entropy between each removal element LA _a,n estimated by each removal element estimation model M _a,n and the correct label LA _ta,n of each removal element. An example of the overall loss function value L is expressed as follows.

where α is a loss weight and is a constant that satisfies 0≦α≦1. β _n is the removal factor weight of the nth removal factor LA _a,n , 0≦β _n ≦1 and

is a constant that satisfies c is a positive constant, for example c=1. That is, the factor extraction model learning unit 212, for example, the loss function value L _r of the reconstruction model _Mr , the removal element weight β _n corresponding to each removal element LA _{a , n} , and the removal element LA a, n The weighted sum of the product β _n L _a,n of the loss function values L _a,n of the removed element models is set as the overall loss function value L _a,n , and the factor extraction model M _f and the reconstruction model M _r , and a removal element estimation model M _a,1 , . . . , M _{a, N} that estimates a plurality of removal elements LA _a,1 , .

Although all removal factor weights β ₁ , . . . , β _N may have the same value ₍ eg, β _{1 =} . It is also possible to learn a factor extraction model _Mf that strongly removes elements. That is, the factor extraction model M _f obtained in the present embodiment extracts a factor series in which the removed element corresponding to the loss function value L _a,n with the larger removed element weight β _n is removed more strongly.

Also, the removal element weight _βn corresponding to a removal element that is difficult to estimate from the acoustic feature vector sequence (acoustic feature amount) of the speech may be increased. That is, the plurality of removal elements includes a first removal element and a second removal element that is easier to estimate from the acoustic feature vector sequence than the first removal element, and the removal element weight value corresponding to the first removal element is It may be greater than the value of the removal factor weight corresponding to the second removal factor. This makes it possible to sufficiently remove removal elements that are difficult to estimate and remove from the acoustic feature vector sequence. The ease and difficulty criteria for estimating removal factors are illustrated.
Criterion 1: A removed element with fewer classes is easier to estimate, and a removed element with more classes is more difficult to estimate. That is, if the first removal factor is one of the CL1 classes, the second removal factor is one of the CL2 classes, and CL1>CL2, the estimation of the second removal factor is the first removal factor is easier than estimating
Criterion 2: Estimation of speaker-related elimination factors is easier than estimation of phonological elimination factors.
Criterion 3: The ease and difficulty of estimating removal factors may be determined experimentally. For each _n =1, . . . , _N , for each _{n = 1} , . As _{ta and n} , the processing of the acoustic feature extraction unit 111-1 and the factor extraction model learning unit 112 of the learning device 11 of the first embodiment is performed to obtain the loss function L of equation (1). Here, it is judged that the smaller the loss function L is, the more appropriately the removal factor is estimated and the easier the estimation of the removal factor is. That is, the loss function corresponding to the first removal element is L _n1 (n1 ∈ {1, ..., N}), and the loss function corresponding to the second removal element is L _n2 (n2 ∈ {1, ..., N} , n1≠n2) and L _n1 >L _n2 , the estimation of the second removal factor is easier than the estimation of the first removal factor.

Similar to the factor extraction model learning unit 112 of the first embodiment, the factor _extraction model learning unit _{212 performs} First, the factor extraction model _Mf is learned so that the error between the removed element LA _a,n estimated by the removed element estimation model M _a, n and the correct label LA _ta,n becomes large. That is, the factor extraction model _Mf is learned so that it becomes difficult for each removal element estimation model M _a,n to estimate the removal element from the factor series extracted by the factor extraction model _Mf . For example, when learning is performed by the error backpropagation method, the gradient reversal layer GRL _n is arranged between the factor extraction model M _f and the removed element estimation model M _a,n as described in the first embodiment. be. Each gradient reversal layer GRL _n is the same as described in the first embodiment.

The factor extraction model learning unit 212 may simultaneously _{learn the N} removal element _estimation models M _a,1 , . The removed element estimation models M _a,1 , . . . , M _a,N may be learned step by step. That is, since it may be difficult to learn a removal element estimation model that removes a plurality of removal elements at once, the removal element estimation model that removes each removal element may be learned step by step. As a result, it is possible to learn a factor extraction model _Mf that can successfully remove each removed element, and possibly improve the accuracy of estimating paralinguistic information. In this case, for example, M is an integer of 2 or more, m = 1, ..., M, and _a set {M _{a , 1 . _ _ _ _ _ _ _ a, N} }. The factor extraction model learning unit 212 first learns a removed element estimation model belonging to Subset ₁ together with the factor extraction model _Mf and the reconstruction model _{Mr r ,} and then learns the subset with the factor extraction model _Mf and the reconstruction model Mr _The removal element estimation model belonging to Subset M is learned step by step, and finally, the removal element estimation model M _a belonging to Subset _M together with the factor extraction model M _f and the reconstruction model M _{r. , 1} , . . . , M _a,N are learned. The factor extraction model learning unit 212 determines the removal element estimation model M a, _{g ( m ,1 ) , . _ (m)) , the} values of the removal element weights β _{g(m, 1)} , . conduct. where g and h are functions, g(m, 1), ..., g(m, h(m)) are function values, and {g(m, 1), ..., g(m, h (m))}ε{1,...,N}, and h(m) is an integer function value satisfying 1≤h(m)≤N. _{, Subset M ,} that is, the learning order of the removed element estimation models is not limited. It is desirable to learn sequentially from the removed element estimation model that estimates the elements. For example, it is desirable that the removal factor estimation performed by any removal element estimation model belonging to Subset _ν is easier than the removal factor estimation performed by the removal element estimation model newly added to the learning target in Subset _ν+1 . As a result, a factor extraction model _Mf capable of successfully removing each removed element can be learned, and the accuracy of estimating paralinguistic information is improved. As the criteria for the easiness and difficulty of estimating the elements to be removed, for example, criteria 1 to 3 described above can be used.

Learning by the factor extraction model learning unit 212 is repeated until a predetermined end condition is satisfied. For example, the factor extraction model learning unit 212 sets the input number of epochs (the number of times all of the acoustic feature vector sequence F _t and the correct label LA _{ta ,1} , . For example, 100 times), the learning is considered to be completed, and the factor extraction model _Mf at that time is output. The factor extraction model _Mf is stored in the factor extraction model storage unit 113 (step S113). As in the first embodiment, the reconstructed model _Mr and the removed element estimation model M _a are not used in subsequent processing. Others are the same as the first embodiment.

[Features of each embodiment]
As described above, in each embodiment, the factors necessary for estimating the paralinguistic information of speech are extracted from the factors representing speech, and the extracted factors are used to estimate the paralinguistic information of speech. Unnecessary factors for estimating are removed, making it possible to estimate paralinguistic information with higher accuracy than before. In the second embodiment, the paralinguistic information of speech is estimated using a factor sequence obtained by removing a plurality of removal elements unnecessary for estimating paralinguistic information from the factors representing speech. Estimation becomes possible.

Further, in each embodiment, when it becomes impossible to estimate a specific factor from the factor series extracted by the factor extraction model, the specific factor is completely removed from the factor series extracted by the factor extraction model. Then, the factor extraction model is learned so that it becomes difficult to estimate the removed elements unnecessary for estimating the paralinguistic information from the factor series extracted by the factor extraction model. This makes it possible to obtain a factor extraction model for extracting factors necessary for estimating paralinguistic information.

[Other modifications, etc.]
The invention is not limited to the embodiments described above. For example, in the above-described embodiments, models based on deep learning were exemplified as the factor extraction model, the reconstruction model, and the removed element estimation model. However, other estimation models may be used as the factor extraction model, the reconstruction model, and the removed element estimation model.

The various types of processing described above may not only be executed in chronological order according to the description, but may also be executed in parallel or individually according to the processing capacity of the device that executes the processing or as necessary. In addition, it goes without saying that appropriate modifications are possible without departing from the gist of the present invention.

Each of the above devices is, for example, a general-purpose or dedicated computer equipped with a processor (hardware processor) such as a CPU (central processing unit) and memories such as RAM (random-access memory) and ROM (read-only memory) is configured by executing a predetermined program. This computer may have a single processor and memory, or may have multiple processors and memories. This program may be installed in the computer, or may be recorded in ROM or the like in advance. Also, some or all of the processing units are configured using an electronic circuit that realizes processing functions without using a program, rather than an electronic circuit that realizes a functional configuration by reading a program like a CPU. may An electronic circuit that constitutes one device may include a plurality of CPUs.

When the above configuration is implemented by a computer, the processing contents of the functions that each device should have are described by a program. By executing this program on a computer, the above processing functions are realized on the computer. A program describing the contents of this processing can be recorded in a computer-readable recording medium. An example of a computer-readable recording medium is a non-transitory recording medium. Examples of such recording media are magnetic recording devices, optical discs, magneto-optical recording media, semiconductor memories, and the like.

The distribution of this program is carried out, for example, by selling, assigning, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Further, the program may be distributed by storing the program in the storage device of the server computer and transferring the program from the server computer to other computers via the network.

A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or the program transferred from the server computer once in its own storage device. When executing the process, this computer reads the program stored in its own storage device and executes the process according to the read program. As another form of execution of this program, the computer may directly read the program from a portable recording medium and execute processing according to the program. , may sequentially execute processing according to the received program. A configuration in which the above processing is executed by a so-called ASP (Application Service Provider) type service, which does not transfer the program from the server computer to this computer and realizes the processing function only by the execution instruction and result acquisition, is also possible. good.

At least a part of these processing functions may be realized by hardware instead of executing a predetermined program on a computer to realize the processing functions of the present apparatus.

The paralinguistic information estimated by the present invention can be used, for example, for dialogue control that takes into account the emotions of the other party in a spoken dialogue (such as changing the topic if the other party is angry), mental health diagnosis using speech (daily speech, etc.). recording and predicting mental health status from the frequency of sadness and anger voices).

11, 21 learning device 12 paralinguistic information estimation device

Claims (11)

a factor extracting unit that receives acoustic features of speech as an input, extracts and outputs factors necessary for estimating paralinguistic information of the speech from among factors that are elements representing each feature of the speech;
a paralinguistic information estimating unit that receives the factor extracted by the factor extracting unit as an input, estimates and outputs the paralinguistic information of the speech;
A paralinguistic information estimation device having Acoustic features of the training speech, and the correct label of the removed element, which is a single factor unnecessary for estimating the paralinguistic information of the training speech among the factors representing each feature of the training speech, or the learning correct labels of a plurality of removed elements that are unnecessary for estimating the paralinguistic information of the training speech among the factors that are elements representing each feature of the training speech, and
(1) a factor extraction model that takes as input acoustic feature values of speech, extracts and outputs factors necessary for estimating paralinguistic information of the speech from factors that are elements representing each feature of the speech; ) a reconstruction model that receives the factors output from the factor extraction model and the correct labels of the removed elements as inputs and reconstructs the acoustic feature of the speech; and (3) the factors output from the factor extraction model. A removed element estimation model for estimating a removed element, which is a single factor unnecessary for estimating the paralinguistic information of the speech, or an element representing each feature of the speech. and a factor extraction model learning unit that learns a removed element estimation model that estimates a plurality of removed elements that are a plurality of factors that are unnecessary for estimating the paralinguistic information of the speech among the factors that are
The factor extraction model learning unit
The factor extraction model is learned so that an error between the removal element estimated by the removal element estimation model and the correct label increases when the acoustic feature amount of the training speech is input to the factor extraction model. A learning device. The learning device of claim 2,
The factor extraction model learning unit
The reconstruction when the factors output from the factor extraction model when the acoustic features of the learning speech are input to the factor extraction model and the correct labels of the removed elements are input to the reconstruction model. The loss function value of the reconstruction model corresponding to the error between the acoustic feature quantity reconstructed by the model and the acoustic feature quantity of the training speech and the acoustic feature quantity of the training speech are input to the factor extraction model. removal element estimation corresponding to an error between the removal element estimated by the removal element estimation model and the correct label of the removal element when the factor output from the factor extraction model is input to the removal element estimation model when the factor is input to the removal element estimation model A learning device that learns the factor extraction model, the reconstruction model, and the removed element estimation model using a weighted sum of the loss function value of the model and the total loss function value. The learning device according to claim 2 or 3,
The factor extraction model learning unit learns the factor extraction model and the removed element estimation model using an error back propagation method,
Partial differentiation corresponding to the error between the removed element output from the removed element estimation model and the correct label when backpropagating from the removed element estimation model to the factor extraction model in the error backpropagation method A learning device, wherein values are back-propagated from the removed element estimation model to a gradient inversion layer, and the gradient inversion layer back-propagates a value obtained by multiplying the partial differential value by a negative constant to the factor extraction model. The learning device according to any one of claims 2 to 4,
The factor extraction model learning unit
The factor extraction model, the reconstruction model, and a plurality of removal element estimation models for estimating the plurality of removal elements, using as inputs the acoustic feature quantity of the training speech and the correct labels of the plurality of removal elements. and learn
A learning device that performs learning in order from a removed element estimation model for estimating a removed element that is easy to estimate among the removed element estimation models for estimating the plurality of removed elements. The learning device according to any one of claims 2 to 5,
The factor extraction model learning unit
Inputting the acoustic feature quantity of the training speech and the correct labels of the plurality of removal elements,
A weighted sum of the loss function value of the reconstructed model, the product of the removed element weight corresponding to each removed element and the loss function value of the removed element model corresponding to each removed element, as the overall loss function value , a learning device for learning the factor extraction model, the reconstruction model, and the removed element estimation model for estimating the plurality of removed elements. The learning device of claim 6,
The plurality of removal elements includes a first removal element and a second removal element that is easier to estimate from the acoustic feature amount of speech than the first removal element,
The learning device, wherein a removal factor weight value corresponding to the first removal factor is greater than a removal factor weight value corresponding to the second removal factor. a factor extraction step of extracting and outputting the factors necessary for estimating the paralinguistic information of the speech from among the factors representing the features of the speech, with the acoustic features of the speech as input;
a paralinguistic information estimating step of estimating and outputting the paralinguistic information of the speech using the factors extracted in the factor extracting step as input;
A method for estimating paralinguistic information. Acoustic features of the training speech, and the correct label of the removed element, which is a single factor unnecessary for estimating the paralinguistic information of the training speech among the factors representing each feature of the training speech, or the learning correct labels of a plurality of removed elements that are unnecessary for estimating the paralinguistic information of the training speech among the factors that are elements representing each feature of the training speech, and
(1) a factor extraction model that takes as input acoustic feature values of speech, extracts and outputs factors necessary for estimating paralinguistic information of the speech from factors that are elements representing each feature of the speech; ) a reconstruction model that receives the factors output from the factor extraction model and the correct labels of the removed elements as inputs and reconstructs the acoustic feature of the speech; and (3) the factors output from the factor extraction model. A removed element estimation model for estimating a removed element, which is a single factor unnecessary for estimating the paralinguistic information of the speech, or an element representing each feature of the speech. and a factor extraction model learning step for learning a removed element estimation model for estimating a plurality of removed elements that are a plurality of factors unnecessary for estimating the paralinguistic information of the speech among the factors that are
The factor extraction model learning step includes:
The factor extraction model is learned so that an error between the removal element estimated by the removal element estimation model and the correct label increases when the acoustic feature amount of the training speech is input to the factor extraction model. A learning method that includes steps to A program for causing a computer to function as the paralinguistic information estimation device according to claim 1 . A program for causing a computer to function as the learning device according to any one of claims 2 to 7.

Images