CN102789594B - Voice generation method based on DIVA neural network model - Google Patents

Abstract

The invention discloses a voice production method based on a DIVA neural network model, which includes voice sample extraction, voice sample classification and learning, voice output and corrected output voice, and the voice sample classification and learning adopts an adaptive growing neural network (AGNN ) to realize the classification learning of speech samples, use the obtained speech formant frequency to further calculate the number of candidate neurons in the input layer, and then determine the hidden layer neurons according to the candidate neurons in the input layer, and finally obtain the output value of AGNN, and according to The output value is used to determine the phoneme, and the neural network with the above structure has high training accuracy and fast learning speed.

Description

A Speech Generation Method Based on DIVA Neural Network Model

technical field

The invention relates to a speech generation method, in particular to a speech generation method based on a DIVA neural network model.

Background technique

With the development of artificial intelligence, people's research in this field continues to deepen. The control of voice generation and acquisition similar to human voice is an urgent problem to be solved in the robot voice system. Speech generation and acquisition is a complex cognitive process involving many parts of the brain. This process includes a hierarchical structure extending from the expression of sentences or phrases organized according to syntax and grammar to the production of phonemes. The interaction of sensory and motor areas builds a corresponding neural network model. At present, the DIVA (Directions Into Velocities of Articulators) model is a mathematical model related to the processing of speech generation and post-acquisition description. It is mainly used to simulate and describe the relevant functions in the brain related to speech generation and speech understanding. It can also be said that it is an adaptive neural network model used to control the movement of the simulated vocal tract in order to generate words, syllables or phonemes. Of today's truly biologically meaningful neural network models of speech generation and acquisition, the DIVA model is relatively the most thoroughly defined and tested, and is the only one that applies a pseudo-inverse control scheme.

People's demand for a unified computational model of human language ability drives the development of the DIVA model. Since this model was first proposed by Guenther of the Speech Laboratory of MIT University in 1994, it has been continuously updated, improved and improved in recent years. The DIVA system consists of a speech channel module, a cochlea module, an auditory cortex model module, an auditory cortex category perception module, a speech cell set module, a motor cortex module, a vocal tract module, a somatosensory cortex module, a sensory module and a sensory channel module.

Through the analysis of the DIVA model, we can find that the classification method used in the category perception module of its auditory cortex is RBF. The RBF neural network is very dependent on samples. For a specific research problem, how to determine the appropriate number of hidden layer nodes, there is no general and effective algorithm or theorem. People rely more on experience and trial and error to determine the size of the network. This trial and error method is very cumbersome and it is difficult to find a suitable structure. The number of nodes in the hidden layer of the network has a great influence on the convergence speed, accuracy and generalization ability of the network. If there are too many nodes in the hidden layer, although the training can be completed, it will affect the convergence speed, and there may be over-learning; if there are too few nodes in the hidden layer, the network cannot fully learn and cannot meet the requirements of training accuracy. In addition, the training time of RBF neural network is not fast enough.

Contents of the invention

The object of the present invention is to provide a voice generation method based on the DIVA neural network model with high pronunciation accuracy and fast learning speed.

The technical solution that realizes the object of the present invention is: a kind of voice generation method based on DIVA neural network model, including voice sample extraction, voice sample classification and learning, voice output and corrected output voice, described voice sample classification and learning adopt adaptive The growing neural network (AGNN) realizes the classification and learning of speech samples, specifically:

Step 1, convert the speech formant frequency of extraction into matrix form by Jacobian determinant, the dimension of the eigenvector of this matrix is the number m of input layer candidate neurons; calculate the fitness function value of input layer candidate neurons and Arrange the candidate neurons in the order of increasing fitness function value, the corresponding list of candidate neuron fitness function values in the input layer is S={S _i1 ≤S _i2 ≤...≤S _im }, and put the candidate neurons in the corresponding order Elements are placed in the list X, X=(x ₁ ,...,x _m ), the calculation formula of the fitness function is:

y _i is the actual output value, is the target value, n is the number of samples in the data set and n is a natural number;

Step 2: The number of neurons in the initial hidden layer is r=0 and C ₀ =S _i1 , where C ₀ is the fitness function value when the number of neurons in the hidden layer is r=0;

Step 3. Set r=r+1 and p=r+1, where r is the rth hidden layer candidate neuron, and generate a hidden layer candidate neuron with p inputs;

Step 4. If r>1, connect the hidden layer candidate neuron to all previous hidden layer neurons and the input node _x1 ; otherwise, connect the hidden layer candidate neuron only to the input node _x1 ;

Step 5. Set the initial value of the position h of the element in the next set X that needs to be connected to the newly added hidden layer candidate neuron to 2, where 2≤h≤m, m and h are positive integers; set The Pth input of this hidden layer candidate neuron is connected to the input node whose position is h in the list X;

Step 6. Train the candidate neurons of the hidden _layer and calculate its fitness function value C _r . If C _r ≥ C _{r- 1} , perform step 7; connected to the network as the rth hidden layer neuron, and then return to step 3 to step 6, until the mth input layer neuron connected to the network does not meet this condition;

Step 7: Set h=h+1, retrain the hidden layer candidate neurons until h=m, if C _r <C _r-1 is still not satisfied, then end the training, the hidden layer candidate neurons have nothing to do with classification , discard this hidden layer candidate neuron, and use the previous hidden layer neuron of this hidden layer candidate neuron as the output layer;

Step 8: Determine the phoneme according to the output value of the output layer.

Further, in the speech generation method based on the DIVA neural network model of the present invention, the hidden layer candidate neuron is trained in step six and its fitness function value C _r is calculated, specifically:

(1), divide the data set formed by normalizing the voice formant frequency into a training set, a verification set and a test set. The number of samples of the training set and the verification set divided here are respectively n _A , n _B , and the division basis is :n _A =n _B ;

(2) According to the three divided sets, use the following formula to calculate the fitness function value C _r of the candidate neurons in the hidden layer, i=1,...,n _B , n _B is the number of samples in the validation set, Among them, y _B ∈ Y _B , Y _B is the target vector in the verification set, U _B is the input of the verification set to the hidden layer neurons and U _B is a matrix of p×1 vectors, W ^k-1 is the weight vector, and k is The number of iterations, the value range is k=0,1,2,3,…,n, where n is a positive integer.

Further, in the speech generation method based on the DIVA neural network model of the present invention, in the eighth step, the phoneme is determined according to the output value of the output layer, specifically: the output value of the output layer is a value between 0 and 1, and The phoneme corresponding to the output value of the AGNN neural network is determined according to the range value corresponding to each phoneme in the DIVA neural network model.

Compared with the prior art, the present invention has significant advantages: since the self-adaptive growth type neural network model learns from an input node, adjusts the weight of neurons according to external rules, and gradually adds new input nodes and new hidden Neurons. Therefore, the constructed AGNN is a narrow and deep network with close to the minimum number of input neurons, hidden neurons and network connections, which can effectively prevent over-fitting of the network, and the network has low computational cost and fast learning speed; The RBF network used in the original DIVA model has an average classification accuracy of 80% of the samples, while the AGNN can reach more than 90%. For the learning samples of general difficulty, the original model classifies and learns and generates speech. The process takes 10s-13s, while using Under the same conditions, the improved system of the AGNN model takes only 8s-10s for the general difficulty learning sample model to classify and learn and generate speech, which is 2-3s faster than that under the same conditions. For learning samples with medium difficulty and above, the performance of the improved system using the AGNN model is better under the same circumstances. Compared with the model before the improvement, the time is 4-5s faster, and the classification accuracy of the samples is reduced by the original system. to 70%-75%, while the improved system using the AGNN model can still maintain a high accuracy rate of 90% under the same conditions; it can be seen that applying the adaptive growth neural network model to the DIVA model makes the pronunciation accuracy of the model more accurate Higher and faster to learn.

Description of drawings

Fig. 1 is a flowchart of the present invention;

Fig. 2 is the structural block diagram of DIVA neural network model;

Fig. 3 is the AGNN neural network structure schematic diagram that is used for classification in the embodiment;

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings.

As shown in Fig. 1, a kind of speech generation method based on DIVA neural network model of the present invention comprises speech sample extraction, speech sample classification and learning, speech output and revised output speech, it is characterized in that, described speech sample classification and learning adopt The adaptive growing neural network (AGNN) realizes the classification and learning of speech samples, specifically:

Step 1, convert the speech formant frequency of extraction into matrix form by Jacobian determinant, the dimension of the eigenvector of this matrix is the number m of input layer candidate neurons; calculate the fitness function value of input layer candidate neurons and Arrange the candidate neurons in the order of increasing fitness function value, the corresponding list of candidate neuron fitness function values in the input layer is S={S _i1 ≤S _i2 ≤...≤S _im }, and put the candidate neurons in the corresponding order Elements are placed in the list X, X=(x ₁ ,…,x _m ), the calculation formula of the fitness function is: y _i is the actual output value, is the target value, n is the number of samples in the data set and n is a natural number;

Step 2: The number of neurons in the initial hidden layer is r=0 and C ₀ =S _i1 , where C ₀ is the fitness function value when the number of neurons in the hidden layer is r=0;

Step 3. Set r=r+1 and p=r+1, where r is the rth hidden layer candidate neuron, and generate a hidden layer candidate neuron with p inputs;

Step 4. If r>1, connect the hidden layer candidate neuron to all previous hidden layer neurons and the input node _x1 ; otherwise, connect the hidden layer candidate neuron only to the input node _x1 ;

Step 5. Set the initial value of the position h of the element in the next set X that needs to be connected to the newly added hidden layer candidate neuron to 2, where 2≤h≤m, m and h are positive integers; set The Pth input of this hidden layer candidate neuron is connected to the input node whose position is h in the list X;

Step 6. Train the candidate neurons of the hidden _layer and calculate its fitness function value C _r . If C _r ≥ C _{r- 1} , perform step 7; connected to the network as the rth hidden layer neuron, and then return to step 3 to step 6, until the mth input layer neuron connected to the network does not meet this condition; where the fitness function value C _r , specifically for:

(1), divide the data set formed by normalizing the voice formant frequency into a training set, a verification set and a test set. The number of samples of the training set and the verification set divided here are respectively n _A , n _B , and the division basis is :n _A =n _B ;

(2) According to the three divided sets, use the following formula to calculate the fitness function value C _r of the candidate neurons in the hidden layer, i=1,...,n _B , n _B is the number of samples in the validation set, Among them, y _B ∈ Y _B , Y _B is the target vector in the verification set, U _B is the input of the verification set to the hidden layer neurons and U _B is a matrix of p×1 vectors, W ^k-1 is the weight vector, and k is The number of iterations, the value range is k=0,1,2,3,...,n, where n is a positive integer, the higher the training accuracy is, the larger the number of iterations k is.

Step 7: Set h=h+1, retrain the hidden layer candidate neurons until h=m, if C _r <C _r-1 is still not satisfied, then end the training, the hidden layer candidate neurons have nothing to do with classification , discard this hidden layer candidate neuron, and use the previous hidden layer neuron of this hidden layer candidate neuron as the output layer;

Step 8, determine the phoneme according to the output value of the output layer, the output value of the output layer is a value in the interval of 0 to 1, and determine the AGNN neural network output value according to the corresponding range value of each phoneme in the DIVA neural network model the corresponding phonemes.

Example

As shown in FIG. 2 , in this embodiment, the voice of the pronunciation device such as a microphone is firstly collected and passed through the voice channel module with a given delay, and the formant frequency of the voice is sent to the cochlear module in the form of a vector. The cochlear module computes the cochlear representation (spectrum) of this speech and sends the formant frequencies to the auditory cortex module. The auditory cortex transmits the speech represented by the formant frequency from the cochlear module to the auditory cortical category perception module. After the auditory cortex category perception module receives the speech, it divides it into the basic unit of speech - phoneme, and the phoneme target of the initialization output reaches the auditory cortex and the somatosensory cortex module respectively through the speech cell set module to form the auditory and somatosensory results respectively. This module recognizes speech fragments by comparing the speech fragments from the auditory cortex module with the stored phoneme representations, where each phoneme is represented as a value range between 0-1, stored in the speech cell set module, specifically the recognition process For: the auditory cortex category perception module matches the divided phonemes (i.e. the output value of AGNN) one by one with the phoneme representations in the speech cell set, if no matching phoneme representation is found in the speech cell set, that is, the phoneme has not been identified After training and learning, the speech cell set module will create a new phoneme representation in a specific area to represent the current phoneme. There is a one-to-one relationship between phoneme targets and speech cell assemblies output by class-aware modules in auditory cortex. Afterwards, the speech cell assembly module starts the generation of phoneme fragments, and sends the index of the phoneme target to be generated to the motor cortex, auditory cortex and somatosensory cortex modules. The motor cortex sends control commands to the vocal tract module after receiving the phoneme target index from the speech cell set module. The calculated auditory effect and parameter configuration are transmitted to the cochlear module and the sensory module through the speech channel and the sensory channel respectively to form feedback. After receiving the channel configuration information sent by the sensory channel in the form of a vector, the sensory module calculates the somatosensory results related to the channel configuration and sends them to the somatosensory cortex module. The somatosensory module then computes the difference in cortical representations between the input somatosensory and the somatosensory target, and passes the somatosensory error to the motor cortex module for correction of the generated speech. After the cochlear module receives the formant frequency of the voice expressed in vector form from the voice generated by the vocal tract module through the voice channel, it transmits it to the auditory cortex module, and the auditory cortex module calculates the voice represented by the cortex and other The difference between the target speech and pass the error to the motor cortex module to modify the generated speech.

As shown in the table below, the numerical ranges corresponding to the 29 phoneme representations stored in the speech cell set module in the existing DIVA neural network model. The classification result of AGNN is a numerical value, and the obtained numerical values represent different phonemes (the obtained numerical values fall in different numerical intervals to represent a specific phoneme).

As shown in Figure 3, the learning efficiency η = 1.9, Δ = 0.0015, and the selection of the initial weight conforms to the normal distribution.

According to the input data set X, calculate the dimension of its feature vector, that is, the number of candidate neurons in the input layer m=8, according to the formula: where y _i is the actual output value, is the target value, and n is the number of samples in the data set. Calculate the fitness function value of each element in the input data set X, arrange them in the order of increasing fitness function, and select the first 8 neurons as candidate neurons, which are x ₈ , x ₅ , x ₁₂ , x ₁₆ , x ₂₄ , x ₂₇ , x ₁₉ , x ₂₃ , where the fitness function value corresponding to the first input neuron x ₈ is the smallest and recorded as C ₀ .

Add a hidden layer candidate neuron z ₁ , which has 2 inputs. Connect its two inputs to the input layer candidate neurons x ₈ and x ₅ , train the hidden layer candidate neurons and then calculate the fitness function C ₁ of z ₁ , and compare C ₁ with C ₀ at this time, there is C ₁ <C ₀ , then add z ₁ to the network as the first hidden layer neuron. Add another hidden layer candidate neuron z ₂ , which has 3 inputs. Connect its first 2 inputs to the previous hidden layer neurons z ₁ and x ₈ , connect the third input to x ₅ , train this candidate hidden layer neuron and calculate its fitness function C ₂ , connect C ₂ with C ₁ compares with C ₂ <C ₁ , and z ₂ is added to the network as the second hidden layer neuron. Add candidate neuron z ₃ of the hidden layer, and it has 4 inputs. Connect the first 3 inputs to hidden layer neurons z ₁ and z ₂ and input layer neuron x ₈ , and connect the fourth input to x ₅ , train this candidate neuron and calculate its fitness function but at this time calculate The fitness function value is less than C ₂ , connect the fourth input to x ₁₂ , train the candidate neuron and calculate the fitness function value C ₃ at this time, if C ₃ <C ₂ , add z ₃ to the network as the third hidden layer neuron. Add z ₄ to the network as a hidden layer candidate neuron with 5 inputs, connect the first 4 inputs to z ₁ ~ z ₃ and x ₈ , connect the fifth input to x ₁₂ , and train this candidate neuron and calculate its fitness function but less than C ₃ , connect the 5th input to x ₁₆ , train this candidate neuron and calculate its fitness function C ₄ , since C ₄ < C ₃ , add z ₄ to In the network as the fourth hidden layer neuron. Then add z ₅ to the network as a hidden layer candidate neuron, which has 6 inputs, connect the first 5 inputs to z ₁ ~ z ₄ and x ₈ , connect the sixth input to x ₁₆ , Train the candidate neuron and calculate its fitness function C ₅ , since C ₅ < C ₄ , add z ₅ into the network as the fifth hidden layer neuron. Continue to add z ₆ as a hidden layer candidate neuron, which has 7 inputs, connect the first 6 inputs to z ₁ ~ z ₅ and x ₈ , connect the 7th input to x ₁₆ , and train this candidate neuron And calculate its fitness function but less than C ₅ , connect the 7th input to x ₂₄ , train the candidate neuron and calculate its fitness function C ₆ , since C ₆ < C ₅ , add z ₆ to the network as the sixth hidden layer neuron. Then connect z ₇ to the network as a hidden layer candidate neuron, then there are 8 inputs, connect the first 7 inputs to z ₁ ~ z ₆ and x ₈ , connect the 8th input to x ₂₄ , and train This candidate neuron calculates its fitness function C ₇ , C ₇ <C ₆ , and adds z7 into the network as the seventh hidden layer neuron. Then add z ₈ to the network as a hidden layer candidate neuron, which has 9 inputs, connect the first 8 inputs to z ₁ ~ z ₇ and x ₈ , and connect the ninth input to x ₂₄ , Train this hidden layer candidate neuron and calculate its fitness function but less than C ₇ , connect the 9th input to x ₂₇ , train this hidden layer candidate neuron and calculate its fitness function C ₈ , because C ₈ <C ₇ , add z ₈ to the network as the eighth hidden layer neuron. Next, add z ₉ to the network as a hidden layer candidate neuron, which has 10 inputs, connect the first 9 inputs to z ₁ ~ z ₈ and x ₈ , connect the tenth input to x ₂₇ , and train the hidden layer layer candidate neuron and calculate its fitness function C ₉ , since C ₉ <C ₈ , add z ₉ to the network as the ninth hidden layer neuron. Continue to add z ₁₀ as a hidden layer candidate neuron, it has 11 inputs, connect the first 10 inputs to z ₁ ~ z ₉ and x ₈ , connect the 11th input to x ₂₇ , and train this hidden layer candidate Neuron and calculate its fitness function but less than C ₉ , then connect the 11th input to x ₁₉ , train this hidden layer candidate neuron and calculate its fitness function C ₁₀ , because C ₁₀ <C ₉ , put z ₁₀ is added to the network as the 10th hidden layer neuron. Then add z ₁₁ as a hidden layer candidate neuron, with 12 inputs, connect the first 11 inputs to z ₁ ~ z ₁₀ and upper x ₈ , connect the 12th input to x ₁₉ , and train this hidden layer candidate Neuron and calculate its fitness function C ₁₁ , because C ₁₁ <C ₁₀ , add z ₁₁ to the network. Add z ₁₂ as a hidden layer candidate neuron with 13 inputs, connect its first 12 inputs to z ₁ ~ z ₁₁ and upper x ₈ , connect the 13th input to x ₁₉ , and train this hidden layer candidate Neuron and calculate its fitness function but less than C ₁₁ , then connect the _13th input to x ₂₃ , train this hidden layer candidate neuron and calculate its fitness function C ₁₂ , because C ₁₂ <C ₁₁ , so put z ₁₂ is added to the network as a hidden layer neuron. Add z ₁₃ as a hidden layer candidate neuron with 14 inputs, connect its first 13 inputs to z ₁ ~ z ₁₂ and upper x ₈ , connect the 14th input to x ₂₃ , and train this hidden layer candidate neuron and calculate its fitness function C ₁₃ <C ₁₂ , and add z ₁₃ into the network as a hidden layer neuron. Add z ₁₄ as a hidden layer candidate neuron with 15 inputs, connect the first 14 inputs to z ₁ ~ z ₁₃ and upper x ₈ , connect the 15th input to x ₂₃ , and train this hidden layer candidate neuron and calculate its fitness function but it is less than C ₁₃ . At this time, there are no candidate neurons to be connected, so z ₁₄ is discarded, and z ₁₃ is used as the output neuron.

The network selects 8 input features, 12 hidden layer neurons and 1 output neuron. The neurons of the first hidden layer are connected to input nodes x ₈ , x ₅ . The input of the output neuron is connected to the output z ₁ ~ z ₁₂ of the hidden layer neuron and the input nodes x ₈ , x ₂₃ .

Claims (3)

1. A method for generating speech based on the DIVA neural network model, comprising speech sample extraction, speech sample classification and learning, speech output and correction output speech, characterized in that said speech sample classification and learning adopts an adaptive growth type neural network Realize the classification and learning of speech samples, specifically: Step 1, convert the speech formant frequency of extraction into matrix form by Jacobian determinant, the dimension of the eigenvector of this matrix is the number m of input layer candidate neurons; calculate the fitness function value of input layer candidate neurons and Arrange the candidate neurons in the order of increasing fitness function value, the corresponding list of the fitness function value of the input layer candidate neurons is S={S _i1 ≤S _i2 ≤...≤S _im }, and put the candidate neurons in the corresponding order Elements are placed in the list X, X=(x ₁ ,...,x _m ), the calculation formula of the fitness function is: y _i is the actual output value, is the target value, n is the number of samples in the data set and n is a natural number; Step 2: The number of neurons in the initial hidden layer is r=0 and C ₀ =S _i1 , where C ₀ is the fitness function value when the number of neurons in the hidden layer is r=0; Step 3, set r=r+1 and p=r+1, where r is the rth hidden layer candidate neuron, and generate a hidden layer candidate neuron with p inputs; Step 4. If r>1, connect the hidden layer candidate neuron to all previous hidden layer neurons and the input node _x1 ; otherwise, connect the hidden layer candidate neuron only to the input node _x1 ; Step 5. Set the initial value of the position h of the element in the next set X that needs to be connected to the newly added hidden layer candidate neuron to 2, where 2≤h≤m, m and h are positive integers; set The Pth input of this hidden layer candidate neuron is connected to the input node whose position is h in the list X; Step 6. Train the candidate neurons of the hidden _layer and calculate its fitness function value C _r . If C _r ≥ C _{r- 1} , perform step 7; connected to the network as the rth hidden layer neuron, and then return to step 3 to step 6, until the mth input layer neuron connected to the network does not meet this condition; Step 7: Set h=h+1, retrain the hidden layer candidate neurons until h=m, if C _r <C _r-1 is still not satisfied, then end the training, the hidden layer candidate neurons have nothing to do with classification , discard this hidden layer candidate neuron, and use the previous hidden layer neuron of this hidden layer candidate neuron as the output layer; Step 8: Determine the phoneme according to the output value of the output layer. 2. the speech generation method based on DIVA neural network model according to claim 1, is characterized in that: in step 6, trains this hidden layer candidate neuron and calculates its fitness function value C _r , specifically: (1) Divide the data set formed by the normalization of speech formant frequencies into training set, verification set and test set. The number of samples in the training set and verification set divided here are respectively n _A and n _B . The division basis is as follows: n _A = n _B ; (2) According to the three divided sets, use the following formula to calculate the fitness function value Cr of the candidate neurons in the hidden layer, $C_r={\mathrm{E.}}_B\left(k\right)={\left(\underset{i}{\mathrm{\&Sigma;}}{\left(e_{\mathrm{Bi}}^k\right)}^2\right)}^{1/2},i=1,\&Center\; Dot;\&CenterDot;\&CenterDot;,{\mathrm{no}}_B,$ n _B is the number of samples in the validation set, Among them, y _B ∈ Y _B , Y _B is the target vector in the verification set, U _B is the input of the verification set to the hidden layer neurons and U _B is a matrix of p×1 vectors, W ^k-1 is the weight vector, and k is The number of iterations, the value range is k=0,1,2,3,...,n, where n is a positive integer. 3. the speech generation method based on DIVA neural network model according to claim 1, is characterized in that: in described step 8, determine phoneme according to the output numerical value of output layer, specifically: the output numerical value of described output layer is 0 to 1, and determine the phoneme corresponding to the output value of the AGNN neural network according to the range value corresponding to each phoneme in the DIVA neural network model.