JPH01243169A - System for learning and preparing pattern - Google Patents

Abstract

PURPOSE:To converge the learning of analog information with a high accuracy and simultaneously, at a high speed by multiplexing a neutral network, obtaining a weighted mean, making it into a final output and causing a weight coefficient between an input layer and an intermediate layer or between the intermediate layer and an output layer to be random. CONSTITUTION:The neutral network is composed of an input layer 10, an intermediate layer 12, an output layer 14, and a final output layer 16, and the intermediate layer 12 and output layer 14 are made into multistages. The weight coefficient between the input layer 10 and intermediate layer 12 or between the intermediate layer 12 and output layer 14 is made random according to a necessity, the learning is independently executed in respective stages in the multiplexed intermediate layer 12 and output layer 14, the weighted mean of the outputs of respective stages of the output layer 14 is obtained in the final output layer 16, and it is made into a final output OT. Thus, the efficient learning of the data of a read world can be executed, and the convergency can be made high-speed.

Description

[Detailed Description of the Invention] [Summary of the Invention] Regarding a pattern learning/generation method, the present invention aims to enable efficient learning of real-world data and to quickly converge. , using a neural network consisting of a multi-stage intermediate layer, a multi-stage output layer, and a final output layer, the input sequence is added to the input layer, and the final output layer takes a weighted average of each output of the multi-stage output layer and finalizes it. As an output, either the weighting coefficient between the input layer and the intermediate layer or the weighting coefficient between the intermediate layer and the output layer is randomized, and each stage is configured to learn independently.

[Industrial application field]

The present invention relates to a pattern learning/generation method, and is effective for synthetic speech generation, speech recognition, etc., but is widely applicable to general patterns including time-series data.

To generate synthesized speech, there is a Percoll system for registered messages, which extracts the features of the speech, compresses the information from the extracted parameters, and outputs the speech. A so-called rule synthesis method is used to generate synthesized speech of arbitrary sentences (character strings) that are not registered. The rule synthesis method creates rules for how humans speak, and uses these rules to convert character strings into synthesized speech. Although the rule synthesis method can generate synthesized speech even in a relatively small-scale system, the rules become complex when trying to create synthesized speech of slightly higher quality, and finding general rules is a fundamental problem in speech. It is difficult to create a natural synthesized voice because it is closely related to the points, and the voice becomes mechanical and non-human.

By introducing a learning system using a neural network (a network that imitates the human brain), it is possible to create synthetic speech that is more natural than the rule synthesis method. Neural networks distribute parameters in space and memorize them. They increase accuracy by arranging many low-accuracy parameters in parallel, and use a method of memorizing rules by learning rather than instructing them. ■
This is learning of C0 symbol strings and symbol strings, and learning can be done depending on whether or not a symbol stands in a predetermined position, and even fairly rough methods are often successful.

A system that automatically generates natural synthetic speech from text (character strings) using a neural network consists of a part that converts the text into a phoneme sequence, and a part that generates synthetic speech from the phoneme sequence.

For the former part, which converts character strings into phoneme sequences, it is sufficient to learn the character strings (state 110), but for the part to generate synthesized speech from the phoneme sequences, it is necessary to learn highly accurate analog data. This has been difficult until now. The present invention relates to a Bakun learning/generation method that is also effective when used for outputting sound source/vocal tract parameters from phoneme sequences.

[Conventional technology]

As shown in Fig. 5, the analog neuron element used in the neural network consists of a resistance for weighting coefficient Wij, an adder Σ, and an output function f4, and has inputs X, threshold value θ,
In response to this, an output Zi and an internal variable y are generated as expressed by the following equation.

7, −, LW i j X i+θ,
・・・・・・(1)x−f;<yi) (==+,
z,...', )--C2) Here, ■ is the number of input elements, and J is the number of output elements. The neural network model consists of an input layer and a middle layer (hidden layer) shown in Figure 6.
A model with a three-layer structure consisting of a hidden layer (hidden layer) and an output layer is used. Weighting is performed between the input layer and the hidden layer, and between the hidden layer and the output layer.

Set any point on the input layer to xh□, tp (1≦is≦Is.

1≦ip≦IP, IS is the number of series in the human power layer, I
P is the number of elements one of the series has), and an arbitrary point on the intermediate layer is Zl'J (1≦j≦J, J is the number of elements in the intermediate layer). At this time, the input/output relationship is zhj=)l+, ('HΣW'is+ ip+ j
X'is+ip10'j)=-(3). Similarly, the human output relationship from the middle layer to the output layer is Z', -fok(t'rbX'j+θ',
−・−・−(4) However, x O,=z h , and 1≦ and≦K (K is the number of elements in the output layer).

The backpropagation learning algorithm used in the prior art is shown below. However, Z)′j+z
o, together as Zj, and xhis+ip+xl″
j will be collectively written as Xl, and f'j+''k will be collectively written as f.

The target input (desired output; target) is set to t, and the weight Wij is corrected (the amount of correction is set to ΔWij) so that the sum of squares of the error between the target value tj and the actual output zJ is minimized. For simplicity, the value of the threshold value θ is assumed to be 0.

In other words, the output error E is E=1/2Σ(ti Z=)"
...... (5) (1/2 makes the coefficient 2 disappear when taking the differentiation later),
A learning method (steepest descent method; weight correction amount Δ is determined so that the slope of the error is the steepest) is used based on the following equation.

ΔW 1JQC-θE/θW0, ...
(6) Here, from equation (5), the following equation holds true.

θE/θzJ= (t, + Z;) ・・
...(7) Now, θE/c? Wi j−θE/ey, −θyJaw
=j ・(8) Therefore, from equation (1), θyJ/θWij=θ/θwHJ−'1w,, x y
= x t ・= (9). Next, δ, = -θE/θyj ...0
0), then from equation (8) and equation (9) - θE/θwIj−δ, χ, ...
・θ1), and from this and the assumption of equation (6), ΔWij−αδ, χi ...
・02). Next, δj is calculated. From 2〇〇), δ4−−θE/θzJ・θzzj/θyJ, so
Considering Equation (7)1 and Equation (2), 1, the amount of backward propagation of error in the output layer δ06 is δ'h-(tk-Z'k)f'h(y'k)-・
- Becomes an OS. Further, the backward propagation amount δh4 of errors in layers other than the output layer (intermediate layer) is as follows.

δhJ=f′= (y′′;) :, δo, W jl
+ (here K is the number of elements in the output layer) ・・・・・・04
) In particular, the output function f (・) is transformed into a logistic curve z,
+-1/(1+exp('/i))...
...Set as θω (Z,, is 1/2 when y is O, becomes 1 when y increases positively, and becomes 0 when it also increases negatively)
) and f'= (y=) −ZJ(1−z=) −=
06), equation 03) and equation (14) each have δz
= (tkz0+,) Z'h (1z'h) ・Old・
・071δhj=z';(1z', 1) 'E
δ'kw'4. ...HQ8). In these, from 202), the weight w between the intermediate layer and the output layer
05. The correction amount ΔW'j is 6w0Jh (n+1)
-αδ0kxoJ...-(+!]) or 6w0Jk(n+1) = (zδ0, x O, +
βΔw', k(1)...QΦ. On the other hand, the weight between the input layer and the hidden layer is
The correction amount ΔWhij of j is Δw12J(n+1) −α δ 11 x 6
□ ・・・・・・ (2a)
Or Δwhz7(n+1) −o:δh jX h , +8
6w h, i (n)...Qa. From the above development, in the conventional learning method, each output value is calculated from the input layer to the intermediate layer to the output layer using the model shown in the figure, and then 207), 010 and the formula QΦ,
Pattern learning is performed by modifying the weights using Q. In other words, in learning by bank propagation, the learning data is input and the results are output (
forward; feedforward), changing the strength of the connections to reduce the error in the result (backward: feedback)
Then, we manually input the training data again and repeat this process until convergence.

[Problem to be solved by the invention]

An automatic synthetic speech generation system using a neural network can generate more natural synthesized speech than the rule synthesis method. The reason is that the rule synthesis method requires all the characteristics of phonological changes to be described as rules, and this is difficult, whereas when introducing a learning method using a neural network, it is possible to This is because it becomes possible to learn the obtained target outputs as a set, and it becomes possible to incorporate a natural phonological environment into the network. However, with the learning methods that have been proposed to date using neural networks, it is difficult to learn anything other than a specific phonological environment. This is because when using conventional technology, it is difficult to learn data other than data that is orthogonal to each other, the learning results up to now are often destroyed during learning, and the convergence of learning is extremely poor. by.

It is an object of the present invention to improve this point, to enable efficient learning of real-world data (data that is not necessarily orthogonal), and to enable quick convergence.

[Means to solve the problem]

As shown in FIG.
6, and the intermediate layer and output layer are multi-staged (multiplexed).

The input layer 10 has one stage and has a point (element) of I=ISXIP inside. Here, Is is the number of elements in a sequence when the input is a sequence, and IP is the number of elements in a column (vector) that one point in the sequence has.

The number of stages of the intermediate layer 12 is M, and here, the middle layer is H8, the top layer is H-(M-11/□, and the bottom layer is H(N-
11/□ Closed. The output layer 14 also has M stages, and is similarly labeled here. The final output layer 16 has one stage.

All points in the input layer are connected to all points in the intermediate layer of all stages, and from the intermediate layer to the output layer, all points in the intermediate layer of the relevant stage are connected to all points in the output layer. Connect to the point and do not connect to other stages. A connection is made from the output layer to the final output layer to take a weighted average based on a certain rule.

[Effect]

In this neural network, input layer 10 and hidden layer 1
2 or between the intermediate layer 12 and the output layer 14, as necessary. In addition, each stage of the multiplexed intermediate layer 12 and output layer 14 is trained independently,
The final output layer 16 takes a weighted average of the outputs of each stage of the output layer 14 and sets this as the final output 07. Next, the learning rules are listed.

■) In the conventional method, the input layer consists of a sequence of a certain length (IS), and the output layer consists of a sequence of data (feature vector ) as the output, and the input series and output series are trained as a set. In contrast, in the present invention, the network is multi-staged, the output strings are increased by the number of stages, and the points of the input string corresponding to the output strings of each stage are: It is determined by selecting an arbitrary point in the input series.

■) Next, when performing learning, randomize one of the weighting coefficients between the input layer and the hidden layer, or the weighting coefficient between the hidden layer and the output layer, as necessary. For example, normal random values are given to the weighting coefficients), and learning is performed independently between each stage. In this case, the randomized weighting coefficients between each stage are not the same set, but are also random. Furthermore, in the final output layer, a weighted average of the output layers of each stage is taken.

■) Also, in the learning process, the amount of backward propagation of error in the output layer, 60k, is set to 60 = (th z'k) K (・) ・・
...(23), and the backward propagation amount δl'j of the error in the intermediate layer is 66, = K(・)'f, δ'Wjh
・==−(24). Here, K(.) is a predetermined function. 07) As is clear from the OI formula, δ has a singular point when Z is 0 and 1, and the value becomes 0. If δ falls to 0, it will no longer float and no corrections will be made. The function K(.) saves this situation. Furthermore, regarding the modification of the weighting coefficient, the modification amount ΔW'jk of the weight W'jk between the intermediate layer and the output layer is 6w0Jk(n+
1) =αδ0kt, (i+βΔw', +k(n) +
M (・) −−(25), and the correction amount ΔWhij of the weight Whij between the input layer and the hidden layer is Δwh+=(n+1) =aδ', tL(-)+βΔW
'+, (n) +M (1-.= (26)).

Here, when L(・)1M(・) is a predetermined function (here, ■), when the coefficient is randomized, ΔW
'jk or ΔWh, either one of j is 0). However, when applying the previous section I) or the previous section 2), the error propagation law and the weight correction amount are not limited to these.

Network multiplexing and final weighted averaging can improve spatial resolution without sacrificing temporal resolution, and randomization can reduce statistical bias of weighting coefficients (
If this occurs, the learning results up to now may be destroyed) can be made uniform, and therefore the learning accuracy can be made uniform. Furthermore, by randomizing one of the weighting coefficients, the converged value of the other weighting coefficient can be spread over the entire space of possible values of the weighting coefficient, which makes it possible to efficiently learn data that is not necessarily orthogonal. It becomes possible.

〔Example〕

An embodiment of the neural network of the present invention will be shown for speech synthesis and speech recognition.

FIG. 2 shows a speech synthesis system that includes a phoneme generator 22 and a speech parameter generator 24, and includes a neural network N.
A NW is provided for each. The input voice 26 is extracted by a voice parameter automatic extraction system 20 (Japanese Patent Laid-Open No. 59-152496,
152497), and also analyze voice parameters, i.e., sound source parameters such as sound source power, voiced/unvoiced parameters, and pitch, and vocal tract parameters such as vocal tract cross-sectional area and PARCOR coefficient, or AR (all-polar) parameters. ,
AR/MA (pole-zero type) parameters, spectra, and other parameters obtained by analyzing the audio are obtained and used as learning inputs (target outputs) for the audio parameter generation section.

Furthermore, the phoneme of the human voice is determined from the voice parameters obtained from the automatic extraction system 20 or the original waveform, and is used as the learning human power (target output) of the phoneme generation section 22.

The input to the phoneme generation unit 22 is the voice to be uttered (input voice 2
6) This is the text (character string) TX that will be used later. The character string T, for example "early in the morning..." is the phonetic sequence rA,
S, A, H, A, Y, A, U,...'' and is converted into the neural network N of the phoneme generating unit 22.
Ranked among the top 10 human resources in NW. (One phoneme is input and output sequentially, so that there are a predetermined number of phonemes in the input layer). The above conversion is performed using average syllable length, rule synthesis, or knowledge of phonology. Although a character string does not have the temporal element that exists in speech, this temporal element is added when converting between T and -11. Furthermore, since a phoneme cannot be determined by a single character, a plurality of characters are referred to and each phoneme is determined successively at frame intervals. In this way, a time element is added, but the speed is average, and the actual speed is N
Corrected by learning in NW.

As described above, the learning input is phoneme sequence data determined from real speech, and the neural network NNW of the phoneme generation unit 22 corrects the input phoneme sequence Ip to the learning input phoneme sequence and outputs it. . (2) becomes an input to the audio parameter generation section 24. The learning input is 1, 0, but the phonological output takes a value between 0 and 1. At this time, if necessary, a threshold value may be provided and the phoneme output may be set to values of only 0 and 1.

The voice parameter generation unit 24 receives the above output, converts it into the learning input (voice parameter) and outputs it, and this output O1T is added to the voice synthesis circuit 28 to generate a synthesized voice "Early in the morning... ..." is output.

FIG. 3 shows details of the audio parameter generation section.

The input layer 10 and the final output layer 16 have one stage, and the intermediate layer 12 and the output layer 14 have M stages. The input to the input layer 10 is the phoneme sequence (1), and 13 sequences (music pieces) containing IP points (data) are input and processed at one time. Each column is entered sequentially, with t in the middle one. , even for those below it. . 1 to 0.3 as well as those above. −
I~t o-i are attached. The direction of progress of time t is indicated by an arrow.

The number M of the intermediate layer 12 and the output layer 14 is preferably large to some extent because if it is small, the meaning of randomization and weighted averaging will be lost. For example, let M=9 for l5=29.

The number of elements J, -J, in each stage of the intermediate layer may be different, but here, without loss of generality, Jl-...
=J, -...=J, 4=J. Output layer 14
The number of each film element is also the same, and here it is 1−...=
K.

＝・・・・・・−、＝＝に。 Also, in the output layer 14, the points of the input series that each stage has (time in the time series of this example)
may be any point (time) in the input series, but without loss of generality, it is assumed that Kl-KM takes values at points adjacent to a certain series point (time) as the center. At this time, ■) Learning randomly selects 13 columns of data that are connected to each other from the input series, and from the center value to both sides (
Output data strings (hectors) corresponding to only M-1)/2 series points are sequentially given to each stage, and the values (target values) and the manual series are set as a set for each stage. This random selection learning is repeated as many times as necessary. In addition, in the final output layer 16, since M pieces of data are given to - points (-times) of the series, appropriate weights (for example, Rectang
ular, or Hamming, Ilann
ing and other Window functions) and take the average value.

■) When performing learning, select one of the weighting coefficients between the human layer and the intermediate layer, or between the intermediate layer and the output layer, as necessary (here, - from the intermediate layer to the output layer without losing membrane properties). The weighting coefficient values between the two are randomized and trained.

(2) Also, in the learning process, the weighting coefficients between the intermediate layer and the output layer are randomized, and learning between the input layer and the intermediate layer is performed according to equations (23), (24), and e6). At this time, the function r−(·) in equation (25) and equation (26).

M(・) can be set as the same as equation 0 QΦ, (22), or ■ be a constant at the initial stage of learning, and a weighting function can be experimentally determined based on the learning results and taking into consideration the nature of human coverage and the convergence of learning. decide.

FIG. 4 shows an outline of an embodiment of speech recognition. Speech recognition is the reverse process of speech synthesis, using a neural network N
Speech parameters are sequentially input to the NW, a phoneme sequence is obtained as an output, and a character string is obtained from this phoneme sequence.

That is, an automatic extraction system is applied to the input speech to obtain speech parameters, that is, sound source parameters and vocal tract parameters (other parameters such as AR parameters, spectral parameters, Walsh-1 (adamard,
Harr parameters may also be used. Using these parameters as input, a multi-stage neural network NNW
The final output is obtained by applying . The final output is a phonological parameter sequence that takes values between 0 and 1, which is the opposite of the case of speech synthesis. In this case, the phoneme parameters have already been weighted and averaged, so the output value indicates the probability that the phoneme is the same.

Therefore, the specific method for ultimately determining which phoneme the input speech is is: ■ A certain point in the series (time in the case of a time series)
The one with the highest output value is selected from among the elements that are firing at the same time. Alternatively, (1) select candidates as candidates in descending order of output value, and use an island drive method or the like to determine the one that is phonologically most likely; ■By applying this system to a large amount of speech data, we will turn the knowledge gained into rules and construct an expert system to determine phonemes. Once the phoneme sequence is determined, it is converted into a character string.

In this case, a neural network is configured that performs the reverse process of the phoneme generation section in Figure 2 (that is, it takes a phoneme sequence as input and a character string as output), and takes the same procedure as for determining the phoneme described above. By (that is, considering the output value as the probability that it is that character, the above ■,
It is also possible to determine the character string (by following steps ① and ②).

〔Effect of the invention〕

As explained above, the present invention multiplexes neural networks, takes a weighted average as the final output, and randomizes the weighting coefficients between the input layer and the intermediate layer or between the intermediate layer and the output layer, so that learning of analog information is extremely efficient. It is possible to converge accurately and quickly, and when used in automatic speech synthesis, it is possible to obtain effects such as obtaining more natural and better synthesized speech.

[Brief explanation of the drawing]

FIG. 1 is an explanatory diagram of the principle of the present invention; FIGS. 2 to 4 illustrate embodiments of the present invention; FIG. 2 is an explanatory diagram of a speech synthesis system; FIG. 3 is an explanatory diagram of a speech parameter generation unit; FIG. 4 is an explanatory diagram of a speech recognition system, FIGS. 5 and 6 are illustrations of a conventional example, FIG. 5 is an explanatory diagram of an analog neuron element, and FIG. 6 is an explanatory diagram of a neural network model. In Figure 1, 10 is an input layer, 12 is an intermediate layer, 14 is an output layer,
16 is the final output layer, ■ is the input sequence, and OT is the final output.

Claims (1)

[Claims] 1. A neural network consisting of an input layer (10), a multi-stage intermediate layer (12), a multi-stage output layer (14), and a final output layer (16) is used, and the input layer has an input sequence ( I), and the final output layer takes a weighted average of each output of the multi-stage output layer and calculates this as the final output (
O^T), and is characterized by randomizing either the weighting coefficient between the input layer and the hidden layer or the weighting coefficient between the hidden layer and the output layer, and learning it independently at each stage. pattern learning/generation method.