JP2996925B2 - Phoneme boundary detection device and speech recognition device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a bi-directional recurrent neural network.
Neural Network; hereinafter referred to as BRNN. ). In this specification, a boundary between phonemes is called a phoneme boundary.

[0002]

2. Description of the Related Art Conventionally, a method of recognizing speech using a speech segment of a speech signal as an acoustic model for a speech recognition apparatus is disclosed in, for example, prior art document 1 "T. Svedsen et al.,
“On the automatic segmentaiton of speech signal
s ", Proceedins of ICASSP-87, pp. 77-80, 1987", prior art document 2 "A. Ljolje et al.," Automatic segmentati
onand labelling of speech, "Proceedings of ICASSP-
91, pp. 473-476, 1991 "and prior art document 3" J. Glas
s et al., “A probabilistic framework for feature-b
ased speech recognition ", Proceedings of IGSLP-96, p
p.2277-2280, 1996 ". Here, in prior art documents 1 and 2, a method of automatic segmentation for creating an acoustic model and speech synthesis in speech recognition is disclosed. In prior art document 3,
A pre-processing for speech recognition is disclosed.

[0003]

In the prior art document 1, a hidden Markov model (hereinafter, referred to as HMM).
In the prior art document 2, the automatic labeling of the phoneme label is further performed by using the duration time model using the utterance voice newly written text data. However, since the HMM model has been learned so as to maximize the likelihood for phoneme detection, there is a problem that the performance of the phoneme detection is relatively low and the processing time is relatively long. Further, in the prior art document 2, since the phoneme is detected using the duration model, there is a problem that the processing time is relatively long.

A first object of the present invention is to solve the above problems and to provide a phoneme boundary detection device capable of detecting a phoneme boundary with higher accuracy and at a higher speed as compared with the conventional example.

A second object of the present invention is to solve the above-mentioned problems and to use the above-mentioned phoneme boundary detection device to perform speech recognition with a higher speech recognition rate and higher speed than in the conventional example. It is to provide a device.

[0006]

According to a first aspect of the present invention, there is provided a phoneme boundary detecting apparatus comprising: an input layer, at least one intermediate layer having a plurality of units, and a phoneme having one unit. An output layer that outputs a phoneme boundary detection value representing a boundary detection probability, using a bidirectional recurrent neural network, a phoneme boundary detection device that detects a phoneme boundary of a speech feature parameter sequence, wherein the input layer is A plurality of units, a first input neuron group having a plurality of units, a forward module, and a backward module, wherein the forward module is based on the plurality of voice feature parameters, A plurality of parameters having a forward feedback connection and being delayed by a predetermined unit time from a plurality of parameters output from the first input neuron group; While output to the intermediate layer to produce a plurality of parameters of time, the above-mentioned backward module,
Based on the plurality of speech feature parameters, a plurality of times delayed backward by a predetermined unit time from the plurality of parameters output from the first input neuron group having a temporally backward feedback connection. Is generated and output to the intermediate layer.

According to a second aspect of the present invention, there is provided the phoneme boundary detection device according to the first aspect, wherein the forward module receives a plurality of speech feature parameters as inputs and has a plurality of units. A neuron group, a first intermediate neuron group having a plurality of units and having a plurality of units input thereto with a plurality of parameters delayed and output by a predetermined unit time from the second intermediate neuron group, and the second input neuron group And a plurality of parameters output from the first intermediate neuron group are connected so as to be input by multiplying each of the plurality of parameters by a respective weighting factor. Wherein the backward module comprises a plurality of speech feature parameters. A third input neuron group having a plurality of units as inputs, and a plurality of units having as input a plurality of parameters output from the fourth intermediate neuron group after being delayed in a reverse direction by a predetermined unit time and output A third intermediate neuron group, a plurality of parameters output from the third input neuron group, and a plurality of parameters output from the third intermediate neuron group are multiplied by respective weighting factors, respectively. A fourth intermediate neuron group having a plurality of units connected to be input and multiplying a plurality of parameters output from the second intermediate neuron group by respective weighting factors. The first input menu is connected to be input to a plurality of units of the hidden layer. A plurality of parameters output from the fourth intermediate neuron group are connected to each other by multiplying each of the plurality of parameters output from the intermediate group by a respective weighting factor and input to each of the plurality of units in the intermediate layer. The parameters are each multiplied by each weighting factor and connected so as to be input to a plurality of units of the intermediate layer, respectively, and the plurality of parameters output from the intermediate layer are each multiplied by each weighting factor. Each is connected so as to be input to the unit of the output layer.

Further, the phoneme boundary detection device according to claim 3 is the phoneme boundary detection device according to claim 1 or 2,
It is characterized by further comprising a first detecting means for detecting a phoneme boundary as a phoneme boundary when a phoneme boundary detection value output from the output layer is equal to or greater than a predetermined threshold value.

Further, the phoneme boundary detection device according to claim 4 is the phoneme boundary detection device according to claim 1 or 2,
It is characterized by further comprising a second detecting means for detecting a phoneme boundary when the phoneme boundary detection value output from the output layer is equal to or greater than a predetermined threshold value and reaches a local maximum value.

Further, the phoneme boundary detection device according to claim 5 is the phoneme boundary detection device according to claim 1 or 2,
When the detected phoneme boundary value output from the output layer is equal to or greater than a predetermined first threshold value, it is detected as a first phoneme boundary, and the detected phoneme boundary value is determined based on the first threshold value. And a third detecting means for detecting as a second phoneme boundary when the second phoneme boundary is not less than the second threshold value and smaller than the first threshold value and has a maximum value. I do.

According to a sixth aspect of the present invention, there is provided the phoneme boundary detecting device according to the fifth aspect, wherein the third detecting means detects a plurality of the phoneme boundaries detected as the first phoneme boundary. Each time, one phoneme boundary is selected and selected as a first phoneme boundary.

Further, the phoneme boundary detecting device according to claim 7 is the phoneme boundary detecting device according to claim 5 or 6,
The third detecting means detects a phoneme boundary based on a lattice of a path formed between the detected or selected first phoneme boundary and the second phoneme boundary.

According to a second aspect of the present invention, there is provided a speech recognition apparatus for extracting a speech feature parameter from a speech signal of an uttered speech sentence composed of an input character string, and the feature extraction means. An utterance composed of an input character string using a phoneme boundary detected by the phoneme boundary detection device according to one of claims 1 to 7 based on a voice feature parameter and a predetermined acoustic model. Voice recognition means for voice-recognizing a voice signal of a voice sentence.

[0014]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows a phoneme boundary detection neural network 1 according to an embodiment of the present invention.
FIG. 3 is a block diagram of a speech recognition device using 0. In this embodiment, the feature parameter file 3 of the learning speech data is used.
1 and a phoneme boundary value file 32 of the learning speech data, a predetermined learning algorithm is used to learn the initial model 33 of the phoneme boundary detection neural network.
The neural network learning unit 20 that obtains 0 is used. The word level matching unit 5 detects a phoneme boundary using the obtained phoneme boundary detection neural network 10 and detects a phoneme to perform word-level speech recognition. Features. Therefore, the word level matching unit 5 includes a phoneme boundary detection device.

In this embodiment, a phoneme boundary detection neural network 10 composed of BRNN shown in FIG. 2 is used for detecting phoneme boundaries. While a general recurrent neural network recursively uses past information in time, BRNN is characterized in that past and future input information can be used.

In FIG. 2, the input is information on speech feature parameters such as cepstrum, and the output is information on whether the input of the frame is a phoneme boundary (for example, 1 if the frame is a phoneme boundary, 1 otherwise 0) is given as a teacher signal at the time of learning. That is, the number of units of the input neuron group A (t) is 26 in the number of dimensions of the speech feature parameter, and the number of output units is 1. Here, the voice feature parameters include a 12-dimensional mel-cepstral coefficient (hereinafter, referred to as MFCC), power, and respective first-order regression coefficients. Hereinafter, the total number of frames in one file is L.

In FIG. 2, the forward-facing module B (t-
1) is delayed by a predetermined unit time from the 26 parameters output from the input neuron group A (t) with a temporally forward feedback connection based on the 26 speech feature parameters. A module that generates ten parameters at time t-1 and outputs them to the hidden neuron group D. (a) An input having ₂₆ units a _{1 to} a ₂₆ that receives 26 speech feature parameters as input A neuron group 51, (b) an intermediate neuron group 52 having _ten units b _{1 to} b ₁₀ which receives ten parameters output from the intermediate neuron group 53 via the delay element 54, and (c) 26) 26 parameters output from the input neuron group 51 and 10 parameters output from the intermediate neuron group 52 Are respectively multiplied by the respective weighting factors and are connected so as to be input respectively. The intermediate neuron group 53 having _ten units b _{1 to} b ₁₀
(D) a delay element 54 for delaying the ten parameters output from the intermediate neuron group 53 by a predetermined unit time and outputting the delayed parameter to the intermediate neuron group 52;
(E) The state neuron group of the forward module B (t-1) having _ten units b _{1 to} b ₁₀ becomes the intermediate neuron group 53 at time t−1 at the time when the operation of the forward module B (t−1) ends. The output value output from is temporarily stored, that is, latched,
An output latch 55 for outputting to the hidden neuron group D of the neural network on the right side for detecting a phoneme boundary.

In the forward module B (t-1) configured as described above, the intermediate neuron group 52
Form a feedback loop through the intermediate neuron group 53 and the delay element 54 to the intermediate neuron group 52.
The output parameter vector B _m (m = 1, 2,..., T−1) output from the intermediate neuron group 53 at time t−1 at the end of the operation of (t−1) is represented by the following equation.

[0019]

[Number 1] _{B m = W FA · A m} + W FB · B m-1

Here, the output value vector B _m is composed of ten parameter values, and the initial value vector B ₀ is represented by the following equation.

(Equation 2)

Further, the input parameter vectors A _m to the input neuron group 51 is expressed by the following equation.

(Equation 3)

Where O _m (1) is the MF at time m
O _m (2) is the primary value of CC at time m
This is a second order value of FCC, and similarly, O _m (2
6) is the 26th-order MFCC value at time m. Further, the weighting factor matrices W _FA and W _FB of _{Equation 1} are 10
It is a 26 × 10 matrix and a 10 × 10 matrix, and is represented by the following equation.

(Equation 4)

(Equation 5)

Further, in FIG. 2, the backward module C (t + 1) has a temporally backward feedback connection and is output from the input neuron group A (t) based on the 26 speech feature parameters. Time t + 1 delayed backward by a predetermined unit time from the number of parameters
(A) 26 units a _{1 which} receive 26 speech feature parameters as inputs.
An input neuron group 61 having a to a ₂₆ ,
(B) an intermediate neuron group 62 having _ten units c _{1 to} c ₁₀ which receives ten parameters output from the intermediate neuron group 63 via the reverse delay element 64; and (c) an input neuron Group 6
10 units c _{1 to} c ₁ to 26 connected so that each of the 26 parameters output from 1 and the 10 parameters output from the intermediate neuron group 62 are multiplied by each weighting factor and input. The intermediate neuron group 63 having c ₁₀ and the intermediate neuron group 6 (d) delaying ten parameters output from the intermediate neuron group 63 by a predetermined unit time.
Opposite to the delay element 64 to be output to 2, (e) backward with 10 units c ₁ to c ₁₀ module C (t +
The state neuron group of 1) is obtained, and the output value output from the intermediate neuron group 63 at the time t + 1 at the end of the operation of the backward module C (t + 1) is temporarily stored, that is, latched to detect a phoneme boundary. And an output latch 65 for outputting to the hidden neuron group D of the neural network on the right side of.

In the backward module C (t + 1) configured as described above, the intermediate neuron group 62
To intermediate neuron group 63 and reverse delay element 6
4, a feedback loop is formed to the intermediate neuron group 62, and the output parameter vector C _m output from the intermediate neuron group 63 at time t + 1 at the end of the operation of the backward module C (t + 1) after repeatedly calculating. (M = L, L-1,..., T + 1) is represented by the following equation.

[0025]

[Formula 6] C _m = W _BA · A _m + W _BC · C _{m + 1}

Here, the output value vector C _m is composed of ten parameter values, and the initial value vector C _{L + 1} is represented by the following equation.

(Equation 7)

Further, the input parameter vectors A _m to the input neuron group 61 is the same as the number 3.

Further, the weighting coefficient matrices W _BA and W _BC of Equation 6 are 10 × 26 matrices and 10 × 10 matrices, respectively, and are represented by the following equations.

(Equation 8)

(Equation 9)

Further, as shown in FIG. 2, a hidden neuron group D having ₃₀ hidden units d _{1 to} d ₃₀ and a phoneme boundary detection having _one output unit e ₁ and representing a phoneme boundary detection probability. Value y (j) (j = 1, 2,...,
L), and an output neuron group E for outputting L).
Each of the output parameters of the units b _{1 to} b ₁₀ of the state neuron group B (t−1) is multiplied by each weighting factor, and the unit d _{1 of the} hidden neuron group D is multiplied.
To d _30, and the output parameters of the units c _{1 to} c ₁₀ of the state neuron group C (t + 1) are respectively multiplied by the respective weighting factors, and the unit d _{1 of the} hidden neuron group D is multiplied. To d ₃₀ , and connected to 26 units a _{1 to} a ₂₆
Are multiplied by the respective weighting factors for the respective output parameters of the input neuron group A (t), and are input to the units d _{1 to} d ₃₀ of the hidden neuron group D. The output parameters of the units d _{1 to} d ₃₀ of the hidden neuron group D are respectively multiplied by the respective weighting coefficients, and the output parameters are connected to the output unit e ₁ of the output neuron group E.

Here, the state neuron group B (t−
1) and C (t + 1) and input neuron group A
The processing from (t) to the output neuron group E via the hidden neuron group D is performed by the forward module B
After the end of the processing operation of (t-1) and the backward module C (t + 1), a learning process or an arithmetic process is executed. In the neural network, the input layer 100
Is a forward module B (t−1) that calculates an output parameter at time t−1 delayed by a unit time from the output time t of the input neuron group A (t), a backward module C (t + 1) that computes output parameters at t + 1 delayed unit time backward from t, the hidden layer 200 comprises a hidden neuron group D, and the output layer 30
0 comprises the output neuron group E. The phoneme boundary detection neural network 10 configured as described above
Is equivalently, as shown in FIG. 3, the forward module and the backward module are connected in the time direction, and the input layer 100
Is a BRNN composed of an input neuron group A (t), a forward module B (t-1), and a backward module C (t + 1).

FIG. 10 shows an output example when a feature parameter time series is input to the phoneme boundary detection neural network 10 after learning by the neural network learning process of FIG. This example is obtained for the open data using the neural network 10 learned under the conditions described later in detail. Here, a dotted line indicates a teacher signal (true value), and a solid line indicates an output value (detected value) of the neural network 10.

Next, the following four methods were devised as algorithms for detecting a phoneme boundary from the output results as shown in FIG. (A) Method 1: An output value exceeding a threshold value h is determined as a phoneme boundary candidate. That is, an output value satisfying the following equation is determined as a phoneme boundary candidate.

(10) y (j) ≧ h

(B) Method 2: From the output values exceeding the threshold value h, the one having the maximum value is selected as a phoneme boundary candidate. That is, an output value satisfying the following equation is determined as a phoneme boundary candidate.

Y (j) ≧ h and y (j)> y (j−1) and y (j)> y (j + 1)

(C) Method 3: Two types of thresholds l and h
By using (> l), all those having the local maximum values from the second threshold value l to the first threshold value h and those exceeding the first threshold value h are selected. That is,

When y (j) ≧ h, a first phoneme boundary candidate is selected.

(13) l ≦ y (j) <h and y (j)> y (j−
1) If y (j)> y (j + 1), select as the second phoneme boundary candidate. (D) Method 4: In method 3, only two continuous first phoneme boundaries are selected as the first phoneme boundaries per k units.

Methods 1 and 2 use only this process,
This is a method for uniquely determining the boundaries of phonemes. The methods 3 and 4 are methods for first leaving as many possible candidates as possible in these processes, and then determining phoneme candidates by another process. For example, a candidate detected beyond the first threshold value h is defined as a first phoneme boundary candidate, and a candidate detected between the second threshold value l and the first threshold value h is defined as a second phoneme boundary candidate.
, A lattice as shown in FIG. 11 can be created for all candidates existing between the first phoneme boundaries. At this time, the optimal phoneme path can be determined by re-evaluating the lattice using an acoustic model such as an HMM or a phoneme model based on a segment model, and thereby a final phoneme boundary can be determined.

In FIG. 1, the A / D converter 2 and
The feature extracting unit 3, the word level matching unit 5, the sentence level matching unit 6, and the neural network learning unit 20 are configured by, for example, an arithmetic control device such as a digital computer, and the buffer memory 4 is configured by, for example, a hard disk memory. And the characteristic parameter file 31 of the voice data for learning.
And a phoneme boundary value file 32 of learning speech data, and an initial model 33 of a phoneme boundary detection neural network.
The phoneme boundary detection neural network 10, the word model 7, the grammar rules 8, and the semantic rules 9 are stored in, for example, a hard disk memory.

FIG. 4 is a flowchart showing a neural network learning process executed by the neural network learning section 20 of FIG. In FIG. In step S1, a feature parameter file 31, a phoneme boundary value file 32 corresponding to the feature parameter file, and an initial model 33 of the phoneme boundary detection neural network are read. Next, in step S2, the number of feature parameter files 31 corresponding to the total number of utterances in the phoneme boundary value file 32 is set as the parameter N, and the number of learning repetitions is set as the parameter I. Then, the parameter i is initialized to 1 in step S3, and the parameter n is initialized to 1 in step S4. Step S
In step 5, the total number of frames of the n-th file is set in the parameter Ln. Next, in step S6, using the feature parameters of the Ln frame, the output values of the state neuron group B (t−1) of the forward module, the state neuron group C (t + 1) of the backward module, and the output neuron group E (each of the Ln group ) Is calculated, and the weight coefficient update parameter of the neural network is calculated.

Then, in step S7, the parameter n is set to 1
After incrementing by only N, it is determined in step S8 whether n> N. If n ≦ N, the process returns to step S5 to repeat the above processing. When n> N in step S8,
In step S9, a process of updating the weight coefficient of the neural network is executed, and in step S10, the parameter i is set to 1
After incrementing by only i, it is determined in step S11 whether i> I. Here, when i ≦ I, it is determined that the predetermined number of repetitions has been reached, the phoneme boundary detection neural network 10 obtained in step S12 is stored in the memory, and the process ends.

FIG. 5 is a flowchart showing a word matching process executed by the word matching unit of FIG. In FIG. 5, first, the feature parameters stored in the buffer memory 4 in step S21 and the phoneme boundary detection neural network 10 are read. Next, in step S22, the log likelihood Pw for the word model 7 is calculated based on the feature parameters. Further, in step S23, based on the characteristic parameters, the output values y (j), j = 1, 2,..., L of the neural network 10 for each of the total L frames of the characteristic parameters are calculated. Then, the log value of the output value y (j) is calculated in step S24, and the log likelihood is calculated.

[Equation 14] Get. Then, after performing the phoneme boundary detection processing in step S25, the weighted sum P of the calculated log likelihoods Pw and Ps is calculated.
_total is calculated using the following formula,

## EQU15 ## P _total = kPw + (1−k) Ps Performs word level collation processing. That is, the candidate word having the maximum likelihood is output to the sentence level matching unit 6 as a recognition result based on the calculated likelihood P _total , and the word level matching processing ends.

FIG. 6 is a flowchart of a phoneme boundary detection process (method 1) (step S25) which is a subroutine in the word matching process of FIG. In FIG. 6, the output value y (j) of the phoneme boundary detection neural network 10 up to L for each frame j is:

It is determined whether y (j) ≧ h. If YES, the phoneme boundary is determined, and if NO, it is determined to be within the phoneme.

FIG. 7 is a flowchart of a phoneme boundary detection process (method 2) (step S25) which is a subroutine in the word matching process of FIG. In FIG. 7, the output value y (j) of the phoneme boundary detection neural network 10 up to L for each frame j is

It is determined whether y (j) ≧ h and y (j)> y (j−1) and y (j)> y (j + 1). If YES, the phoneme boundary is determined. When the result is NO, it is determined to be within a phoneme.

FIG. 8 is a flowchart of a phoneme boundary detection process (method 3) (step S25), which is a subroutine in the word matching process of FIG. In FIG. 8, the output value y (j) of the phoneme boundary detection neural network 10 up to L for each frame j is

If y (j) ≧ h, it is determined to be the first phoneme boundary,

[Equation 19] l ≦ y (j) <h and y (j)> y (j−
1) If y (j)> y (j + 1), it is determined to be the second phoneme boundary; otherwise, it is determined to be within the phoneme.

FIG. 9 is a flowchart of a phoneme boundary detection process (method 4) (step S25), which is a subroutine in the word matching process of FIG. In FIG. 9, the same processes are denoted by the same step numbers. FIG.
The flowchart of the step S57 is different from that of FIG.
Is inserted before the step S58.
Is characterized in that only two continuous first phoneme boundaries are thinned out and selected as the first phoneme boundaries for every k units.

Next, the configuration and operation of the free speech recognition apparatus shown in FIG. 1 will be described. In FIG. 1, a speaker's uttered voice, which is a uttered voice sentence composed of a character string, is input to a microphone 1 and converted into a voice signal, and then input to an A / D converter 2. The A / D converter 2 performs A / D conversion on the input audio signal at a predetermined sampling frequency, and outputs the converted digital data to the feature extractor 3. Next, the feature extraction unit 3 performs, for example, LPC analysis on the digital data of the input audio signal,
An 11-dimensional feature parameter including a 0-dimensional MFCC and power is extracted. The time series of the extracted feature parameters is input to the word level matching unit 5 via the buffer memory 4.

In generating a word model, a word model is generated by executing a word model generation process with the maximum likelihood based on predetermined model parameters as follows. That is, a sample of a representative phoneme label of the word having the maximum likelihood is detected from the acoustic features of a plurality of N words that are the same word in the model parameters, and a sample of the detected representative phoneme label is detected. , Temporally normalizing the phoneme label samples of a plurality of N words with the phoneme label samples by the dynamic time matching method, and temporally normalized representative phoneme label samples; By mixing the plurality of N phoneme label samples with each word, a word model including an acoustic feature for each word is generated and stored in the word model memory 7. In short, a word model including a speech feature parameter for each word of the text is generated based on the generated stochastic phoneme model of the mixture distribution.

The word model in the word model memory 7 connected to the word level collating unit 5 is composed of a plurality of cascade-connected environment-dependent phoneme models for connecting preceding and succeeding phoneme environments. Each state includes the following information. (A) state number, (b) average value of 11-dimensional acoustic features, (c) variance of 11-dimensional acoustic features, (d) duration, (e) weight of each cluster, and ( f) Phoneme code corresponding to phoneme label.

The word level collating unit 5 and the sentence level collating unit 6 constitute a speech recognition circuit unit. The sentence level collating unit 6 includes the output probabilities of parts of speech and words, the transition probabilities between parts of speech and between words, and the like. The grammar rules stored in the grammar rule memory 8 are linked to the semantic rules stored in the semantic rule memory 9 including the output probabilities of the thesaurus and the dialog management rules. The word level collating unit 5 performs word level speech recognition by executing the word level collation processing of FIG. That is, the word level collating unit 5 detects at least one speech recognition candidate word by collating with the word model in the memory 7 based on the time series of the input acoustic feature amounts, and detects the detected candidate word. , And executes the above-described phoneme boundary detection processing to detect phoneme boundaries, and outputs the candidate word having the maximum likelihood to the sentence level matching unit 6 as a recognition result word. . Further, the sentence level matching unit 6 executes a sentence level matching process by referring to a language model including the grammatical rule and the semantic rule based on the input word of the recognition result, thereby obtaining a final speech. Output sentence of recognition result. If there is a word that is not accepted by the language model, the information is returned to the word level collating unit 5 and the word level collation is executed again. The word level collating unit 5 and the sentence level collating unit 6 recognize a continuous speech of a free utterance by sequentially connecting words composed of a plurality of phonemes, and output the speech recognition result data.

[0048]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present inventor has compared (1) Method 2 with a phoneme boundary obtained by phoneme recognition based on HMM as a phoneme detection result using a speech database owned by the present applicant. 2) Two types of performance evaluations of the methods 2 to 4 were performed. As input to the neural network 10, a frame length of 25.6 msec and a frame period of 10
26-dimensional MFCC (12-dimensional MF
CC, power and their respective first-order regression coefficients) were used. The output is given using phoneme label information in the database, and is given as 1 when the frame is a phoneme boundary, 0.5 when the frame is adjacent to the phoneme boundary, and 0 otherwise. The number of units of the forward and backward modules in the neural network 10 was set to 10, the number of units of the hidden module D was set to 30, and the number of times I of learning was set to 1,000. At this time, the total number of weighting factors of the neural network 10 is 2,181. The training data is 462 speakers (3,696 sentences), the total number of phoneme boundaries is about 140,000 (about 1.1 million frames), and the evaluation data is 168 speakers (1,344 sentences) different from the learning data, and the total number of phoneme boundaries is 50. , 318 (about 410,000 frames). Average 2 between true value and detected value of neural network 10
The squared errors were 0.0604 and 0.0621 for the learning data and the evaluation data, respectively. The threshold value in methods 2 to 4 is experimentally h = 0.
4, 1 = 0.1.

Next, an evaluation method will be described. If the result of the detection of the phoneme boundary is included in the margin within a predetermined ± M frame with respect to the phoneme boundary visually labeled, the correct answer (hereinafter, the number of correct answers is H).
If it was not included, it was determined as dropout (hereinafter, the dropout number is D). The source of phonemes was inserted (hereinafter, the number of insertions is referred to as I). When a plurality of detection candidates are included in a predetermined ± M frame, one is excluded except for one. At this time, the accuracy rate and accuracy are defined by the following equations.

[0050]

[Equation 20] Correct answer rate = H / N × 100 (%)

[Mathematical formula-see original document] Accuracy = (N-DI) / N * 100 (%)

In this embodiment, the performance of detecting a phoneme boundary was evaluated using the above two measures. Where N is the total number of phonemes visually labeled,

N = H + D

First, a comparison between the method 2 and the result based on the HMM will be described. Table 1 shows the detection results obtained by the method 2 for M = 0, 1, and 2. In order to compare the performance of the phoneme boundary detection neural network 10 of the present embodiment, H
Phoneme recognition was performed using phoneme bigrams based on MM, and the resulting phoneme boundaries were compared with those obtained as phoneme boundaries. Here, attention is paid only to the phoneme boundary (time) information, and the recognition result is not considered. Table 2 shows the results of creating an environment-independent model with three states and five mixtures each for 61 phonemes with a phoneme label number of 61. Junichi et al.,
“Automatic Generation of Hidden Markov Networks by Sequential State Division Method”,
IEICE Transactions D-II, Vol. J76-D
-II, No. 10, pp. 2155-2164,19
October 1993. " Table 3 shows the results in the case where () (silent model is an HMM of 10 mixtures of 3 states). Table 1
Comparing Table 2 with Tables 2 and 3, a higher accuracy is obtained with the method based on the neural network 10. This is because the HMM does not learn the model parameters in order to detect the phoneme boundaries, but evaluates using the phoneme boundary information obtained secondarily, whereas the neural network 10 uses the phoneme boundaries. It is considered that the learning for detecting is performed.

[0053]

[Table 1] Phoneme boundary detection result based on BRNN (method 2) Threshold value h = 0.4─────────────────────────── {M 0 1 2} Correct answer 23,175 38,248 40,056 Insertion 18,983 4,066 2,293 dropout 27,143 12,070 10,262 ──────────────────────────── Correct answer rate 46.06 76.01 79 .61 Accuracy 8.33 67.93 75.05

[0054]

Table 2 Results of phoneme boundary detection based on HMM (a) Environment-independent model Ｍ M 0 1 2 ────────────────────────────── Correct answer 8,806 28,214 38,847 Insert 35,372 16,253 5 , 915 dropped 41,512 22,104 11,471 ────────────────────────────── correct answer rate 17.50 56.07 77.20 Accuracy -52.80 23.77 65.45

[0055]

Table 3 Results of phoneme boundary detection based on HMM (b) Environment-dependent model Ｍ M 0 1 2 ────────────────────────────── Correct answer 14,198 35,967 42,611 Insert 32,970 11,521,5 110 dropout 36,120 14,351,707 ────────────────────────────── Correct answer rate 28.22 71.47 84 .68 Accuracy -37.31 48.58 74.53

Next, Table 4 shows a comparison of the performance by the methods 2, 3, and 4.

[0057]

[Table 4] {Method 2 34} ──────────────────── Correct answer 40,056 48,856 48,856 Insertion 2,293 67,570 30,629 Dropout 10,262 1,462 1,461 ─────────────────────────────── Correct answer rate 79.61 97.10 97.10 Accuracy 75.05 -37. 19 36.22 ───────────────────────────────

Here, the thinning interval of Method 4 was set to k = 2, and all evaluations were performed with M = 2. It can be seen that Method 2 has the highest accuracy, but has a large number of drops. As described above, when the phoneme boundary candidates can be re-evaluated, this method with a large number of dropouts is not considered appropriate. In method 3, the number of dropouts can be significantly reduced as compared to method 2, but on the contrary, the number of insertions has increased significantly. In Method 4, the number of insertions is less than half without increasing the number of drops compared to Method 3. Also, when the detection result of the method 4 is expressed in a lattice, it can be seen that as many as 97.10% of correct answers are included in the lattice.

As described above, according to the present embodiment, the neural network 10 which is a BRNN is learned using the speech feature parameters, and the phoneme boundary is determined based on only the speech feature parameters using the learned neural network 10. The position can be quickly and accurately detected. By obtaining the phoneme boundary position more accurately, (a) the performance of speech recognition can be improved, and the calculation amount of speech recognition can be significantly reduced. (B) The phoneme boundary detection neural network 10
When an initial model of an HMM, which is an acoustic model, is created by using the above, the accuracy can be greatly improved. (C) Further, a phoneme boundary detection neural network 1
0 can be used for cutting out a speech waveform signal for speech synthesis, and in this case, a waveform cutting error can be significantly reduced.

[0060]

As described above in detail, according to the phoneme boundary detecting device of the first aspect of the present invention, an input layer, at least one intermediate layer having a plurality of units, and one unit An output layer that outputs a phoneme boundary detection value representing a phoneme boundary detection probability and having a bidirectional recurrent neural network, a phoneme boundary detection device that detects a phoneme boundary of a speech feature parameter sequence, The input layer receives a plurality of speech feature parameters as inputs, and includes a first input neuron group having a plurality of units, a forward module, and a backward module, wherein the forward module is based on the plurality of speech feature parameters. ,
A plurality of parameters having a temporally forward feedback connection and having a time delayed by a predetermined unit time from a plurality of parameters output from the first input neuron group are generated and output to the intermediate layer. On the other hand, the backward module has a temporally backward feedback connection based on the plurality of speech feature parameters, and has a backward unit of a predetermined unit time from the plurality of parameters output from the first input neuron group. Are generated and output to the intermediate layer. Therefore, the phoneme boundary position can be quickly and accurately detected based only on the voice feature parameters. Further, by obtaining the phoneme boundary position more accurately, the performance of speech recognition can be improved and the calculation amount of speech recognition can be significantly reduced.

Further, in the phoneme boundary detecting device according to the second aspect, in the phoneme boundary detecting device according to the first aspect,
The forward module receives a plurality of speech feature parameters as input, and inputs a second input neuron group having a plurality of units and a plurality of parameters output from the second intermediate neuron group delayed by a predetermined unit time. A first intermediate neuron group having a plurality of units, a plurality of parameters output from the second input neuron group, and a plurality of parameters output from the first intermediate neuron group. A second intermediate neuron group having a plurality of units and connected so as to be multiplied by the respective weighting factors, and wherein the second module has a plurality of units.
A plurality of speech feature parameters are input, and a plurality of parameters output from a third input neuron group having a plurality of units and delayed from the fourth intermediate neuron group by a predetermined unit time in the reverse direction are input. , A third intermediate neuron group having a plurality of units, a plurality of parameters output from the third input neuron group, and respective weights for a plurality of parameters output from the third intermediate neuron group. A fourth intermediate neuron group having a plurality of units and connected so as to be multiplied by a coefficient, and having a plurality of units, each of which has a weighting factor for each of a plurality of parameters output from the second intermediate neuron group And connected to be input to a plurality of units in the above-mentioned intermediate layer. , A plurality of parameters output from the first input neuron group are respectively multiplied by respective weighting coefficients, and the parameters are connected so as to be input to a plurality of units of the intermediate layer, respectively. Are connected so as to be input to the plurality of units of the intermediate layer, respectively, by multiplying each of the plurality of parameters output from the respective units by the respective load coefficients, and are respectively connected to the plurality of parameters output from the intermediate layer. They are connected so that they are multiplied by a load coefficient and input to the units of the output layer. Therefore, the phoneme boundary position can be quickly and accurately detected based only on the voice feature parameters. Further, by obtaining the phoneme boundary position more accurately, the performance of speech recognition can be improved and the calculation amount of speech recognition can be significantly reduced.

Further, in the phoneme boundary detection device according to the third aspect, the phoneme boundary detection device according to the first or second aspect, wherein the detected phoneme boundary value output from the output layer is equal to or more than a predetermined threshold value. First detected as a phoneme boundary
Is further provided. Therefore, the phoneme boundary position can be quickly and accurately detected based only on the voice feature parameters. In addition, by obtaining more accurate phoneme boundary positions, while improving the performance of speech recognition,
The amount of calculation for speech recognition can be greatly reduced.

Further, in the phoneme boundary detection device according to the fourth aspect, in the phoneme boundary detection device according to the first or second aspect, the phoneme boundary detection value output from the output layer is not less than a predetermined threshold value. And a second detecting means for detecting a maximum value as a phoneme boundary. Therefore, the phoneme boundary position can be quickly and accurately detected based only on the voice feature parameters. Further, by obtaining the phoneme boundary position more accurately, the performance of speech recognition can be improved and the calculation amount of speech recognition can be significantly reduced.

Further, in the phoneme boundary detection device according to the fifth aspect, in the phoneme boundary detection device according to the first or second aspect, the phoneme boundary detection value output from the output layer is:
When it is not less than a predetermined first threshold value, it is detected as a first phoneme boundary, and the phoneme boundary detection value is not less than a second threshold value smaller than the first threshold value, and First
And a third detecting means for detecting the second phoneme boundary when the value is less than the threshold value and reaches the maximum value.
Therefore, the phoneme boundary position can be quickly and accurately detected based only on the voice feature parameters. Further, by obtaining the phoneme boundary position more accurately, the performance of speech recognition can be improved and the calculation amount of speech recognition can be significantly reduced.

According to a sixth aspect of the present invention, there is provided the phoneme boundary detection device according to the fifth aspect.
The third detecting means selects one phoneme boundary for each of a plurality of detected ones as the first phoneme boundary and selects it as the first phoneme boundary. Therefore, the phoneme boundary position can be quickly and accurately detected based only on the voice feature parameters. Further, by obtaining the phoneme boundary position more accurately, the performance of speech recognition can be improved and the calculation amount of speech recognition can be significantly reduced.

Further, in the phoneme boundary detecting device according to claim 7, in the phoneme boundary detecting device according to claim 5 or 6, the third detecting means includes: A phoneme boundary is detected based on a lattice of a path formed between the two phoneme boundaries. Therefore,
The phoneme boundary position can be quickly and accurately detected based only on the voice feature parameter. Further, by obtaining the phoneme boundary position more accurately, the performance of speech recognition can be improved and the calculation amount of speech recognition can be significantly reduced.

In the speech recognition apparatus according to the present invention, a feature extraction means for extracting a speech feature parameter from a speech signal of an uttered speech sentence consisting of an input character string, and a feature extraction means for extracting the speech feature parameter. A phoneme boundary detected by the phoneme boundary detection device according to any one of claims 1 to 7 based on the obtained speech feature parameter, and a character string input using a predetermined acoustic model. Voice recognition means for voice-recognizing the voice signal of the uttered voice sentence. Therefore, the phoneme boundary position can be quickly and accurately detected based only on the voice feature parameters. Further, by obtaining the phoneme boundary position more accurately, the performance of speech recognition can be improved and the calculation amount of speech recognition can be significantly reduced.

[Brief description of the drawings]

FIG. 1 is a block diagram of a speech recognition apparatus using a phoneme boundary detection neural network according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a phoneme boundary detection neural network of FIG.

FIG. 3 is a block diagram showing an equivalent structure of the phoneme boundary detection neural network of FIG. 2;

FIG. 4 is a flowchart illustrating a neural network learning process performed by the neural network learning unit of FIG. 1;

FIG. 5 is a flowchart illustrating a word matching process performed by the word matching unit of FIG. 1;

6 is a flowchart of a phoneme boundary detection process (method 1) which is a subroutine in the word matching process of FIG.

FIG. 7 is a flowchart of a phoneme boundary detection process (method 2) which is a subroutine in the word matching process of FIG.

8 is a flowchart of a phoneme boundary detection process (method 3) which is a subroutine in the word matching process of FIG.

9 is a flowchart of a phoneme boundary detection process (method 4) which is a subroutine in the word matching process of FIG.

FIG. 10 is a graph showing an example detected by the phoneme boundary detection processing of FIG.

11 is a diagram illustrating a lattice representation of a phoneme boundary candidate in the phoneme boundary detection processing of FIG. 5;

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Microphone, 2 ... A / D converter, 3 ... Feature extraction part, 4 ... Buffer memory, 5 ... Word level collation part, 6 ... Sentence level collation part, 7 ... Word model, 8 ... Grammar rule, 9 ... Meaning 10: Phoneme boundary detection neural network, 20: Neural network learning unit, 31: Feature parameter file of learning speech data, 32: Phoneme boundary value file of learning speech data, 33: Initial of phoneme boundary detection neural network Model: 100 input layer, 200: middle layer, 300: output layer, A (t), 51, 61: input neuron group, B (t-1): forward module, C (t + 1): backward module, 52, 53, 62, 63: intermediate neuron group, 54: delay element, 64: reverse delay element, D: hidden neuron group Flop, E ... output neuron group.

Continuation of the front page (56) References JP-A-4-165400 (JP, A) JP-A-4-324500 (JP, A) JP-A-4-60600 (JP, A) IEICE technical report [ Voice] Vol. 96 No. 319 SP96-56 "Speech Recognition Based on Bidirectional Recurrent Neural Network" p. 7-12 (Oct. 18, 1996) Proceedings of the Autumn Meeting of the Acoustical Society of Japan 1996 I-3-2-15 "Bi-Directional Recurrence Neural Networks for Speech Recognition" p. 77-78 (September 25, 1996) Transactions of the Institute of Electronics, Information and Communication Engineers, Vol. J 73-D- ▲ II No. 5 May “Learning method for phonological spotting using time-delay neural network (TDNN) and its effects” (May 25, 1990) Proceedings of the Acoustical Society of Japan 1997 Spring Meeting, I- 3- 6-7 "Acoustic Models based on non-Uniform Segments and Bidirectional Citational Recurrent Neural Networks" p. 101-102 (March 17, 1997), Proceedings of the Acoustical Society of Japan. I ▼ 3-6-8 “Segment boundary estimation using a current neural network” p. 103-104 (March 17, 1997) IEICE Technical Report [Voice] Vol. 97 No. 114 SP97-15 “Application of phoneme boundary estimation and speech recognition using a recurrent neural network” p. 41-48 (1997/6/19) Proceedings of the Acoustical Society of Japan, Fall Meeting, 1997, I-Q 2-Q-10, "Automatic segmentation of speech using phoneme boundary estimation network" p. 135-136 (September 17, 1997) IEEE Transactions on Signal Processing, Vol. 45, no. 11, Nov member 1997, "Directional Recurring Neural Networks", p. 2673-2681 International Joint Conference on Neural Networks, 1989, Vol. 2, "Parallelism, Hierarchy, Scaling in Time-Delay Neural Networks for Spotting Japanese Phones / CV-Syllables", p. ▲ II ▼ -81 to ▲ II ▼ -88 (58) Fields investigated (Int. Cl. ⁷ , DB name) G10L 3/00 515 G10L 3/00 539 G10L 9/10 301 G06F 15/18 560 INSPEC (DIALOG ) JICST file (JOIS) WPI (DIALOG)

Claims (8)

(57) [Claims] An input layer, an intermediate layer having at least one layer having a plurality of units, and an output layer having one unit and outputting a phoneme boundary detection value representing a phoneme boundary detection probability. A phoneme boundary detection device for detecting a phoneme boundary of a speech feature parameter sequence using a directional recurrent neural network, wherein the input layer has a plurality of speech feature parameters as inputs and a first input having a plurality of units. A neuron group, a forward module, and a backward module, the forward module having a temporally forward feedback connection based on a plurality of speech feature parameters,
While generating a plurality of parameters at times delayed by a predetermined unit time from a plurality of parameters output from the input neuron group and outputting the plurality of parameters to the intermediate layer, the backward module is configured to generate a plurality of parameters based on a plurality of speech feature parameters. And has a temporally backward feedback connection,
And generating a plurality of parameters at times delayed by a predetermined unit time in a backward direction from a plurality of parameters output from the input neuron group of the plurality of input neuron groups and outputting the generated parameters to the intermediate layer. 2. The forward module receives a plurality of speech feature parameters as input, outputs a second input neuron group having a plurality of units, and is delayed by a predetermined unit time from a second intermediate neuron group and output. With multiple parameters as input,
A first intermediate neuron group having a plurality of units, a plurality of parameters output from the second input neuron group, and respective weighting factors for a plurality of parameters output from the first intermediate neuron group And a second intermediate neuron group having a plurality of units, each of which is connected so as to be input by multiplying by a plurality of units. The backward module has a plurality of speech feature parameters as inputs, and has a third unit having a plurality of units. An input neuron group, a third intermediate neuron group having a plurality of units, and having a plurality of units inputting a plurality of parameters delayed and output by a predetermined unit time in a backward direction from the fourth intermediate neuron group; A plurality of parameters output from the input neuron group of A fourth intermediate neuron group connected to each of the plurality of parameters output from the intermediate neuron group and multiplied by a respective weighting factor, and having a plurality of units; A plurality of parameters output from the neuron group are connected to each other by multiplying each of the plurality of parameters by the respective weighting factors and input to each of the plurality of units of the intermediate layer. Each of the parameters is multiplied by each weighting factor and connected so as to be input to a plurality of units of the intermediate layer, respectively, and each of the weighting factors is output to a plurality of parameters output from the fourth intermediate neuron group. Are connected so as to be input to a plurality of units of the intermediate layer, respectively, 2. The phoneme boundary detection device according to claim 1, wherein the plurality of parameters output from the intermediate layer are multiplied by the respective weighting factors, and the parameters are connected so as to be input to the units of the output layer. . 3. The apparatus according to claim 1, further comprising: a first detection unit that detects a phoneme boundary as a phoneme boundary when a phoneme boundary detection value output from the output layer is equal to or more than a predetermined threshold value. A phoneme boundary detection device according to the above. 4. The method according to claim 1, further comprising a second detecting unit that detects a phoneme boundary when the phoneme boundary detection value output from the output layer is equal to or more than a predetermined threshold value and reaches a maximum value. The phoneme boundary detection device according to claim 1 or 2, wherein 5. When a phoneme boundary detection value output from the output layer is equal to or greater than a predetermined first threshold, the phoneme boundary detection value is detected as a first phoneme boundary, and the phoneme boundary detection value is set to the first phoneme boundary.
A third detection means for detecting a second phoneme boundary when the second phoneme boundary is equal to or more than a second threshold value smaller than the threshold value and smaller than the first threshold value and reaches a maximum value. The phoneme boundary detection device according to claim 1 or 2, wherein: 6. The method according to claim 1, wherein the third detecting means selects one of the detected phoneme boundaries as the first phoneme boundary and selects one as a first phoneme boundary. The phoneme boundary detecting device according to claim 5, wherein 7. The third detection means detects a phoneme boundary based on a lattice of a path formed between the detected or selected first phoneme boundary and the second phoneme boundary. The phoneme boundary detection device according to claim 5 or 6, wherein 8. A feature extracting means for extracting a speech feature parameter from a speech signal of an uttered speech sentence comprising an input character string, and based on the speech feature parameter extracted by the feature extracting means. Speech recognition means for recognizing a speech signal of an uttered speech sentence composed of an input character string using a phoneme boundary detected by the phoneme boundary detection device described in any one of the above and a predetermined acoustic model. A voice recognition device comprising: