JP7160170B2 - Speech recognition device, speech recognition learning device, speech recognition method, speech recognition learning method, program - Google Patents

Description

The present invention relates to a speech recognition device, a speech recognition learning device, a speech recognition method, a speech recognition learning method, and a program.

Along with the progress of deep learning technology, a speech recognition modeling method called end-to-end speech recognition, which uses speech as input and text as output, has appeared, and technological progress is progressing. Speech recognition, which has been widely used so far, consists of three models: an acoustic model that models the relationship between speech and phoneme sequences, a pronunciation model that models the relationship between phoneme sequences and words, and a language model that models the relationships between words. A speech recognition algorithm (apparatus) was constructed by combining models and learning each model independently using different data. On the other hand, end-to-end speech recognition can configure a speech recognition algorithm (device) with only one model that models the relationship between speech and text, and the data used for learning is only paired data of speech and text. is.

The configuration of the conventional technology will be described. X=(x ₁ ,…,x _T ) is the acoustic feature sequence that can be automatically extracted from the input speech for end-to-end speech recognition, and W=(w ₁ ,…,w _N ) is the output word sequence. and model P(W|X, θ). where θ represents a model parameter. Modeling of P(W|X, θ) is expressed by the following equation.

In this modeled speech recognition algorithm (apparatus), the word sequence W^ of the speech recognition result when the acoustic feature quantity sequence X is input is determined based on the following equation.

The model parameter θ is learning data D=(W ₁ ,X ₁ ),...,(W _|D| ,X _|D| (where |D| is the number of elements in learning data D), and is determined by learning in advance. The parameter θ^ optimized by D obeys the following equation.

Various methods can be employed for detailed modeling. For example, a method using a neural network is typical, and the methods of Non-Patent Document 1 and Non-Patent Document 2 can be used.

Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, “End-to-end continuous speech recognition using attention-based recurrent NN: first results,” in NIPS: Workshop Deep Learning and Representation Learning Workshop, 2014. Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio, “Attention-based models for speech recognition,” in Advances in Neural Information Processing Systems (NIPS), 2015, pp. 577-585.

The above-mentioned prior art modeled the problem of recognizing a single utterance. In the case of recognition, no relations between utterances of multiple utterances can be exploited. That is, there is a problem that information such as what kind of word sequence has been output in response to voice input of past utterances cannot be taken into account when performing voice recognition of current utterances.

A specific example will be given for explanation. For example, in a scene where speech recognition is performed on speech of a lecture of about 10 minutes, it is assumed that the speech of the lecture is divided into 0.5 seconds of silence, and a total of 200 utterances are included. These 200 utterances are a continuous sequence, and it is highly likely that the continuous utterances are utterances about mutually related information. However, when the conventional technology is applied, 200 utterances are independently recognized, and context information cannot be used for speech recognition. For example, if the 100th utterance is "This year's business performance is wonderful," and the 101st utterance is "It's because it's wonderful." There is a high possibility that the utterance can be recognized as "a wonderful achievement", but if the 100th utterance cannot be considered as a context, the 101st utterance may be misrecognized as "wonderful confectionery" or "wonderful torch". have a nature.

For example, it is assumed that all the utterances (200 utterances in the above example) are collected and treated as one utterance with a long utterance length to solve the above problem. In this case, since the end-to-end speech recognition algorithm (apparatus) converts the entire speech into a vector and handles it, there arises a problem that it does not work well for long utterance lengths. It is unrealistic to treat all utterances collectively as one utterance with an end-to-end speech recognition algorithm (device). Therefore, in the past, there was a problem that end-to-end speech recognition considering the context could not be realized.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a speech recognition apparatus capable of realizing end-to-end speech recognition in consideration of context.

A speech recognition apparatus of the present invention includes an utterance speech recognition unit.

Based on recognition data consisting of a set of acoustic feature sequences acquired in chronological order, the utterance speech recognition unit uses a word sequence to be recognized as an observed value, and recognizes a word sequence that has already been recognized prior to the word sequence to be recognized. The maximum likelihood criterion for the likelihood function of the probability that an observed value occurs under the parameters of the word sequence that has already been processed, the acoustic feature value sequence corresponding to the word sequence to be recognized, and the model parameter θ that has been trained in advance. , the process of recognizing a word sequence to be recognized is repeated in chronological order.

According to the speech recognition apparatus of the present invention, it is possible to realize end-to-end speech recognition in consideration of context.

1 is a block diagram showing the configuration of a speech recognition apparatus according to a first embodiment; FIG. 4 is a flow chart showing the operation of the speech recognition apparatus according to the first embodiment; FIG. 2 is a block diagram showing the configuration of the speech recognition unit of the speech recognition apparatus according to the first embodiment; FIG. 4 is a flow chart showing the operation of the speech recognition unit of the speech recognition apparatus of the first embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail. Components having the same function are given the same number, and redundant description is omitted.

The speech recognition apparatus 1 (minimum configuration, see the configuration within the dashed frame in FIG. 1) of this embodiment will be described below. Here, it is assumed that the model parameter θ is learned in advance by a device other than the speech recognition device 1 .

It should be noted that, in this specification, characters may be followed by "^" due to the convenience of word processing software, but this "^" is treated as being displayed above the character. For example, if you write W ^L ^

shall mean

<Overview of input, output, and operation of speech recognition device 1 (minimum configuration)>
Input 1: Sequence X ¹ , …, X ^L of acoustic feature sequences of L consecutive utterances
Input 2: Model parameter θ (learned on another device and input to this device)
Output: A sequence of L consecutive word sequences W ¹ ^,...,W ^L ^

The speech recognition apparatus 1 of the present embodiment receives ^L sequences of acoustic feature quantity sequences X ¹ , . Output a series of consecutive word sequences W ¹ ^,...,W ^L ^. ^Here , let X ¹ , . Here, X ^l is the acoustic feature value sequence of the l-th utterance and is expressed as X ^l =(x ^l ₁ ,...,x ^l _Tl ). The sequence of output word sequences is W ¹ ^,...,W ^L ^, where W ^l ^ is the word sequence of the l-th utterance,

is represented as

Here, as the acoustic feature sequence, any feature sequence that can be calculated from speech can be used. For example, a feature sequence such as a mel filter bank cepstrum coefficient or a logarithmic mel filter bank can be used. A description of the Mel filter bank cepstrum coefficients and the logarithmic Mel filter bank is omitted.

The word sequence may be an expression separated by spaces in the case of English, an expression automatically divided by morphological analysis in the case of Japanese, or an expression separated by characters.

Next, with reference to FIG. 1, the general configuration of the speech recognition apparatus of Example 1 will be described. Here, it is assumed that the model parameter θ is learned within the speech recognition apparatus 1 . As shown in the figure, the speech recognition apparatus 1 of this embodiment includes a model parameter learning unit 11, a model parameter storage unit 11a, an utterance speech recognition unit 12, and a word sequence storage unit 12a. However, as described above, the model parameter learning section 11 and the model parameter storage section 11a may be components of separate devices. The operation of each component will be described below with reference to FIG.

<Model parameter learning unit 11>
Input: Learning data D=(A ₁ ,B ₁ ),…,(A _|D| ,B _|D| )
Output: model parameter θ

The model parameter learning unit 11 acquires learning data D=(A ₁ , B ₁ ), . Based on ( ^A _|D| ,B _|D| ), the word sequence of interest (W ¹ ,...,W ^l-1 ), the acoustic feature sequence (X ^l in the following equation) corresponding to the word sequence of interest (W ^l in the following equation), and the model parameter θ as parameters. ^{The model _ _} A parameter θ is learned (S11). Note that (A _m , B _m )={(W ¹ ,X ¹ ),...,(W ^Lm ,X ^Lm )}. The parameter θ^ optimized by D obeys the following equation.

θ ̂ learned here is used as θ in the speech recognition unit 12 .

<Model parameter storage unit 11a>
The model parameter storage unit 11a stores the learned θ̂.

<Speech speech recognition unit 12>
Input 1: Acoustic feature sequence X ^l of the l-th utterance
Input 2: Word sequence W ¹ ^,...,W ^l-1 ^ from the 1st to the l-1th utterance already obtained as a speech recognition result
Input 3: model parameter θ
Output: word sequence W ^l ^ of l-th utterance

The utterance speech recognition unit 12 recognizes a word sequence to be recognized ^{( W l} ) is the observed value, and the already recognized word sequence (W ¹ ^,...,W ^l-1 ^ in the following equation) that is older than the word sequence to be recognized (W ^l in the following equation), and the recognition target Using the acoustic feature value sequence (X ^l in the following equation) corresponding to the word sequence (W ^l in the following equation) and the trained model parameter θ as parameters, the parameters (W ¹ ^,...,W ^l in the following equation) ^{−1 ^} , X ^l , θ), the likelihood function of the probability of occurrence of the observed value (W ^l is repeated in chronological order (S12).

That is, the utterance speech recognition unit 12 recognizes the acoustic feature sequence X ^l of the l-th utterance and the recognized word sequence W ¹ ^, . . . When W ^l-1 ^ is input, the posterior probability distribution for the l-th utterance is calculated according to the model parameter θ

and determine the word sequence W ^l ^ of the speech recognition result of the l-th utterance by the maximum likelihood criterion. That is, determination by the maximum likelihood criterion follows the following equation.

As described above, the speech recognition unit 12 recursively executes step S12 in chronological order. For example, by taking the word sequence W ^l ^ of the speech recognition result of the l-th utterance as a known recognition result, the posterior probability distribution

Similarly, the word sequence W ^l+1 ^ of the speech recognition result of the l+1 th utterance is determined as follows.

note that,

The detailed formulation of and the detailed calculation method will be described later.

<Word sequence storage unit 12a>
The word sequence storage unit 12a stores word sequences that the spoken voice recognition unit 12 recursively uses. For example, in step S12, when the word sequence W ¹ ^ is recognized, the word sequence storage unit 12a stores the word sequence W ¹ ^, and when the word sequence W ^l ^ is recognized, the word sequence storage unit 12a stores the word sequence W 1 ^. stores the word sequence W ^l ̂, and when the word sequence W ^L ̂ is recognized, the word sequence storage unit 12a stores the word sequence W ^L ̂.

<Detailed Configuration of Spoken Voice Recognition Unit 12>
As shown in FIG. 3 , the utterance speech recognition unit 12 includes an utterance vector calculation unit 121 , an utterance sequence embedding vector calculation unit 122 , a context vector calculation unit 123 and a posterior probability calculation unit 124 .

As described above, the speech recognition unit 12

to calculate This detailed formulation is represented by the following equation.

note that,

is realized by the utterance vector calculator 121 , the utterance sequence embedding vector calculator 122 , the context vector calculator 123 , and the posterior probability calculator 124 in the uttered speech recognizer 12 . In the following, referring to Fig. 4, the probability for the nth word of the lth utterance

represents the detailed processing for computing

<Utterance vector calculator 121>
Input 1: word sequence W ^l-1 ^ of the l-1th utterance
Input 2: model parameter θ
Output: utterance vector u ^l-1 of the l-1th utterance

The utterance vector calculation unit 121 converts the word sequence W ^l-1 ^ of the already recognized l-1th utterance, which is older than the word sequence W ^l of the l-th utterance to be recognized, based on the model parameter θ. The function is used to convert the l-1th utterance into an utterance vector u ^l-1 (S121). At this time, the word sequence W ^l-1 ^ of the l-1-th utterance contains one or more words. An utterance vector represents a vector in which information contained in a word sequence is embedded, and semantic information of the utterance necessary for speech recognition of the next utterance is embedded. As the number of dimensions of the vector increases, more information can be embedded. For example, the number of dimensions is determined manually as a 512-dimensional vector. At this time, any conversion function can be used as long as it is a function that converts a variable-length symbol string into a single vector. A recurrent neural network, a bidirectional recurrent neural network, or the like can also be used.

Note that when l=1, there is no input word sequence W ⁰ , so the output u ⁰ should be a vector whose elements are all 0.0.

Note that step S121 is performed for each of W ¹ ̂, . . . , W ^1-1 ̂. Therefore, the utterance vector calculator 121 outputs u ¹ , . . . , u ^{1 -1} .

<Utterance sequence embedding vector calculator 122>
Input 1: Sequence of utterance vectors u ¹ ,...,u ^l-1 for past utterances
Input 2: model parameter θ
Output: l-1th utterance sequence embedding vector v ^l-1

^The utterance sequence embedding vector calculation unit 122 converts the utterance vector sequence ^{u 1} , . (S122). This utterance sequence embedding vector is a single vector in which semantic information necessary for speech recognition of the next utterance is embedded. As the number of dimensions of the vector increases, more information can be embedded. For example, the number of dimensions is determined manually as a 512-dimensional vector. At this time, any conversion function can be used as long as it is a function that converts a variable-length vector sequence into a single vector. functions can be used. When averaging, the number of dimensions of the utterance sequence embedding vector depends on the number of dimensions of the utterance vector sequence.

When l=1, there is no utterance vector sequence for the input past utterance sequence, so the output ^v0 should be a vector whose elements are all 0.0.

<Context Vector Calculator 123>
Input 1: Word string w ^l ₁ ,...,w ^l _n-1 past n-th word w ^l _n in word string W ^l of l-th utterance
Input 2: Acoustic feature sequence X ^l of the l-th utterance
Input 3: model parameter θ
Output: the context vector s ^l _n for the nth word of the lth utterance

The context vector calculation unit 123 calculates word sequences w ^{l 1} , . ^{. . , w l n} _{- 1 (} word sequence and the l-th acoustic feature value sequence X ^l corresponding to the l-th word sequence W ^l to be recognized is converted to the l-th utterance by a conversion function based on the model parameter θ to the context vector s ^l _n for the _n -th word w ^l n in the word sequence W ^l (S123). This context vector is embedded with integrated semantic and phonological information required for speech recognition of the next word. At this time, any conversion function can be used as long as it is a function that converts two types of variable-length vector sequences into a single vector. It is also possible to provide a recurrent neural network for each of the , and add an attention mechanism to use a function that is represented as a single context vector. In the simplest case, a combination vector of the frequency vector of the word sequence past the n-th word of the l-th utterance and the vector obtained by averaging the acoustic feature value sequence of the l-th utterance is constructed. function can also be used.

<Posterior probability calculator 124>
Input 1: l-1th utterance sequence embedding vector v ^l-1
Input 2: the context vector s ^l _n for the nth word of the lth utterance
Input 3: model parameter θ
Output: posterior probability for the nth word of the lth utterance

^The posterior probability calculation unit 124 converts the ^utterance vector sequences ^{u 1} , . ^-1 and the context vector s ^l _n for the n-th word of the l-th word sequence W ^l to be recognized, the n-th word of the l-th word sequence W ^l is converted by a transformation function based on the model parameter θ the posterior probability about

is calculated (S124). The posterior probability can be expressed as a vector with each word as an element, and the posterior probability distribution can be expressed by vector conversion. At this time, any conversion function can be used as long as it is a function that converts two types of vectors into a posterior probability distribution. It can be realized by a function that performs Other than that, it is possible to apply a function that can convert the sum of the elements of the output vector corresponding to the posterior probability distribution to 1.0.

According to the speech recognition apparatus 1 of the present embodiment, instead of conventional end-to-end speech recognition that handles a single utterance, modeling of end-to-end speech recognition that handles an utterance sequence is introduced. Context-aware end-to-end speech recognition can be achieved when the input is represented as an utterance sequence. That is, when recognizing an utterance in an utterance sequence, information from the first utterance in the utterance sequence to the utterance immediately preceding the target utterance can be used as a context. For example, in the same way as described above, it is assumed that speech recognition is performed on speech of a lecture of about 10 minutes, and that speech of 200 utterances is included when this speech is segmented at intervals of 0.5 seconds of silence. In this case, according to the speech recognition apparatus 1 of this embodiment, all related contextual information prior to a certain utterance in the 200 consecutive utterances can be used for current speech recognition. For example, when recognizing the 100th utterance, the speech recognition apparatus 1 can use the speech recognition results of the 1st to 99th utterances as the context.

The speech recognition apparatus 1 of this embodiment can improve the recognition performance of speech recognition for lectures, telephone calls, meetings, and the like.

<Addendum>
The apparatus of the present invention includes, for example, a single hardware entity, which includes an input unit to which a keyboard can be connected, an output unit to which a liquid crystal display can be connected, and a communication device (for example, a communication cable) capable of communicating with the outside of the hardware entity. can be connected to the communication unit, CPU (Central Processing Unit, which may include cache memory, registers, etc.), memory RAM and ROM, external storage device such as hard disk, input unit, output unit, communication unit , a CPU, a RAM, a ROM, and a bus for connecting data to and from an external storage device. Also, if necessary, the hardware entity may be provided with a device (drive) capable of reading and writing a recording medium such as a CD-ROM. A physical entity with such hardware resources includes a general purpose computer.

The external storage device of the hardware entity stores a program necessary for realizing the functions described above and data required for the processing of this program (not limited to the external storage device; It may be stored in a ROM, which is a dedicated storage device). Data obtained by processing these programs are appropriately stored in a RAM, an external storage device, or the like.

In the hardware entity, each program stored in an external storage device (or ROM, etc.) and the data necessary for processing each program are read into the memory as needed, and interpreted, executed and processed by the CPU as appropriate. . As a result, the CPU realizes a predetermined function (each component expressed as above, . . . unit, . . . means, etc.).

The present invention is not limited to the above-described embodiments, and can be modified as appropriate without departing from the scope of the present invention. Further, the processes described in the above embodiments are not only executed in chronological order according to the described order, but may also be executed in parallel or individually according to the processing capacity of the device that executes the processes or as necessary. .

As described above, when the processing functions of the hardware entity (apparatus of the present invention) described in the above embodiments are implemented by a computer, the processing contents of the functions that the hardware entity should have are described by a program. By executing this program on a computer, the processing functions of the hardware entity are realized on the computer.

A program describing the contents of this processing can be recorded in a computer-readable recording medium. Any computer-readable recording medium may be used, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or the like. Specifically, for example, magnetic recording devices include hard disk devices, flexible discs, and magnetic tapes, and optical discs include DVDs (Digital Versatile Discs), DVD-RAMs (Random Access Memory), CD-ROMs (Compact Disc Read Only). Memory), CD-R (Recordable) / RW (ReWritable), etc. as magneto-optical recording media, such as MO (Magneto-Optical disc), etc. as semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. can be used.

Also, the distribution of this program is carried out by selling, assigning, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Further, the program may be distributed by storing the program in the storage device of the server computer and transferring the program from the server computer to other computers via the network.

A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or the program transferred from the server computer once in its own storage device. Then, when executing the process, this computer reads the program stored in its own recording medium and executes the process according to the read program. Also, as another execution form of this program, the computer may read the program directly from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to this computer. Each time, the processing according to the received program may be executed sequentially. In addition, the above processing is executed by a so-called ASP (Application Service Provider) type service, which does not transfer the program from the server computer to this computer, and realizes the processing function only by the execution instruction and result acquisition. may be It should be noted that the program in this embodiment includes information that is used for processing by a computer and that conforms to the program (data that is not a direct instruction to the computer but has the property of prescribing the processing of the computer, etc.).

Moreover, in this embodiment, the hardware entity is configured by executing a predetermined program on the computer, but at least part of these processing contents may be implemented by hardware.

Claims (7)

Based on recognition data consisting of a set of acoustic feature sequences acquired in chronological order, a word sequence to be recognized is taken as an observed value, and a word sequence that has already been recognized prior to the word sequence to be recognized, and Using the acoustic feature value sequence corresponding to the word sequence to be recognized and the pre-trained model parameter θ as parameters, the likelihood function of the probability that the observed value occurs under the parameters is calculated according to the maximum likelihood criterion. A speech recognition device including an utterance speech recognition unit that repeats processing for recognizing a word sequence in chronological order. The speech recognition device according to claim 1,
The utterance speech recognition unit
An utterance in which an already recognized word sequence that is older than the word sequence to be recognized is converted into an utterance vector, which is a vector containing semantic information necessary for speech recognition of the next utterance, by a conversion function based on the model parameter θ. a vector calculator;
an utterance sequence embedding vector calculator that converts the utterance vector sequence into an utterance sequence embedding vector containing semantic information necessary for speech recognition of the next utterance by a conversion function based on the model parameter θ;
A word string in the word sequence to be recognized that is older than the word of interest in the word sequence to be recognized, and an acoustic feature amount sequence corresponding to the word sequence to be recognized are used as the model parameters. a context vector calculation unit that converts a context vector containing information that integrates semantic information and phonological information required for speech recognition of words in a word sequence to be recognized using a conversion function based on θ;
Transformation based on the model parameter θ from the utterance sequence embedding vector obtained by transforming the utterance vector sequence up to one past the word sequence to be recognized and the context vector for the word in the word sequence to be recognized A speech recognition apparatus including a posterior probability calculator that calculates posterior probabilities of words in a word sequence to be recognized by a function. Based on learning data consisting of sets of word sequences acquired in chronological order and pairs of corresponding acoustic feature value sequences, the word sequence of interest is taken as an observed value, and word sequences older than the word sequence of interest are extracted. , and the acoustic feature value sequence corresponding to the word sequence of interest, and the model parameter θ as parameters, and by performing maximum likelihood estimation on the likelihood function of the probability that the observed value occurs under the parameters, the above A speech recognition learning device including a model parameter learning unit for learning a model parameter θ. Based on recognition data consisting of a set of acoustic feature sequences acquired in chronological order, a word sequence to be recognized is taken as an observed value, and a word sequence that has already been recognized prior to the word sequence to be recognized, and Using the acoustic feature value sequence corresponding to the word sequence to be recognized and the pre-trained model parameter θ as parameters, the likelihood function of the probability that the observed value occurs under the parameters is calculated according to the maximum likelihood criterion. A speech recognition method including an utterance speech recognition step of repeating a process of recognizing a word sequence in chronological order. Based on learning data consisting of sets of word sequences acquired in chronological order and pairs of corresponding acoustic feature value sequences, the word sequence of interest is taken as an observed value, and word sequences older than the word sequence of interest are extracted. , and the acoustic feature value sequence corresponding to the word sequence of interest, and the model parameter θ as parameters, and by performing maximum likelihood estimation on the likelihood function of the probability that the observed value occurs under the parameters, the above A speech recognition learning method including a model parameter learning step of learning a model parameter θ. A program that causes a computer to function as the speech recognition device according to claim 1 or 2. A program that causes a computer to function as the speech recognition learning device according to claim 3.

Images