JP2019120841A - Speech chain apparatus, computer program, and dnn speech recognition/synthesis cross-learning method - Google Patents

Abstract

To provide a speech chain device for reproducing a mechanism of a speech chain of a human by a machine.SOLUTION: A speech chain apparatus (100) includes a DNN speech recognition part (10) for inputting speech feature sequence data and outputting character sequence data, a DNN speech synthesis part (20) for inputting character sequence data and outputting speech feature sequence data, a speech feature extraction part (30) for generating speech feature sequence data from input speech, and a text feature extraction part (40) for generating character sequence data from input text are generated, a first learning control part (70) learns the speech recognition part by using the speech feature sequence data outputted from the speech synthesis part as learning data and character sequence data generated by the text feature extraction part as teacher data, and a second learning control part (80) learns the speech synthesis part by using character sequence data outputted from the speech recognition part as learning data and speech feature sequence data generated by the speech feature extraction part as teacher data.SELECTED DRAWING: Figure 6

Description

  The present invention relates to automatic speech recognition (ASR) and automatic text-to-speech synthesis (TTS), and in particular, to a speech recognition unit and a speech constructed with Deep Neural Network (DNN). The present invention relates to a technology to make a synthesis unit learn from each other.

  In recent years, speech and language information processing technology by ASR and TTS has been developed, and machines and humans can communicate through speech. As far as ASR is concerned, strict statistical models such as template-based schemes by dynamic time warping (DTW) and hidden Markov mixed Gaussian models (HMM-GMM) Approaches based on acoustic phonetics have been tried, such as data driven methods by. Speaking of TTS, from rule-based system by waveform coding and analysis synthesis method, there is more freedom by using waveform segment connection method and hidden semi Markov mixed Gaussian model (HSMM-GMM: hidden semi-Markov model-GMM) It is shifting to the method.

  And, due to significant performance improvement of computer hardware in recent years, DNN can be put into practical use in various fields, and deep learning using DNN is being adopted in ASR and TTS (see, for example, Non-Patent Documents 1 and 2 below) ).

W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 4960-4964. Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, RJ Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio et al., “Tacotron: A full end -to-end text-to-speech synthesis model, "arXiv preprint arXiv: 1703.10135, 2017.

  People utter words while listening to their own voices. That is, based on the volume, tone and clarity of the voice of the human being heard from the ear, the human brain decides what kind of utterance to make next and gives instructions to the vocal organs. Thus, in human speech recognition and speech speech, a speech chain that is a closed loop consisting of a hearing organ, a brain and a vocal organ plays a very important role. For example, it is known that children who have deafened can not cope well by the inability of the speech chain. Thus, although human speech recognition and speech utterance are closely related to each other, research and development of ASR and TTS have independently advanced.

The separation of AST and TTS has not changed since deep learning using DNN. And, the following problems occur because ASR and TTS are separated.
1. It is necessary to prepare a large amount of supervised data consisting of speech and text pairs in order to train ASR and TTS to sufficient levels respectively. Since supervised data must be created manually, it takes a great deal of effort and cost.
2. In the actual inference stage, noise is mixed into the signal input on-line, which may result in increase in output error of learned ASR and TTS or inability to obtain an output. Therefore, it is necessary to re-learn ASR and TTS based on the signal input online, but the signal input online is originally unsupervised data, and a mechanism for learning ASR and TTS using unsupervised data is established. Not.

  SUMMARY OF THE INVENTION In view of the above problems, the present invention aims to provide a speech chain device that reproduces the mechanism of human speech chain by machine.

  According to one aspect of the present invention, a speech recognition unit constructed of a deep neural network which receives speech feature series data as an input and outputs character series data, and a deep neural network which receives character series data as an input and outputs speech feature series data A speech synthesis unit constructed on a network, a speech feature extraction unit processing the inputted speech to generate the speech feature series data input to the speech recognition unit, and the speech output from the speech recognition unit A text generation unit that generates a text corresponding to the voice input to the voice feature extraction unit based on the character sequence data; and the text sequence data input to the voice synthesis unit by processing the input text A text feature extraction unit for generating the text feature, and the text feature extraction unit based on the voice feature series data output from the voice synthesis unit. A voice generation unit that generates a voice corresponding to the text input to the extraction unit; and the voice feature series data output from the voice synthesis unit as learning data to the voice recognition unit, the text feature extraction unit A first learning control unit that causes the voice recognition unit to learn using the generated character sequence data as teacher data, and the character sequence data output from the voice recognition unit is input to the voice synthesis unit as learning data And a second learning control unit for learning the voice synthesis unit using the voice feature series data generated by the voice feature extraction unit as training data.

  Specifically, the speech feature series data input to the speech recognition unit may be a mel spectrum feature quantity, and the speech feature series data output from the speech synthesis unit may be a linear spectrum feature quantity and a mel spectrum The feature amount may be used, and the voice feature extraction unit may generate, as the voice feature sequence data, linear spectral feature amounts and mel spectral feature amounts of the voice input to the voice feature extraction unit. Preferably, the second learning control unit may learn the speech synthesis unit using the linear spectrum feature and the mel spectrum feature generated by the speech feature extraction unit as training data. .

  Also, specifically, the speech synthesis unit may have an output layer representing the probability of termination of the utterance, and the second learning control unit further uses the probability of termination of the utterance as teacher data. It may be used to learn the speech synthesis unit.

  Also, specifically, the voice synthesis unit may have an input layer to which speaker identification information is input.

  According to another aspect of the present invention, there is provided a computer program for causing a computer to realize each component of the speech chain device.

  According to still another aspect of the present invention, a speech recognition unit constructed of a deep neural network that receives speech feature series data as an input and character series data as an output, and character series data as an input takes speech feature series data as an output. A DNN speech recognition / synthesizing mutual learning method for mutually learning a speech synthesis unit constructed by a deep neural network, wherein when speech / text pairs are given as supervised data, speech feature series data of the speech is The voice recognition unit is input as learning data, and the voice recognition unit is trained using character series data of the text as teacher data, and character series data of the text is input as learning data to the voice synthesis unit. The speech synthesis unit is operated using speech feature series data of the speech as teacher data. In the first step to learn and when only voice is given as unsupervised data, voice feature series data of the voice is inputted to the voice recognition unit, and character sequence data outputted from the voice recognition unit is learned data The second step of learning the speech synthesis unit using the speech feature series data of the speech as teacher data, and the speech synthesis when only the text is given as unsupervised data The character sequence data of the text is input to the unit, and the voice feature sequence data output from the voice synthesis unit is input as learning data to the voice recognition unit, and the character sequence data of the text is used as teacher data A third step of training a recognition unit is provided. A DNN speech recognition / synthesis mutual learning method is provided.

  The DNN speech recognition and synthesis mutual learning method includes a fourth step of extracting a certain amount of each type of data from a data set in which three types of data of speech and text, text only and speech only are mixed, and the data And a fifth step of performing batch learning of the speech recognition unit and the speech synthesis unit by sequentially repeating the first step to the third step using each type of data extracted from a set. It is also good.

  According to the present invention, the mechanism of the human speech chain can be reproduced on a machine. As a result, online learning of speech synthesis and speech recognition can be performed using speech input for speech recognition and text input for speech synthesis as unsupervised data, and speech as supervised data can be obtained. You can reduce the effort and cost of preparing a large number of text pairs. Furthermore, as the speech chain device according to the present invention is used as a speech recognition device and a speech synthesis device, learning progresses and the accuracy of speech recognition and speech synthesis improves as it is used.

Figure showing the architecture of the machine speech chain on which the speech chain device according to the invention is based A schematic diagram showing how ASR is learned using machine output of ASR as TTS learning data in a machine speech chain A schematic diagram showing how to learn TTS by using the output of TTS as learning data of ASR in a machine speech chain Schematic diagram of DNN speech recognition model according to an example Schematic diagram of DNN speech synthesis model according to an example Block diagram of speech chain device according to one embodiment of the present invention Flow chart of voice feature series data generation process Flow chart of character series data generation process Overall flow chart of DNN speech recognition and synthesis mutual learning Flow chart of batch learning process of ASR and TTS

  Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters and redundant description of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art.

  It is noted that the inventors provide the attached drawings and the following description so that those skilled in the art can fully understand the present invention, and intend to limit the subject matter described in the claims by these is not.

  Also, it should be noted that when simply referred to herein as "voice", it refers to a voice waveform signal.

1. Machine Speech Chain
FIG. 1 shows the architecture of the machine speech chain on which the speech chain device according to the invention is based. The machine speech chain 1 is a machine reproduction of the mechanism of the human speech chain described above, and may be called a speech recognition unit (hereinafter simply referred to as "ASR") which receives speech x and converts it into text y ^{^} And a speech synthesis unit (which may be simply referred to as “TTS” hereinafter) 20 that receives text y and converts it into speech x ^{^} . In the machine speech chain 1, ASR10 and TTS20 are connected to each other so that the output (text y ^{^} ) of ASR10 is input to TTS20 and the output of TTS20 (voice x ^{^} ) is input to ASR10 to form a closed loop. There is.

  Both ASR 10 and TTS 20 are configured as a sequence-to-sequence type model in which sequence data are input / output. Specifically, the ASR 10 is configured as a model having voice feature series data as an input and character series data as an output, and the TTS 20 is configured as a model having character series data as an input and voice feature series data as an output. By configuring both the ASR 10 and the TTS 20 as a sequence-to-sequence type model as described above, it is possible to input one output to the other between the ASR 10 and the TTS 20.

Further, by forming the closed loop of ASR 10 and TTS 20 in the machine speech chain 1, it becomes possible to learn each model by using the output of one model as learning data of the other model. For example, ASR10 can be learned using speech x ^{^} output from TTS 20 in the process of speech synthesis processing as learning data of ASR 10, and conversely, the text y ^{^} output from ASR 10 in the process of speech recognition processing can be TTS 20 The TTS 20 can be learned by using it as learning data of

FIG. 2 is a schematic diagram showing how the TTS 20 is learned using the output of the ASR 10 as learning data of the TTS 20 in the machine speech chain 1. As shown in FIG. 2, when speech recognition processing is performed in machine speech chain 1, ASR 10 receives speech x and converts it into text y ^{^} . TTS 20 receives the text y ^{^} converted by ASR 10 and reconverts it into speech x ^{^} . At this time, using the text y ^{^} converted by ASR 10 as training data and the original speech x input to ASR 10 as teaching data, the error between the output of TTS 20 (speech x ^{^} ) and teaching data (speech x) is The parameter adjustment of TTS 20, that is, deep learning is performed so as to be smaller (so that the value of the loss function LostTTS (x, x ^{^} ) becomes smaller).

FIG. 3 is a schematic diagram showing how the output of the TTS 20 is used as learning data of the ASR 10 in the machine speech chain 1 to make the ASR 10 learn. As shown in FIG. 3, when speech synthesis processing is performed in machine speech chain 1, TTS 20 receives text y and converts it into speech x ^{^} . The ASR 10 receives the speech x ^{^} converted by the TTS 20 and converts it back to the text y ^{^} . At this time, the speech x ^{^} converted by TTS 20 is used as learning data, and the original text y input to TTS 20 is used as teacher data, and the output of ASR 10 (text y ^{^} , more specifically, each character constituting y ^{^} Parameter adjustment of ASR 10, that is, deep learning is performed so that the error between the probability of occurrence p _y ) and the teacher data (text y) becomes smaller (the value of loss function LostASR (y, p _y ) becomes smaller) .

  If the speech recognition model and the speech synthesis model are not interconnected as in the conventional case, a pair of speech and text is prepared as supervised data, and each model is learned offline (the speech is for speech recognition model learning) Learning data and text are used as teacher data, and for speech synthesis model learning, text needs to be used as learning data and speech is used as teacher data. On the other hand, machine speech chain 1 can not only learn ASR 10 and TTS 20 offline in teacher-forcing mode using supervised data, but also using speech input online for speech recognition. The TTS 20 can be trained, and the ASR 10 can be trained using text input online for speech synthesis. That is, the machine speech chain 1 can learn ASR 10 and TTS 20 online while performing speech recognition or speech synthesis.

2. DNN Speech Recognition / Synthesis Model Next, the details of the ASR 10 and TTS 20 constituting the machine speech chain 1 will be described. In the embodiment of the present invention, both ASR 10 and TTS 20 are constructed with a deep neural network (DNN).

First, the DNN speech recognition model will be described. FIG. 4 is a schematic view of a DNN speech recognition model according to an example. In the ASR 10, speech x is speech feature series data of length S (ie, x = [x ₁ ,..., X _S ]), text y is character series data of length T (ie, y = [y ₁ ,. _It is comprised as a sequence-to-sequence type | mold model which calculates | requires conditional probability p (y | x) when setting it as _T ]. Specifically, the ASR 10 can be configured as an auto encoder that applies a recursive neural network (RNN: Recurrent Neural Network). Each element x _s of the voice feature series data is a D-dimensional real value vector. Each element y _t of the character series data is a phoneme (phoneme) or a grapheme (grapheme).

More specifically, the ASR 10 includes an encoder 11, a decoder 12, and an attention 13. The encoder 11 includes three layers of Bidirectional Long Short-Term Memory (LSTM) layers 111, 112, 113. In the encoder 11, the audio feature series data _x 1 is represented by the logarithmic Mel spectral feature quantity to the first stage of the bidirectional LSTM layer 111, ..., _{x S} is inputted intermediate layer vector from the bidirectional LSTM layer 113 of the final stage ^{h e} _s (s = 1,..., S) is output.

The decoder 12 includes a Character Embed (Char Emb.) Layer 121 and an LSTM layer 122. In the decoder 12, character series data _y 0 to a character embedding layer 121, _..., character series data _y 1 from _{y T-1} is input LSTM layer 122, ..., _{y T} is output. Character series data y _t which is an input of the decoder 12 is not a phoneme or grapheme itself but an id or index number of a phoneme or grapheme. The output y _t of the decoder 12 at time t is to weight the vector obtained by connecting the context vector c _t that is calculated by the intermediate layer vector h ^d _t and Attention 13 output from LSTM layer 122 at a predetermined linear operator, further It is calculated by inputting it to a predetermined activation function. Although not shown, the character series data _y 1 output from LSTM layer 122, ..., _{y T} is the probability of occurrence of each character by softmax function _p y1, _..., it is normalized as _{p yT.}

Attention 13 is a module for calculating a context vector _{c t.} More specifically, the attention 13 is an intermediate layer vector h ^d _{t at} time t output from the L STM layer 122 of the decoder 12 and an intermediate layer vector h ^e ₁ ,... Held by the bidirectional L STM layer 113 of the encoder 11. , calculated from ^h _{e S} value _{a t,} further the value _{a t} the intermediate layer vector ^h _e 1, ^..., to calculate the context vector _{c t} from ^h _{e S.} The calculation formula for the value a _t and the context vector c _t is omitted the description here since it is well known.

ここで、Ｃは出力クラスの数であり、ｙは正解（ground truth）のテキストである。 The parameters of the ASR 10 are adjusted using a stochastic gradient descent method, an error back propagation method, or the like so that the value of the next loss function is minimized.

Here, C is the number of output classes and y is the text of the ground truth.

  When a pair of voice and text is given as supervised data at the time of offline learning, learning of ASR 10 can be performed using voice feature series data of the voice as learning data and character series data of the text as teacher data. On the other hand, when only text is given as unsupervised data, for example, when using text input online for speech synthesis, as described with reference to FIG. 3, the voice feature series data output from TTS 20 is Learning of ASR 10 can be performed using learning data and character series data of text input online for speech synthesis as teaching data.

Next, the DNN speech synthesis model will be described. FIG. 5 is a schematic view of a DNN speech synthesis model according to an example. TTS 20 is text y, character series data of length T (ie, y = [y ₁ ,..., Y _T ]), speech x is speech feature series data of length S (ie, x = [x ₁ ,. It is constructed as a sequence-to-sequence type model for obtaining the conditional probability p (x | y) when _S ] is taken. Specifically, the TTS 20 can be constructed as an auto encoder to which a recursive neural network is applied. Each element x _s of the voice feature series data is a D-dimensional real value vector. Each element y _t of the character series data is a phoneme or a grapheme.

More specifically, the TTS 20 includes an encoder 21, a decoder 22, and an attention 23. The encoder 21 includes a character embedding layer 211, a fully connected (FC: Fully Connected) layer 212, and a CBHG (1-D Convolution Bank + Highway network + bidirectional GRU) 213. In the encoder 21, the character series data _y 1 to a character embedding layer 211, ..., an intermediate layer vector ^h _e t from _{y T} is input CBHG213 (t = 1, ..., T) are output. Character series data y _t which is an input of the encoder 21 is not a phoneme or grapheme itself but an id or index number of a phoneme or grapheme.

The decoder 22 comprises an all coupled layer 221, an LTM layer 222, a CBHG 223, and an all coupled layer 224. In the decoder 22, speech feature series data x ^M ₀ ,..., X ^M _S-1 represented by logarithmic mel spectral features are input to all combined layers 221 and are represented by logarithmic mel spectral features from the LSTM layer 222 Speech feature series data x ^M ₁ ,..., X ^M _S are output. Further, in the decoder 22, speech feature series data x ^M ₁ ,..., X ^M _S outputted from the LSTM layer 222 are inputted to the CBHG 223 and speech feature series data x represented by linear spectrum feature quantities from all the combined layers 224 ^R _s (s = 1,..., S) is output. The input x ^M _s of the CBHG 223 weights a vector obtained by connecting the intermediate layer vector h ^d _s output from the LSTM layer 222 and the context vector c _s calculated by the attention 23 with a predetermined linear operator, and further weights the vector Calculated by inputting into the activation function of Although not shown, the speech feature series data x ^R _s (s = 1,..., S) output from all the combined layers 224 is processed according to the Griffin-Lim algorithm to reconstruct speech.

The decoder 22 further includes an output layer 225 and an input layer 226. The output layer 225 is a layer that outputs the probability of the end of the speech. The reason why the output layer 225 is provided is that voice feature series data voice feature series data x ^M _s and x ^R _s (s = 1,..., S) output from the decoder 22 are all real value vectors, Is because the end of the utterance can not be determined. If the output layer 225 does not exist, the end of the speech can not be determined, and the speech synthesis is forcibly terminated when the speech feature series data output from the TTS 20 reaches a predetermined length, which may result in unnatural endings. is there. On the other hand, the provision of the output layer 225 makes it possible to determine the end of the speech, so that speech synthesis can be ended at the end of the speech without forcibly terminating the speech feature series data with a predetermined length, which is natural The speech of the end can be synthesized.

  The input layer 226 receives speaker identification information. The speaker's id can be used as the speaker's identification information. The id of the speaker input to the input layer 226 is input to the embed function, converted to a distributed representation by a real-valued vector, and input to the LSTM layer 222, 224 or the like. In order to be able to cope with unknown speakers, the ids of the speakers may be mapped to an i-vector for speaker recognition. By providing the input layer 226 to which the identification information of the speaker is input as described above, the TTS 20 can synthesize a voice similar to the voice of the speaker.

  As described above, in the online learning of ASR 10, the voice feature series data output from TTS 20 is used as learning data, but at this time it is assumed that the voice quality of the voice input for voice recognition and the voice synthesized by TTS 20 are different. There is a risk that learning on the ASR 10 will not progress properly. Therefore, by providing the input layer 226 so that speech similar to the voice of the speaker can be synthesized, the quality of online learning of the ASR 10 can be improved. It is desirable to provide an input layer 226, especially when machine speech chain 1 must recognize the speech of multiple speakers.

The attention 23 is a module that calculates a context vector c _s . More specifically, the attention 23 is an intermediate layer vector h ^d _{s at} time s output from the L STM layer 222 of the decoder 22 and an intermediate layer vector h ^e ₁ ,..., H ^e _T held by the CBHG 213 of the encoder 21. the value _{a s} is calculated from the further values _{a s} and the intermediate layer vector ^h _e 1, ^..., to calculate the context vector _{c s} from ^h _{e T.} The formulas for calculating the value a _s and the context vector c _s are well known, and therefore the description thereof is omitted here.

ここで、ｘ^＾M、ｘ^＾Ｒ、ｂ^＾はそれぞれＴＴＳ２０から出力される対数メルスペクトル特徴量、リニアスペクトル特徴量、発話終端確率であり、ｘ^M、ｘ^Ｒ、ｂはそれぞれそれらの正解（ground truth）である。 The parameters of TTS 20 are adjusted using a stochastic gradient descent method or an error back propagation method so that the value of the next loss function becomes minimum.

Here, x ^{^ M} , x ^{^ R} and b ^{^} are the log mel spectral feature, linear spectral feature and speech termination probability output from TTS 20, and x ^M , x ^R and b are their correct answers ( ground truth).

  When a pair of speech and text is given as supervised data during offline learning, character series data of the text is learned data, and speech feature series data of the speech (log mel spectral feature quantity and linear spectrum feature quantity) is taken as teaching data It can be used to learn TTS 20. On the other hand, when only speech is given as unsupervised data, for example, when speech input online for speech recognition is used, as described with reference to FIG. 2, the character series data output from ASR 10 is learned. TTS 20 can be learned using data and voice feature series data (log mel spectral feature and linear spectral feature) of speech input online for speech recognition as teacher data.

3. Embodiment Next, the configuration of a speech chain apparatus according to an embodiment of the present invention will be described. FIG. 6 is a block diagram of a speech chain device 100 according to an embodiment of the present invention. The speech chain apparatus 100 includes a speech recognition unit (ASR) 10, a speech synthesis unit (TTS) 20, a speech feature extraction unit 30, a text generation unit 40, a text feature extraction unit 50, and a speech generation unit 60. An ASR learning control unit 70 and a TTS learning control unit 80 are provided. The elements that make up speech chain device 100 can be implemented as hardware or software or a combination thereof. For example, by installing dedicated computer software in a computer device such as a personal computer or a smartphone, the computer device can function as the speech chain device 100. For example, the speech chain apparatus 100 may be implemented on a server on the cloud and implemented as software as a service (SaaS). Also, the speech chain device 100 can be realized by distributing the components of the speech chain device 100 to a plurality of computer devices and connecting the components with each other via a telecommunication network. The ASR 10 and TTS 20 requiring a large amount of calculation may be processed by a dedicated processor such as a GPU (Graphics Processing Unit), and the other components may be processed by a CPU (Central Processing Unit).

  Next, details of each component of the speech chain apparatus 100 will be described. In addition, since ASR10 and TTS20 are as having mentioned above, repeated description is abbreviate | omitted.

The audio feature extraction unit 30 is a module that processes the input audio and generates audio feature series data (x = [x ₁ ,..., X _S ]) input to the ASR ₁₀ . The text generation unit 40 generates a text corresponding to the voice input to the voice feature extraction unit 30 based on the character series data (y ^{^} = [y ₁ ,..., Y _T ]) output from the ASR ₁₀ . It is. The voice feature extraction unit 30 can input voices collected by a microphone (not shown) in real time, and can also input voice recordings etc. held in a storage device or a memory device (not shown). The text output from the text generation unit 40 can be displayed in real time on the display device not shown, and can also be stored in a storage device or memory device not shown.

  FIG. 7 is a flowchart of the voice feature series data generation process performed by the voice feature extraction unit 30. When voice is input to the speech chain apparatus 100 (S11), the voice feature extraction unit 30 performs pre-emphasis processing on the input voice (S12), and then performs short-time Fourier transform (S13). Thus, the voice feature extraction unit 30 calculates the linear spectrum feature amount from the input voice (S14), and outputs it (S15). The output linear spectral feature is temporarily stored in a memory device (not shown) or the like. Furthermore, the voice feature extraction unit 30 calculates a logarithmic mel spectral feature amount from the linear spectral feature amount (S16), and outputs it (S17). The outputted logarithmic mel spectral feature quantity is temporarily stored in a memory device or the like (not shown).

Returning to FIG. 6, the text feature extraction unit 50 is a module that processes the input text and generates character series data (y = [y ₁ ,..., Y _T ]) input to the TTS 20. The speech generation unit 60 generates speech corresponding to the text input to the text feature extraction unit 50 based on the speech feature series data (x ^{^} = [x ₁ , ..., x _S ]) output from the TTS 20. It is a module. The text feature extraction unit 50 can be input in real time in a text input through a not-shown input device or a text read by an optical character recognition (OCR) device or the like in real time, and is stored in a not-shown storage device or memory device. You can also enter text etc. in the document. The sound output from the sound generation unit 60 can be output in real time from a not-shown speaker, and can also be stored in a not-shown storage device or memory device.

  FIG. 8 is a flowchart of the character series data generation process performed by the text feature extraction unit 50. When text is input to the speech chain apparatus 100 (S21), the text feature extraction unit 50 performs normalization processing of characters, symbols, and numbers included in the input text (S22). Specifically, the text feature extraction unit 50 converts all upper-case letters to lower-case letters, replaces some symbols such as double quotation marks with other symbols such as single quotation marks, and converts numbers into text representing their readings. (For example, "5" → "five"). Thereafter, the text feature extraction unit 50 divides the normalized text into each character (S23) (for example, "five" → "f", "i", "v", "e"). Thereafter, the text feature extraction unit 50 converts each character into an index (S24) (for example, “f” → 6, “i” → 9, “v” → 22, “e” → 5), and the normalized text And the character index (S25). The output normalized text and character index are temporarily stored in a memory device or the like (not shown).

Returning to FIG. 6, the ASR learning control unit 70 is a module that controls learning of the ASR 10. One of the voice feature series data x generated by the voice feature extraction unit 30 and the voice feature series data x ^{^} output from the TTS 20 is selectively input to the ASR 10. When text is input to the speech chain device 100 and the speech chain device 100 operates as a speech synthesizer, the ASR learning control unit 70 inputs speech feature sequence data x ^ generated by the TTS 20 as learning data into the ASR 10, Using the character series data y generated by the text feature extraction unit 50 as training data, the parameters of the ASR 10 are adjusted by the method described above.

The TTS learning control unit 80 is a module that controls the learning of the TTS 20. One of the character series data y generated by the text feature extraction unit 50 and the character series data y ^{^} output from the ASR 10 is selectively input to the TTS 20. When speech is input to the speech chain device 100 and the speech chain device 100 operates as a speech recognition device, the TTS learning control unit 80 inputs character sequence data y ^{^} generated by the ASR 10 to the TTS 20 as learning data, and Using the voice feature series data x generated by the feature extraction unit 30 as training data, the parameters of the TTS 20 are adjusted by the method described above.

  FIG. 9 is an overall flowchart of DNN speech recognition and synthesis mutual learning implemented in speech chain apparatus 100. When data is input to the speech chain apparatus 100 (S31), and it is a voice-text pair (YES in S32), the voice feature extraction unit 30 generates voice feature series data x from the input voice. The text feature extraction unit 50 generates character series data y from the input text (S33). The generation process of the voice feature series data and the character series data is as described with reference to FIGS. 7 and 8. When these series data are generated, the ASR learning control unit 70 inputs the voice feature series data x as learning data to the ASR 10, and uses the character series data y as training data to learn the ASR 10, and the TTS learning control unit 80 inputs the character series data y as learning data to the TTS 20, and learns the TTS 20 using the voice feature series data x as teacher data (S34).

  Thus, when supervised data of a voice and text pair is given, the ASR learning control unit 70 and the TTS learning control unit 80 use the supervised data to perform offline learning of ASR 10 and TTS 20 in the teacher forced mode. It can be done.

If the data input to the speech chain apparatus 100 is only voice (NO in S32, YES in S35), the voice feature extraction unit 30 generates voice feature series data x from the input voice (S36), and ASR10 Receives it and performs speech recognition (S37). Then, the TTS learning control unit 80 inputs the character series data y ^{^} output from the ASR 10 to the TTS 20 as learning data, and uses the voice feature series data x generated by the voice feature extraction unit 30 as teacher data. It is made to learn (S38). On the other hand, if the data input to the speech chain apparatus 100 is only text (NO in S32, NO in S35), the text feature extraction unit 50 generates character series data y from the input text (S39), The TTS 20 receives it and performs speech synthesis (S40). Then, the ASR learning control unit 70 inputs the voice feature series data x ^{^} output from the TTS 20 to the ASR 10 as learning data, and uses the character series data y generated by the text feature extraction unit 50 as teacher data. It is made to learn (S41).

  As described above, when only voice is given, the TTS learning control unit 80 can use the speech recognition result by the ASR 10 as learning data of the TTS 20 to learn the TTS 20. On the other hand, when only text is given, the ASR learning control unit 70 can make the ASR 10 learn using the speech synthesis result by the TTS 20 as learning data of the ASR 10. That is, online learning of ASR 10 and TTS 20 is possible using unsupervised data.

  As described above, in the speech chain apparatus 100, ASR 10 and TTS 20 can learn either of supervised data and unsupervised data, so that there are three types of speech / text pair, text only and speech only. It is possible to prepare a data set including mixed data and perform batch learning of ASR 10 and TTS 20.

FIG. 10 is a flowchart of batch learning processing of the ASR 10 and the TTS 20. The ASR learning control unit 70 and the TTS learning control unit 80 extract a predetermined amount of voice and text pairs from a data set stored in a storage device or the like (not shown) and input them to the voice feature extraction unit 30 and the text feature extraction unit 50 respectively. Then, the ASR 10 and TTS 20 are respectively learned using the speech feature series data x generated by the speech feature extraction unit 30 and the character series data y generated by the text feature extraction unit 50 (S52). Subsequently, the TTS learning control unit 80 extracts a predetermined amount of voice-only data from the data set and inputs the data to the voice feature extraction unit 30 (S53), and uses the character series data y ^{^} generated by the ASR 10 as learning data to the TTS 20. The voice feature sequence data x generated and input by the voice feature extraction unit 30 is used as training data to learn the TTS 20 (S54). Subsequently, the ASR learning control unit 70 extracts a predetermined amount of text only data from the data set and inputs it to the text feature extraction unit 50 (S55), and uses ASR 10 as speech data to generate speech feature series data x ^{^} generated by TTS 20. , And the ASR 10 is learned using the character series data y generated by the text feature extraction unit 50 as teacher data (S56). The ASR learning control unit 70 and the TTS learning control unit 80 repeat the above steps until there is no data in the data set.

  As described above, the mechanism of the human speech chain can be reproduced by machine by the speech chain device 100 according to the present embodiment. As a result, online learning of speech synthesis and speech recognition can be performed using speech input for speech recognition and text input for speech synthesis as unsupervised data, and speech as supervised data can be obtained. You can reduce the effort and cost of preparing a large number of text pairs. Furthermore, the more the speech chain device 100 according to the present embodiment is used as a speech recognition device and a speech synthesis device, the more learning progresses and the accuracy of speech recognition and speech synthesis improves.

  As described above, the embodiment has been described as an example of the technology in the present invention. For that purpose, the attached drawings and the detailed description are provided.

  Therefore, among the components described in the attached drawings and the detailed description, not only components essential for solving the problem but also components not essential for solving the problem in order to exemplify the above-mentioned technology May also be included. Therefore, the fact that those non-essential components are described in the attached drawings and the detailed description should not immediately mean that those non-essential components are essential.

  Moreover, since the above-mentioned embodiment is for illustrating the technique in the present invention, various changes, replacements, additions, omissions and the like can be made within the scope of the claims or the equivalent scope thereof.

  100: speech chain device 10: speech recognition unit 20: speech synthesis unit 30: speech feature extraction unit 40: text generation unit 50: text feature extraction unit 60: speech generation unit 70: ASR learning control unit 80 80 TTS learning control unit 225 output layer 226 input layer

Claims (7)

A speech recognition unit constructed of a deep neural network which receives speech feature series data and outputs character series data;
A speech synthesis unit constructed of a deep neural network which receives character sequence data as an input and speech characteristic sequence data as an output;
A voice feature extraction unit that processes the input voice and generates the voice feature series data input to the voice recognition unit;
A text generation unit that generates a text corresponding to the voice input to the voice feature extraction unit based on the character series data output from the voice recognition unit;
A text feature extraction unit that processes the input text and generates the character series data input to the speech synthesis unit;
A voice generation unit that generates a voice corresponding to the text input to the text feature extraction unit based on the voice feature series data output from the voice synthesis unit;
The speech feature sequence data output from the speech synthesis unit is input as learning data to the speech recognition unit, and the character recognition data generated by the text feature extraction unit is used as training data to learn the speech recognition unit A first learning control unit to cause
The character sequence data output from the speech recognition unit is input as learning data to the speech synthesis unit, and the speech synthesis unit is learned using the speech feature series data generated by the speech feature extraction unit as teacher data And a second learning control unit. The voice feature series data input to the voice recognition unit is a mel spectrum feature amount,
The speech feature series data output from the speech synthesis unit is a linear spectrum feature quantity and a mel spectrum feature quantity,
The voice feature extraction unit generates, as the voice feature sequence data, a linear spectral feature amount and a mel spectral feature amount of the voice input to the voice feature extraction unit,
The second learning control unit is configured to learn the speech synthesis unit using the linear spectrum feature and the mel spectrum feature generated by the speech feature extraction unit as training data. Speech chain equipment. The speech synthesis unit has an output layer representing the probability of termination of the speech,
The speech chain apparatus according to claim 1 or 2, wherein the second learning control unit further causes the speech synthesis unit to learn using the probability of the end of an utterance as teacher data.   The speech chain apparatus according to any one of claims 1 to 3, wherein the speech synthesis unit has an input layer to which speaker identification information is input. A speech recognition unit constructed of a deep neural network which receives speech feature series data and outputs character series data;
A speech synthesis unit constructed of a deep neural network which receives character sequence data as an input and speech characteristic sequence data as an output;
A voice feature extraction unit that processes the input voice and generates the voice feature series data input to the voice recognition unit;
A text generation unit that generates a text corresponding to the voice input to the voice feature extraction unit based on the character series data output from the voice recognition unit;
A text feature extraction unit that processes the input text and generates the character series data input to the speech synthesis unit;
A voice generation unit that generates a voice corresponding to the text input to the text feature extraction unit based on the voice feature series data output from the voice synthesis unit;
The speech feature sequence data output from the speech synthesis unit is input as learning data to the speech recognition unit, and the character recognition data generated by the text feature extraction unit is used as training data to learn the speech recognition unit A first learning control unit to cause
The character sequence data output from the speech recognition unit is input as learning data to the speech synthesis unit, and the speech synthesis unit is learned using the speech feature series data generated by the speech feature extraction unit as teacher data Computer program for causing a computer to realize a second learning control unit. A speech recognition unit constructed by a deep neural network which receives speech feature series data as an input and outputs character series data, and a speech synthesis unit constructed as a deep neural network which receives character series data as an input and outputs speech feature series data It is a DNN speech recognition / synthesis mutual learning method for mutually learning,
When a voice-text pair is given as supervised data, voice feature series data of the voice is input as learning data to the voice recognition unit, and character series data of the text is used as teacher data, and the voice recognition unit is used. A first step of learning character sequence data of the text as learning data to the speech synthesis unit and learning the speech synthesis unit using speech feature sequence data of the speech as teacher data;
When only voice is given as unsupervised data, voice feature series data of the voice is inputted to the voice recognition unit, and character series data outputted from the voice recognition unit is inputted to the voice synthesis unit as learning data. A second step of training the speech synthesis unit using speech feature series data of the speech as teacher data;
When only text is given as unsupervised data, character series data of the text is input to the speech synthesis unit, and speech feature series data output from the speech synthesis unit is input to the speech recognition unit as learning data. And a third step of training the speech recognition unit using character series data of the text as teacher data, a DNN speech recognition and synthesis mutual learning method. A fourth step of extracting a certain amount of each type of data from a data set including a mixture of speech and text pairs, text only and speech only data;
A fifth step of performing batch learning of the speech recognition unit and the speech synthesis unit by sequentially repeating the first step to the third step using each type of data extracted from the data set; The DNN speech recognition / synthesis mutual learning method according to claim 6, comprising.

Images