JP6615736B2 - Spoken language identification apparatus, method thereof, and program - Google Patents

Description

  The present invention relates to a language identification technique for identifying in which language an input utterance is spoken.

  The spoken language identification technique is a technique for identifying which language the input voice is. For example, it identifies whether the input language is English, Japanese or Chinese. In the spoken language identification technology, a framework is generally used in which the linguistic character of each language is statistically modeled in advance to calculate and identify which input utterance is closest to. Specifically, a large number of sets of voice data and language labels (labels indicating the language of the voice data, for example, information indicating the content of "this voice is spoken in Japanese"), A language identification device is constructed by capturing the uniqueness of each language within the framework of machine learning.

  As the above-mentioned framework, attention is paid to the neural network spoken language identification mentioned in Non-Patent Document 1. In Non-Patent Document 1, the above-described framework is realized using a statistical model called a neural network, particularly a deep neural network. Specifically, the linguistic character is statistically grasped in frame units of about several milliseconds. For this purpose, a statistical model called a neural network is used. For example, if the possibility of language labels is 3 languages (Japanese, English, and Chinese), the neural network is an arbitrary model that learns a frame-by-frame model with a certain voice width (for example, 25 milliseconds). A framework for calculating the probability value of which of the three languages mentioned above when a frame is input can be realized. In addition, an acoustic feature such as MFCC (Mel Frequency Cepstrum Coefficient) is used as an input to the neural network in units of frames. Note that the deep neural network learning method and MFCC are well-known techniques, and thus description thereof is omitted here.

  Next, when performing language identification using the learned neural network, the speech is divided into frames of the speech width adopted by the neural network for the input speech for which it is unknown in which language it is spoken. In addition, the above-described acoustic feature amount is input to the neural network to obtain a probability value for each language. This probability value represents an estimated probability value indicating which language the input acoustic feature quantity is. Thereafter, the logarithmic probability obtained by taking the logarithm of the probability is averaged over all frames. Then, the language having the highest average log probability is identified as the language of the input speech.

  For example, if the input voice is 3 seconds and the length of one frame is defined as 25 milliseconds, there are 120 frames in the input voice. At this time, when the first frame of 120 frames is input to the learned deep neural network, a probability value for each language for the first frame is output. For example, the probability of English is 0.5, the probability of Japanese is 0.3, and the probability of Chinese is 0.2. Such processing is similarly performed for all the remaining 119 frames. Thereafter, an average value of log probability values is calculated for each language (every English, every Japanese, every Chinese). For example, in the case of English, it can be calculated by adding logarithmic probability values which are all English from the first frame to the 120th frame and dividing by 120. When such processing is performed, it is assumed that the average log probability of English is -10, the average log probability of Japanese is -50, and the average log probability of Chinese is -100. As a result, the spoken language identification device identifies the input speech as being English, which is the language with the largest average log probability.

Javier Gonzalez-Dominguez, Ignacio Lopez-Moreno, Pedro J. Moreno, Joaquin Gonzalez-Rodriguez, "Frame by Frame Language Identification in Short Utterances using Deep Neural Networks", Neural Networks Special Issue: Neural Network Learning in Big Data, 2014.

  However, in the spoken language identification using the conventional neural network, the spoken language identification in which the phoneme information included in the speech is accurately captured cannot be performed. In conventional spoken language identification, posterior probabilities for each language are directly modeled from acoustic features such as MFCC, but this framework does not include a structure that accurately captures phonemes. In other words, although information related to phonemes is included in acoustic feature quantities such as MFCC, the above-described neural network is a model for language identification and cannot be said to be a structure that accurately captures phonemes. However, in phonetic language identification, the arrangement of phonemes is important.

  Therefore, an object of the present invention is to provide a spoken language identification apparatus, a method thereof, and a program capable of robustly capturing phonological information included in speech and performing speech language identification using the information.

  In order to solve the above-described problem, according to one aspect of the present invention, the spoken language identification device has s = 1, 2,..., S, S is any integer of 1 or more, and is obtained from the speech data. Frame-based acoustic feature value obtained from the target speech data using a neural network acoustic model that receives the acoustic feature value in frame units as input and outputs phonological information of the predetermined language s for the speech data in frame units A bottleneck feature quantity calculation unit that calculates a bottleneck feature quantity series from the series X, a parallel bottleneck feature that constitutes a parallel bottleneck feature quantity sequence including S bottleneck feature quantity sequences and an acoustic feature quantity series X Quantitative component and parallel bottleneck feature series as input to spoken language identification neural network to identify which language the target speech data is in And a language identification unit that, the bottleneck feature quantity is the output value of the bottleneck layer is either the intermediate layer or the output layer of the neural network acoustic models.

  In order to solve the above-mentioned problem, according to another aspect of the present invention, the spoken language identification method has s = 1, 2,..., S, S is any integer of 1 or more, and a bottleneck feature Using the neural network acoustic model in which the quantity calculation unit receives the acoustic feature quantity in units of frames obtained from the voice data and outputs phonological information of the predetermined language s for the voice data in units of frames, the target voice data A bottleneck feature amount calculating step for calculating a bottleneck feature amount sequence from a frame-unit acoustic feature amount sequence X obtained from a parallel bottleneck feature amount component, S bottleneck feature amount sequences, and an acoustic feature amount A parallel bottleneck feature quantity configuration step including a series X and a parallel bottleneck feature quantity sequence, and a language identification unit converts the parallel bottleneck feature quantity sequence into a speech language A language identification step for identifying which language the target speech data is in as an input to the separate neural network, and the bottleneck feature amount is in either the intermediate layer or the output layer of the neural network acoustic model. This is the output value of a certain bottleneck layer.

  According to the present invention, there is an effect that it is possible to perform spoken language identification using phoneme information captured more robustly than in the past.

The functional block diagram of the speech language identification device which concerns on 1st embodiment. The figure which shows the example of the processing flow of the speech language identification device which concerns on 1st embodiment. The figure which shows the example of the neural network acoustic model for speech recognition which narrowed down the number of units. The figure which shows the example of the neural network acoustic model for speech recognition which deleted the intermediate | middle layer and output layer of the back | latter stage rather than the bottleneck layer.

  Hereinafter, embodiments of the present invention will be described. In the drawings used for the following description, constituent parts having the same function and steps for performing the same process are denoted by the same reference numerals, and redundant description is omitted. In the following description, it is assumed that processing performed for each element of a vector or matrix is applied to all elements of the vector or matrix unless otherwise specified.

<Points of first embodiment>
In the present embodiment, a part for capturing phonological information is installed in the preceding stage of the language identifying unit, and neural network speech language identification including the captured phonological information is performed.

  As a specific implementation method, in order to capture phonological information robustly, a plurality of neural network acoustic models (hereinafter also simply referred to as “NN acoustic models”) used in a speech recognition system are used. A plurality of NN acoustic models respectively correspond to a plurality of languages. For example, it means an NN acoustic model for Japanese speech recognition, an NN acoustic model for English speech recognition, an NN acoustic model for Chinese speech recognition, and the like. Each NN acoustic model is for identifying phonemes of a target language in units of frames. In this embodiment, the information (bottleneck feature) of the intermediate layer of the neural network that can be extracted from the NN acoustic model in units of frames is extracted using a plurality of NN acoustic models, and these are integrated. "Parallel bottleneck features" are used in addition to the input of a normal neural network for speech language identification, thereby implementing speech language identification.

<First embodiment>
FIG. 1 is a functional block diagram of the spoken language identification apparatus 100 according to the first embodiment, and FIG. 2 shows a processing flow thereof.

The spoken language identification apparatus 100 includes an acoustic feature quantity sequence X = {x ₁ , x ₂ ,..., X _T } obtained from target input speech, and S speech recognition NN acoustic models F ^s The speech language identification neural network F is input, the language of the input speech is identified, and the identification result is output as a language label L. However, X = {x ₁ , x ₂ , ..., x _T }, T is the length of the time series (total number of frames), t = 1,2, ..., T, and x _t is t This represents a vector of acoustic features of the frame. S represents the total number of language types targeted by each NN acoustic model for speech recognition, s is an index representing the language type, and s = 1, 2,. For example, if three languages, French, German, and Portuguese, are used as identification target languages, S = 3.

  For example, the spoken language identification device 100 includes a CPU, a RAM, and a computer including a ROM that stores a program for executing the following spoken language identification processing, and is functionally configured as follows. Has been. The spoken language identification device 100 includes an acoustic model deletion unit 110, a bottleneck feature amount calculation unit 120, a parallel bottleneck feature amount configuration unit 130, and a language identification unit 140. Hereinafter, the processing content of each part is demonstrated.

<Acoustic model deletion unit 110>
The acoustic model deletion unit 110 receives the S speech recognition NN acoustic models F ^s as input, deletes the intermediate layer and the output layer after the predetermined bottleneck layer k _s , and deletes the S speech after the deletion The recognition NN acoustic model F ^{s, k_s} is output. The superscript k_s means k _s .

Here, the number of intermediate layers of the NN acoustic model for speech recognition F ^s is N _s . N _s is any integer of 1 or more. The bottleneck layer k _s (N _s ≧ k _s ≧ 1) is determined manually. For example, in the case of an NN acoustic model for speech recognition with N _s = 5, k _s is any integer from 1 to 5. For example, it is determined as k _s = 4. For example, the bottleneck layer k _s of each speech recognition NN acoustic model F ^s may be determined experimentally.

The NN acoustic model F ^s for speech recognition can use any structure as long as the bottleneck layer k _s can be set. For example, the number of units may be reduced as compared with other intermediate layers for an intermediate layer that is to be set in advance as the bottleneck layer k _s (see FIG. 3). By reducing the number of units, the number of dimensions of the bottleneck feature amount can be controlled. For example, the NN acoustic model F ^s for speech recognition of the intermediate layer 5 layers, set the fourth layer as a bottleneck layer, 4-layer only unit number 64, other number of units you prepare the like 512. Any existing technique may be used as the learning method of the NN acoustic model for speech recognition. For example, the learning method of Reference Document 1 is used.
(Reference 1) Geoffrey Hinton, etc., "Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups", IEEE Signal Processing Magazine, Volume: 29, Issue: 6, Pages: 82-97, Nov. 2012.

FIG. 3 shows an example of the NN acoustic model F ^s for speech recognition with a reduced number of units, and FIG. 4 shows the NN acoustic model F ^{s, k_s} for speech recognition in which the intermediate layer and output layer after the bottleneck layer k _s are deleted. An example of
Note that when learning the speech recognition NN acoustic model F ^s , the intermediate layer and the output layer cannot be deleted. Therefore, they are deleted after learning, and used for speech language identification.

<Bottleneck feature amount calculation unit 120>
The bottleneck feature amount calculation unit 120 receives an acoustic feature amount sequence X = {x ₁ , x ₂ ,..., X _T } in units of frames obtained from target speech data, and uses the NN acoustic model F ^s for speech recognition. ^{, k_s} to calculate the bottleneck feature sequence V ^s = {v ₁ ^s , v ₂ ^s ,…, v _T ^s } from the acoustic feature sequence X = {x ₁ , x ₂ ,…, x _T } (S120) and output. The bottleneck feature value v _t ^s is the output value of the bottleneck layer k _s described above (the output value of the k _s- th intermediate layer of the speech recognition NN acoustic model F ^s ), and is a numerical vector that explicitly represents the phoneme information It is.

The bottleneck feature amount calculation unit 120 is a part that is implemented for each NN acoustic model F ^{s, k_s} for speech recognition. That is, when the S NN acoustic models for speech recognition F ^{1, k_1} , F ^{2, k_2} ,..., F ^{S, k_S} are used, the bottleneck feature amount series V ^s is represented by S time series V ¹ , V ² , ... you get V ^S.

The bottleneck feature quantity sequence V ^s is expressed as an output of the bottleneck layer when the acoustic feature quantity sequence X is input to the speech recognition NN acoustic model F ^{s, k_s} .
v _t = f (x _t )
Here, f () represents calculation up to the bottleneck layer by the NN acoustic model for speech recognition F ^{s, k_s} . Similarly, v _t represents a vector of bottleneck feature values in the t-th frame. That is, the calculation of the bottleneck feature amount is performed in units of frames, and the bottleneck feature amount sequence V ^s of length T is obtained for the acoustic feature amount sequence X of length T.

<Parallel Bottleneck Feature Quantity Configuration Unit 130>
The parallel bottleneck feature quantity configuration unit 130 receives the S bottleneck feature quantity series V ¹ , V ² ,..., V ^S and the acoustic feature quantity series X, and includes parallel bottleneck feature quantities including these pieces of information. A series P is formed (S130) and output.

For example, the parallel bottleneck feature quantity constructing unit 130 combines the S bottleneck feature quantity sequences V ¹ , V ² ,..., V ^S and the acoustic feature quantity series X for each frame t to generate a new vector series. make. S number of bottlenecks feature amount of the t-th frame _{^{v t 1, v t 2,}} ..., put a v _t ^S. parallel bottleneck feature amount p _t of t th frame,
p _t = [{v _t ¹ } ^T , {v _t ² } ^T ,…, {v _t ^S } ^T , x _t ^T ] ^T
Configure as. However, the superscript T indicates transposition. That is, the acoustic feature quantity of the original x _t and a plurality of bottleneck feature quantity _{^{v t 1, v t 2,}} ..., constitute a vector obtained by arranging the elements of v _t ^S. The final parallel bottleneck feature quantity sequence is configured as P = p ₁ , p ₂ ,..., P _T.

<Language identification unit 140>
The language identification unit 140 receives the parallel bottleneck feature amount series P and the speech language identification neural network F as inputs, and identifies the language in which the target speech data is based on these values (S140). ), And output the language label L as the identification result.

Note that this framework can be realized by handling a parallel bottleneck feature quantity part, which conventionally used an acoustic feature quantity series, as an input for neural network spoken language identification. That is, in the present embodiment, the parallel bottleneck feature amount series P is used as an input to the speech language identification neural network F to identify which language the target speech data is in. Therefore, for each frame t of P = p ₁ ,..., P _T , it is input to the speech language identification neural network F to obtain a probability value for each language. This probability represents an estimated probability value indicating which language the acoustic feature quantity x _t is. Thereafter, the logarithmic probability obtained by taking the logarithm of the probability is averaged over all frames. Then, the language having the highest average log probability is identified as the language of the input speech.

  The neural network for speech language identification F calculates a model by calculating a parallel bottleneck feature amount sequence for each learning data and learning a set of the parallel bottleneck feature amount sequence and a language label as teacher data. Can be

<Effect>
With the above configuration, it is possible to perform speech language identification using phoneme information captured more robustly than in the past. As mentioned above, the alignment of phonemes is important in spoken language identification, so the performance of spoken language identification can be greatly improved by performing spoken language identification using phoneme information captured more robustly. It is possible to expect higher accuracy than before.

<Modification>
Note that the language of the acoustic model used in the bottleneck feature quantity calculation unit 120 and the language targeted by the final spoken language identification are completely independent, and the acoustic model of the language not handled by the spoken language identification is used. Also good. In short, it is only necessary to be able to calculate a bottleneck feature value (a numerical vector that explicitly represents phonological information) from an acoustic feature value sequence X of speech data issued in a language A. For example, an NN acoustic model of another language B that has a phoneme similar to a certain language A is diverted to an NN acoustic model of another language A, or an NN of another language B that contains most of the phonemes included in a certain language A A method of diverting an acoustic model to a language A NN acoustic model is conceivable. For example, there are more phoneme types included in language B than Japanese phonemes, and if all of the phoneme types in Japanese are included, Japanese language speech is used using the NN acoustic model of language B. The bottleneck feature amount sequence V may be calculated from the acoustic feature amount sequence X obtained from the data, and the language B may not be included in the target language in the final speech language identification.

  The NN acoustic model for speech recognition is not necessarily required. In short, any NN acoustic model can be used as long as it is an acoustic model that receives acoustic features in units of frames obtained from speech data and outputs phonological information of a predetermined language s for the speech data in units of frames. It may be a thing and does not necessarily have to be for voice recognition. In addition, there is an advantage that an acoustic model for speech recognition can be shared by incorporating the speech language identification device of the present embodiment into the speech recognition device. In addition, the speech recognition apparatus has an advantage that the language can be specified with higher accuracy than before, and as a result, the accuracy of speech recognition can be improved.

In the present embodiment, the acoustic model deletion unit 110 deletes the intermediate layer and the output layer after the bottleneck layer, but the speech recognition NN acoustic model may be used without being deleted. In that case, the acoustic model deletion unit 110 may not be provided. When the speech recognition NN acoustic model is used as it is, the bottleneck feature amount calculation unit 120 is informed of the bottleneck layer k _s for each speech recognition NN acoustic model F ^s , and the bottleneck feature amount calculation unit 120 The output value of the bottleneck layer k _s of the NN acoustic model F ^s for speech recognition is output as a bottleneck feature amount.

  In the present embodiment, the bottleneck layer is one of the NN intermediate layers, but the intermediate layer is not necessarily required, and the output layer may be used as the bottleneck layer.

<Other variations>
The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

<Program and recording medium>
In addition, various processing functions in each device described in the above embodiments and modifications may be realized by a computer. In that case, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its storage unit. When executing the process, this computer reads the program stored in its own storage unit and executes the process according to the read program. As another embodiment of this program, a computer may read a program directly from a portable recording medium and execute processing according to the program. Further, each time a program is transferred from the server computer to the computer, processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program includes information provided for processing by an electronic computer and equivalent to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In addition, although each device is configured by executing a predetermined program on a computer, at least a part of these processing contents may be realized by hardware.

Claims (5)

s = 1, 2,..., S, S being one of integers of 1 or more, an acoustic feature amount in units of frames obtained from speech data as input, and phoneme information of a predetermined language s for the speech data as a frame Using a neural network acoustic model that is output in units, a bottleneck feature amount calculation unit that calculates a bottleneck feature amount sequence from an acoustic feature amount sequence X in frame units obtained from target speech data;
A parallel bottleneck feature quantity constituting a parallel bottleneck feature quantity sequence including S bottleneck feature quantity series and the acoustic feature quantity series X;
The parallel bottleneck feature quantity series as an input of a spoken language identification neural network, and a language identification unit that identifies which language the target speech data is in,
The bottleneck feature amount is an output value of a bottleneck layer that is either an intermediate layer or an output layer of the neural network acoustic model.
Spoken language identification device. The spoken language identification device of claim 1,
The neural network for speech language identification is learned using a parallel bottleneck feature quantity sequence in units of frames obtained from speech data for learning and a language label indicating the language of the speech data for learning.
Spoken language identification device. s = 1, 2,..., S, S is one of integers of 1 or more, and the bottleneck feature quantity calculation unit inputs the acoustic feature quantity in units of frames obtained from the voice data, and predetermined for the voice data A bottleneck feature quantity calculation that calculates a bottleneck feature quantity sequence from a frame-unit acoustic feature quantity sequence X obtained from target speech data, using a neural network acoustic model that outputs phonological information of each language s in frame units Steps,
A parallel bottleneck feature quantity configuring unit configured to construct a parallel bottleneck feature quantity sequence including S bottleneck feature quantity series and the acoustic feature quantity series X; and
A language identification step, wherein the parallel bottleneck feature quantity sequence is input to a neural network for speech language identification, and a language identification step for identifying which language the target speech data is in,
The bottleneck feature amount is an output value of a bottleneck layer that is either an intermediate layer or an output layer of the neural network acoustic model.
Spoken language identification method. The spoken language identification method of claim 3,
The neural network for speech language identification is learned using a parallel bottleneck feature quantity sequence in units of frames obtained from speech data for learning and a language label indicating the language of the speech data for learning.
Spoken language identification method.   A program for causing a computer to function as the spoken language identification apparatus according to claim 1 or 2.

Images