KR20140066640A - Apparatus and method for constructing multi-languges acoustic model and computer readable recording medium storing for a program for performing the method - Google Patents

Abstract

Provided are a device for building a multi-language acoustic model, a method for building the multi-language acoustic model, and a computer-readable medium in which a program for executing the method is recorded. The method for building the multi-language acoustic model classifies input features into a language common part and a language individual part, acquires tandem features by applying the divided language common part and language individual part to a neural network-based training, divides parameters of an acoustic model for the acquired tandem features into language common parameters and language individual parameters, adopts the language common parameters using data of a training language, adopts the language individual parameters using data of a target language, and builds an acoustic model for the target language using the adopted language common parameters and language individual parameters.

Description

TECHNICAL FIELD [0001] The present invention relates to a multi-lingual acoustic model building apparatus, a multi-lingual acoustic model building apparatus, and a computer-readable medium storing a program for performing the method, method}

The present invention relates to a method for constructing a multi-lingual acoustic model and an apparatus for constructing the multi-lingual acoustic model using the same, and more particularly to a method for constructing an acoustic model reflecting a dialect of multi- And more particularly, to a multilingual voice model construction apparatus.

Recently, various electronic devices such as smart phones and smart TVs are provided with voice recognition function. In particular, in the case of acoustic models used for speech recognition, statistical based techniques are used.

However, it is not possible to perform speech recognition for all languages with one acoustic model because the language characteristics are different for each country or region. That is, in the speech recognition technology, an acoustic model for a language to be recognized is required for speech recognition.

On the other hand, in order to construct an acoustic model for speech recognition, sufficient data must be secured for each language. It is easy to acquire comparatively sufficient data such as English, Chinese, Italian, German, and Spanish, which is used by a relatively large number of people. However, it is difficult to obtain sufficient data for a language used by a relatively small population and languages with limited access.

To solve these problems, acoustic models for multi-language and dialect have been secured using adaptive technology based on HMM (Hidden Markov Model) / GMM (Gaussian Mixture Model). Specifically, a seed acoustic model is constructed from data of a language in which sufficient data is secured, and a seed acoustic model constructed using an HMM / GMM-based adaptive technique is adapted to an acoustic model of a language to be actually constructed to be.

However, the method of constructing an acoustic model for multilingual and dialect based on HMM / GMM as described above has a problem that the same phoneme unit (Phone-Level unit) is used between languages used in adaptive technology. In other words, although the US English acoustic model can be used as a training acoustic model in order to secure the British English acoustic model, there is a disadvantage that the Korean acoustic model can not be used. In addition, the method of constructing an acoustic model for multilingual and dialect based on HMM / GMM has a limitation in that a target language for acquiring an acoustic model also requires a large amount of data in order to improve speech recognition performance.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide a method and apparatus for generating an acoustic model using an acoustic model built using an adaptive method based on a tandem feature and a subspace Gaussian Mixture Model (SGMM) The present invention provides a multilingual speech model construction apparatus for constructing a multilingual speech model capable of performing speech recognition on a multilingual speech model, a method for constructing the multilingual speech model, and a computer readable medium having recorded thereon a program for performing the method.

According to an aspect of the present invention, there is provided a method for constructing a multilingual speech model, the method comprising: dividing input features into a language common part and a language characteristic part; Applying training to a neural network based training to obtain a tandem feature; And separating the parameters of the acoustic model for the obtained tandem feature into a language common parameter and a language characteristic parameter, adapting the common language parameter using data of a training language, and using the data of the target language, And constructing an acoustic model for the target language using the adapted language common parameter and the adapted language characteristic parameter.

The obtaining step may include: dividing an input feature of the target language into a language common part and a language characteristic part; Applying the segmented language common portion and the language characteristic portion to the neural network based training; And merging the language common portion and the language characteristic portion output according to the training based on the neural network to obtain a tandem feature.

The applying step may include estimating a posterior probability value of the phonemes for the common language part and the language characteristic part using the neural network; And removing the association between the phonemes.

The estimating step may include estimating a posterior probability value of phonemes for the training language using MLP (Multi Layer Perceptron), and the removing step may include calculating a correlation between the phonemes using PCA (Principal Component Analysis) Can be removed.

The language common part may be applied to the neural network based training using the training language data, and the language characteristic part may be applied to the neural network based training using the target language data.

The building step may include constructing an SGMM acoustic model by performing a Subspace Gaussian Mixture Model (SGMM) training on the obtained tandem feature; Separating the parameters of the SGMM acoustic model into a language common parameter and a language characteristic parameter; Adapting the language common parameter using the training language data and adapting the language characteristic parameter using the target language data; And constructing an acoustic model for the target language by merging the language common parameter adapted using the training language and the language characteristic parameter adapted using the target language.

Also, the adaptation step may be adaptable using one of Maximum Likelihood Linear Regression (MLLR) and Maximum A Posteriori (MAP) algorithms.

The data amount of the training language may be larger than the data amount of the target language.

According to another aspect of the present invention, there is provided an apparatus for constructing a multi-lingual acoustic model, the apparatus comprising: a speech synthesis unit configured to classify input features of a target language into a language common part and a language characteristic part; A tandem feature acquisition unit for acquiring a tandem feature by applying a language characteristic portion to training based on a neural network; And separating the parameters of the acoustic model for the obtained tandem feature into a language common parameter and a language characteristic parameter, adapting the common language parameter using data of a training language, and using the data of the target language, And an acoustic model training unit for adapting the parameters and constructing an acoustic model for the target language using the adapted language common parameter and the adapted language characteristic parameter.

The tandem feature acquisition unit may include a separator for separating the input features of the target language into a language common portion and a language characteristic portion; A training unit for applying the divided language common portion and the language characteristic portion to the neural network-based training; And a feature merging unit for merging the language common portion and the language characteristic portion output according to the neural network-based training to obtain a tandem feature.

The training unit may further include: an estimating unit that estimates a posterior probability value of the phonemes for the common language part and the language characteristic part using the neural network; And removing the association between the phonemes.

The estimator may estimate a posterior probability value of the phonemes for the training language using a MLP (Multi Layer Perceptron), and the eliminator may remove a correlation between the phonemes using PCA (Principal Component Analysis) have.

The language common part may be applied to the neural network based training using the training language data, and the language characteristic part may be applied to the neural network based training using the target language data.

The acoustic model training unit may include an SGMM acoustic model unit configured to perform an SGMM (Subspace Gaussian Mixture Model) training on the obtained tandem feature to construct an SGMM acoustic model; A parameter separator for separating the parameters of the SGMM acoustic model into a language common parameter and a language characteristic parameter; An adaptation unit adapted to adapt the language common parameter using the training language data and adapt the language characteristic parameter using the target language data; And a parameter merge unit for constructing an acoustic model for the target language by merging the language common parameter adapted using the training language and the language characteristic parameter adapted using the target language.

Also, the adaptation unit may adapt using one of Maximum Likelihood Linear Regression (MLLR) algorithm and MAP (Maximum A Posteriori) algorithm.

The data amount of the training language may be larger than the data amount of the target language.

According to another aspect of the present invention, there is provided a computer-readable recording medium storing a program for performing a method for constructing a multilingual acoustic model, the method comprising: dividing input features into a language common part and a language characteristic part; And applying the segmented language common and language characteristic portions to training based on a neural network to obtain a tandem feature; And separating the parameters of the acoustic model for the obtained tandem feature into a language common parameter and a language characteristic parameter, adapting the common language parameter using data of a training language, and using the data of the target language, And constructing an acoustic model for the target language using the adapted language common parameter and the adapted language characteristic parameter.

According to various embodiments of the present invention as described above, by building an acoustic model for a target language having a much smaller amount of data using a language having a large amount of data, a user can also recognize speech in multiple languages and dialects . Further, by using the acoustic models of the tandem feature and the SGMM together, the speech recognition function can be further improved.

1 is a block diagram showing a configuration of a multilingual speech recognition apparatus according to an embodiment of the present invention;
2 is a block diagram showing the configuration of a tandem feature obtaining unit according to an embodiment of the present invention;
3 is a block diagram showing a configuration of an acoustic model training unit according to an embodiment of the present invention;
4A and 4B are views for explaining an HMM / GMM-based acoustic model and an SGMM-based acoustic model according to an embodiment of the present invention,
5 is a flowchart illustrating a method for constructing a multi-lingual acoustic model according to an embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings. 1 is a block diagram showing a configuration of an apparatus 100 for constructing a multi-lingual voice model according to an embodiment of the present invention. 1, the multilingual speech recognition apparatus 100 includes a tandem feature acquisition unit 110, an acoustic model training unit 120, a training language input unit 130, and a target language input unit 140. On the other hand, the "target language" described below is a language with a small amount of data to be recognized by the user, and the "training language" is a language of a large amount of data used for acquiring a target language, Lt; / RTI > For example, the target language may be Arabic, and the training language may be English.

The tandem feature acquisition unit 110 divides an input feature into a language common portion and a language characteristic portion, as shown in FIG. 1, and divides the divided common language portion and the language characteristic portion into neural network-based training to obtain a tandem feature.

The tandem feature obtaining unit 110 will be described in detail with reference to FIG. The tandem feature obtaining unit 110 may include a separating unit 111, a training unit 113, and a feature merging unit 115.

The separator 111 can distinguish between the language common portion and the language characteristic portion for the input feature. In this case, the common part of the language is a common part of the language regardless of the input language, and the characteristic part of the language is a characteristic part of the language associated with the input language.

The training unit 113 applies neural network-based training to the divided language common parts and language characteristic parts. In particular, the training unit 113 can train the common language part using the data of the training language, and can train the language characteristic part using the data of the target language. In particular, as shown in FIG. 2, the training unit 113 may include an estimation unit 113-1 and a removal unit 113-2. The estimator 113-1 estimates a posterior probability value for the phonemes of the training language from the training language data input from the training language input unit 130 for the common language part. The estimator 113-1 estimates a posterior probability value for the phoneme of the target language from the data of the target language input from the target language input unit 140 for the language characteristic portion. At this time, the estimator 113-1 may estimate a posterior probability value of the phonemes using a neural network such as MLP (Multi Layer Perceptron). The removal unit 113-2 removes the correlation between the phonemes through Principal Component Analysis (PCA). Specifically, the PCA generates a small number of new variables that maximize the information held by the plurality of variables when the mutually related variables (x1, x2, x3, ... xp) are observed. -2) can obtain tandem features for the common language and linguistic features of the language by removing the association between the phonemes of the training languages associated with each other through the PCA.

The feature merge unit 115 merges the tandem features for the language common portion and the tandem feature for the language characteristic portion to obtain a tandem feature for the target language.

Through the tandem feature acquisition unit 110 as described above, the multilingual acoustic model construction apparatus 100 can acquire tandem features of the target language that the user desires to recognize from the training language data and the target language data.

Meanwhile, in the above-described embodiment, the estimation unit 113-1 estimates the posterior probability value using the MLP is an example only, and the posterior probability value of the phonemes can be estimated using another neural network.

1, the acoustic model training unit 120 constructs an acoustic model using the tandem feature obtained through the tandem feature acquisition unit 110, and sets the acoustic model parameters as a language common parameter and a language characteristic parameter Adapt the language common parameters using the training language data input through the training language input unit 130, adapt the language characteristic parameters using the target language data input through the target language input unit 140, An acoustic model used to recognize the target language can be constructed using the adapted language common parameters and language characteristic parameters.

In particular, the acoustic model training unit 120 will be described with reference to Figs. 3 to 4B. The acoustic model training unit 120 includes an SGMM acoustic model training unit 121, a parameter separating unit 123, an adaptive unit 125 and a parameter merging unit 127, as shown in FIG.

The SGMM acoustic model training unit 121 constructs an SGMM-based acoustic model for the input tandem feature. The SGMM acoustic model training unit 121 can construct an SGMM based acoustic model instead of the existing HMM / GMM based acoustic model. The HMM / GMM-based acoustic model and the SGMM-based acoustic model will be described with reference to FIGS. 4A and 4B.

4A is a schematic diagram illustrating a conventional HMM / GMM based acoustic model structure. The HMM is connected in a plurality of states as shown in FIG. 4A. Each state is modeled as a mixture with different weights in a plurality of Gaussian in the acoustic model space. Equation (1) is a formula representing a state probability in an HMM / GMM-based acoustic model.

Where ω is the Gaussian weight, μ is the mean, and Σ is the variance.

The HMM / GMM based acoustic model has its states directly connected to the parameters in the acoustic model space. That is, each state has a GMM representative of each state, and a state probability value for an input feature value is calculated using a representative GMM. Since the GMM of each state is a statistical model estimated from a large amount of data, reliable HMM / GMM-based acoustic models can be estimated as more data are available. That is, a lot of data is required for the target language.

Meanwhile, FIG. 4B is a schematic diagram illustrating an SGMM-based acoustic model structure according to an embodiment of the present invention. As shown in FIG. 4B, the SGMM-based acoustic model structure is similar to the HMM / GMM-based acoustic model structure in that the HMM is composed of connections of various states, but the GMM parameter values are directly connected to the state And is connected to the sub-state and connected to the state. Equation (2) is an equation representing the state probability in an SGMM-based acoustic model.

Where ω is the Gaussian weight, μ is the mean, Σ is variant, M is the sub-state matrix, and v and w are the sub-state vectors. By introducing the sub-state concept, the SGMM-based acoustic model can generate acoustic models using data in a small amount of target language, and parameters can be shared even in heterogeneous languages with different phoneme units.

The parameter separating unit 123 separates the parameters of the SGMM acoustic model obtained through the SGMM acoustic model training unit 132 into a language common parameter and a language characteristic parameter. At this time, the parameter separator 123 extracts the Gaussian weight w, mean (), variance (), and sub-state vector v as common language parameters and outputs the sub- ), And the sub-state (w).

The parameter separating unit 123 may output the separated language common parameter and the language characteristic parameter to the adaptation unit 125. [

The adaptation unit 125 can separate from the parameter separation unit 123 and adapt the language common parameter and the language characteristic parameter through various algorithms. At this time, the adaptation unit 125 may adapt the common language parameter using the training language data input through the training language input unit 130, and may use the target language data input through the target language input unit 140, Parameters can be adapted.

Here, the adaptation unit 125 may adapt to the language common parameter and the language characteristic parameter using one of a Maximum Likelihood Linear Regression (MLLR) algorithm and a Maximum A Posteriori (MAP) algorithm.

The adaptation unit 125 may output the information on the language common parameter and the language characteristic parameter to the merging unit 127. [

The parameter merging unit 127 may construct an acoustic model for the target language by merging the language common parameter and the language characteristic parameter output from the adaptation unit 125. [

The acoustic model training unit 120 as described above can build an acoustic model for a target language having a much smaller amount of data using a language having a large amount of data, So that recognition can be performed.

1, the training language input unit 130 provides the training language data to the tandem feature obtaining unit 110 and the acoustic model training unit 120. As shown in FIG. In particular, the training language input unit 130 provides training language data to the tandem feature acquisition unit 110 to acquire a tandem feature for a common part of the language, and provides training language data to the acoustic model training unit 120 Training language data. At this time, the training language provided by the training language input unit 130 may be a language having a large amount of data such as English.

The target language input unit 140 provides target language data to the tandem feature acquisition unit 110 and the acoustic model training unit 120. In particular, the target language input unit 140 provides target language data to the tandem feature acquisition unit 110 to obtain a tandem feature for the linguistic characteristic portion, and supplies the target language data to the acoustic model training unit 120 It is possible to provide target language data. At this time, the target language provided by the target language input unit 140 may be a language having a small amount of data such as Arabic, African, and the like.

Hereinafter, a method for constructing a multilingual acoustic model of an apparatus 100 for constructing a multilingual acoustic model according to an embodiment of the present invention will be described with reference to FIG.

First, the multi-lingual acoustic model construction apparatus 100 divides an input feature into a language common part and a language characteristic part, and applies a divided common language part and a language characteristic part to training based on a neural network to acquire a tandem feature (S510) . In this case, the common part of the language is irrelevant to the input language, and it is possible to apply the neural network-based training using the training language data. The language characteristic part is an element dependent on the input language, Neural network based training can be applied. Here, the training language may be a language having a larger amount of data than the target language.

Then, the multilingual acoustic model construction apparatus 100 separates the parameters of the acoustic model for the tandem feature into language common parameters and language characteristic parameters, adapts language common parameters using the training language data, and uses the target language data Adapt the language characteristic parameters, and build an acoustic model for the target language using the adapted language common parameters and the adapted language characteristic parameters (S520). At this time, the multilingual acoustic model construction apparatus 100 can construct an SGMM-based acoustic model for the input tandem feature, and can separate the parameters of the SGMM-based acoustic model into language common parameters and language characteristic parameters.

As described above, by building an acoustic model for a target language having a much smaller amount of data using a language having a large amount of data, the user can be enabled to recognize speech in multiple languages and dialects with different phoneme units. Further, by using the acoustic models of the tandem feature and the SGMM together, the speech recognition function can be further improved.

Meanwhile, the method for constructing a multilingual acoustic model according to various embodiments described above may be implemented as a program and provided to a multilingual speech recognition apparatus.

Specifically, the method includes acquiring a tandem feature using training language data, constructing a first acoustic model using the training language data, and setting parameters of the first acoustic model as a language common parameter and a language characteristic parameter Constructing a second acoustic model using the language common parameter and the language characteristic parameters adapted to the target language, adapting the target language using the language characteristic parameter; And performing a speech recognition for the target language using the tandem feature and the second acoustic model. ≪ RTI ID = 0.0 > [0006] < / RTI >

A non-transitory readable medium is a medium that stores data for a short period of time, such as a register, cache, memory, etc., but semi-permanently stores data and is readable by the apparatus. In particular, the various applications or programs described above may be stored on non-volatile readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM,

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

110: voice input unit 120: tandem feature obtaining unit
130: Acoustic model training unit 140: Speech recognition unit
150:

Claims (17)

Classifying the input features into a language common portion and a language characteristic portion and applying the neural network based training to the separated language common portion and language characteristic portion to obtain a tandem feature; And
Separating a parameter of an acoustic model for the obtained tandem feature into a language common parameter and a language characteristic parameter, adapting the common language parameter using data of a training language, and using the data of the target language, And constructing an acoustic model for the target language using the adapted language common parameter and the adapted language characteristic parameter. The method according to claim 1,
Wherein the acquiring comprises:
Dividing an input feature of the target language into a language common portion and a language characteristic portion;
Applying the neural network-based training to the segmented language common portion and the language characteristic portion; And
And acquiring a tandem feature by merging the language common portion and the language characteristic portion output according to the training based on the neural network. The method according to claim 1,
Wherein the applying comprises:
Estimating a posterior probability value of the phonemes for the common language part and the language characteristic part using the neural network; And
And removing the association between the phonemes. The method of claim 3,
Wherein the estimating step comprises:
Estimates a posteriori probability value of phonemes for the training language using MLP (Multi Layer Perceptron)
Wherein the removing comprises:
Wherein the association between the phonemes is eliminated using PCA (Principal Component Analysis). 3. The method of claim 2,
Wherein the language common portion is applied to the training based on the neural network using the training language data,
Wherein the language characteristic portion is applied to the neural network-based training using the target language data. The method according to claim 1,
Wherein the constructing comprises:
Constructing an SGMM acoustic model by performing a Subspace Gaussian Mixture Model (SGMM) training on the obtained tandem feature;
Separating the parameters of the SGMM acoustic model into a language common parameter and a language characteristic parameter;
Adapting the language common parameter using the training language data and adapting the language characteristic parameter using the target language data; And
And constructing an acoustic model for the target language by merging the language common parameter adapted using the training language and the language characteristic parameter adapted using the target language. 5. The method of claim 4,
Wherein the adapting comprises:
A maximum Likelihood Linear Regression (MLLR) algorithm, and a MAP (Maximum A Posteriori) algorithm. The method according to claim 1,
Wherein the data amount of the training language is larger than the data amount of the target language. A tandem feature for obtaining a tandem feature by dividing the input features of the target language into a language common portion and a language characteristic portion and applying the neural network based training to the separated language common portion and language characteristic portion An acquisition unit; And
Separating a parameter of an acoustic model for the obtained tandem feature into a language common parameter and a language characteristic parameter, adapting the common language parameter using data of a training language, and using the data of the target language, And constructing an acoustic model for the target language by using the adapted language common parameter and the adapted language characteristic parameter. 10. The method of claim 9,
Wherein the tandem feature obtaining unit comprises:
A separator for separating the input features of the target language into a language common portion and a language characteristic portion;
A training unit for applying the neural network-based training to the segmented language common portion and the language characteristic portion; And
And a feature merge unit for merging a language common part and a language characteristic part output according to the training based on the neural network to obtain a tandem feature. 11. The method of claim 10,
The training unit includes:
An estimator for estimating a posterior probability value of the phonemes for the common language part and the language characteristic part using the neural network; And
And removing the association between the phonemes. 12. The method of claim 11,
Wherein the estimating unit comprises:
Estimates a posteriori probability value of phonemes for the training language using MLP (Multi Layer Perceptron)
The removing unit
And the association between the phonemes is removed using PCA (Principal Component Analysis). 11. The method of claim 10,
Wherein the language common portion is applied to the training based on the neural network using the training language data,
Wherein the language characteristic portion is applied to the training based on the neural network using the target language data. 10. The method of claim 9,
The acoustic model training unit includes:
An SGMM acoustic model unit configured to perform an SGMM (Subspace Gaussian Mixture Model) training on the obtained tandem feature to construct an SGMM acoustic model;
A parameter separator for separating the parameters of the SGMM acoustic model into a language common parameter and a language characteristic parameter;
An adaptation unit adapted to adapt the language common parameter using the training language data and adapt the language characteristic parameter using the target language data; And
And a parameter merging unit for merging the language common parameter adapted using the training language and the language characteristic parameter adapted using the target language to build an acoustic model for the target language. 15. The method of claim 14,
Wherein the adaptation unit comprises:
A maximum Likelihood Linear Regression (MLLR) algorithm, and a MAP (Maximum A Posteriori) algorithm. 10. The method of claim 9,
Wherein the data amount of the training language is larger than the data amount of the target language. A computer-readable recording medium having recorded thereon a program for performing a method for constructing a multilingual acoustic model,
In the multi-lingual acoustic model building method,
Classifying the input features into a language common portion and a language characteristic portion and applying the neural network based training to the separated language common portion and language characteristic portion to obtain a tandem feature; And
Separating a parameter of an acoustic model for the obtained tandem feature into a language common parameter and a language characteristic parameter, adapting the common language parameter using data of a training language, and using the data of the target language, And constructing an acoustic model for the target language using the adapted language common parameter and the adapted language characteristic parameter.

Images