KR20170119152A - Ensemble of Jointly Trained Deep Neural Network-based Acoustic Models for Reverberant Speech Recognition and Method for Recognizing Speech using the same - Google Patents

Abstract

A combined deep - learning neural network ensemble - based acoustic model for speech recognition in reverberant environments and a speech recognition method using it are presented. A speech recognition method using an acoustic model of a deep-learning neural network ensemble for speech recognition in a reverberant environment, the method comprising: extracting a feature vector from an input speech signal; Combining the feature vectors using an ensemble of acoustic models based on the deepened neural network ensembles previously learned for each reverberation environment; And recognizing speech by classifying the phonemes, wherein the pre-learned deepening neural network ensemble-based acoustic model can estimate a phoneme probability for each reverberation environment in a plurality of reverberation environments.

Description

[0001] The present invention relates to an acoustic model based on a combined deep-learning neural network ensemble for speech recognition in a reverberant environment, and a speech recognition method using the same. [0002]

The following embodiments relate to a combined learned deep-fired neural network ensemble-based acoustic model for speech recognition in reverberation environments and a speech recognition method using the same.

Speech recognition technology identifies verbally meaningful contents from speech by automatic means. It is a process of recognizing words and word sequences by inputting speech waveforms and extracting meaning. It is widely used in speech analysis, phoneme recognition, Word recognition, sentence analysis, and semantic extraction.

Recently, speech recognition technology is implemented based on HMM (Hidden Markov Model). The HMM-based speech recognition technology consists of two Gaussian components with the most probable distribution among the Gaussian mixture components of each HMM state, and then constructs a binary tree. Then, the constructed binary tree is pruned at an appropriate level Generates an optimal acoustic model, and performs speech recognition based on the generated acoustic model.

Recently, as the application field of speech recognition technology has been expanded, the environment in which the speech recognition system is applied is diversified. If the surrounding environment is changed, the magnitude and type of the ambient noise and the sound waveform may be changed. Thus, it is necessary to re-learn the acoustic model according to the applied environment.

The acoustic model using the conventional deepening neural network technique is not effectively modeled on various real environments since it is a simple structure. In addition, even if voice signal data of various environments are used, there is a limit, and if the learning data and the test environment are different, the recognition performance may deteriorate.

Korean Patent Laid-Open No. 10-2016-0015005 describes a method and apparatus for discriminating learning of class-based acoustic models and a technique for speech recognition apparatus using the same.

Embodiments describe a combined learning deepened neural network ensemble based acoustic model for speech recognition in reverberation environment and a speech recognition method using the same. More specifically, in order to effectively model speech signals of various reverberation environments, And a technique using an ensemble structure of a learned deepened neural network model.

Embodiments are based on the deepening neural network ensemble structure, and by estimating the phoneme probability of the acoustic model optimized for the test environment, the combined learning deepened neural network ensembles based on the speech recognition in the reverberation environment having excellent recognition performance in various reverberation environments And a speech recognition method using the same.

Embodiments also provide an acoustic model that is more effective in a reverberation environment and a speech recognition method using the same, by generating a deepening neural network in which a reverberation elimination and an acoustic model are integrated by using a combined deepened neural network model in an ensemble.

A method of recognizing speech using an acoustic model of a deep-learning neural network based ensemble for speech recognition in a reverberation environment, the method comprising: extracting a feature vector from an input speech signal; Combining the feature vectors using an ensemble of acoustic models based on the deepened neural network ensembles previously learned for each reverberation environment; And recognizing speech by classifying the phonemes, wherein the pre-learned deepening neural network ensemble-based acoustic model can estimate a phoneme probability for each reverberation environment in a plurality of reverberation environments.

Estimating a reverberation time from the voice signal; And calculating a weight based on the reverberation time, wherein the step of classifying the phonemes and recognizing the speech comprises: applying the calculated weight to an ensemble combination of the learned neural network ensemble-based acoustic model .

Wherein combining the feature vectors using an ensemble of deepened neural network ensemble-based acoustic models previously learned for each reverberation environment comprises coupling the feature vectors to a deep- The phoneme probability can be estimated by passing the model-based ensemble model.

The step of recognizing speech by classifying the phonemes may classify the phonemes by calculating a posterior probability of the deepening ensemble-based acoustic model using the phoneme probability and the weight.

The estimating of the reverberation time from the voice signal may include estimating the reverberation time through a maximum likelihood by selecting two acoustic models having the greatest likelihood ratios among the deepening neural network ensemble based acoustic models.

Learning the ensemble-based acoustic model based on the deepened neural network ensemble based acoustic model, wherein the step of learning the deepened neural network ensemble-based acoustic model comprises the steps of: receiving a speech signal in a plurality of reverberation environments and extracting a feature vector; ; Separating characteristics of the reverberation environment of the extracted feature vector; And learning the extracted feature vector for each reverberation environment through a deepening neural network.

Learning the ensemble-based acoustic model based on the deepened neural network ensemble based acoustic model, wherein the step of learning the deepened neural network ensemble-based acoustic model comprises the steps of: receiving a speech signal in a plurality of reverberation environments and extracting a feature vector; ; Separating characteristics of the reverberation environment of the extracted feature vector; Passing the feature mapping enrichment neural network to map the extracted feature vector to a reverberant voice feature; Learning the acoustic modeling deepening neural network using the output of the feature mapping deepening neural network; And the acoustic modeling enrichment neural network may be stacked directly on the feature mapping enrichment neural network and the combined deepening neural network ensemble based acoustic model may be linked and re-learned.

A combined neural network ensemble-based acoustic model for speech recognition in a reverberation environment according to another embodiment includes: a feature vector extractor for extracting a feature vector from an input speech signal; An ensemble model that combines the feature vectors using an ensemble of acoustic models based on deepened neural network ensembles previously learned for each reverberation environment; And a phoneme classifier for classifying the phonemes and recognizing the phonemes, and the pre-learned deepen network ensemble-based acoustic model can estimate a phoneme probability for each reverberation environment in a plurality of reverberation environments.

A reverberation time predicting unit for estimating a reverberation time from the voice signal; And a weight determining unit for calculating a weight based on the reverberation time, and the phoneme classifying unit may apply the calculated weight to the ensemble combination of the deepened neural network ensemble-based acoustic model previously learned.

The ensemble model may estimate the phoneme probability by passing the feature vector through an ensemble model based on the deepened neural network ensemble-based acoustic model previously learned for each reverberation environment.

The phoneme classifier may classify the phoneme by calculating a posterior probability of the deepening neural network ensemble-based acoustic model using the phoneme probability and the weight.

The reverberation time predicting unit may estimate the reverberation time through a maximum likelihood by selecting two acoustic models having the greatest likelihood ratios among the deepening neural network ensemble based acoustic models.

Wherein the learning unit further comprises a feature vector extraction unit for extracting a feature vector by receiving speech signals in a plurality of reverberation environments in a learning step; A reverberation feature classifier for classifying a feature of a reverberation environment of the extracted feature vector; And a multi-depth neural network learning unit for learning the extracted feature vectors for each of the reverberation environments through a deepening neural network.

And a learning unit for learning the deepened neural network ensemble-based acoustic model, wherein the learning unit uses a structure that combines a feature mapping deepening neural network and an acoustic modeling deepening neural network that maps the reverberation- It is possible to construct a learned acoustic model for another reverberation environment.

Wherein the learning unit comprises: a feature vector extracting unit that receives a speech signal in a plurality of reverberation environments and extracts a feature vector in a learning step; A reverberation feature classifier for classifying a feature of a reverberation environment of the extracted feature vector; The feature mapping enrichment neural network mapping the extracted feature vector to a reverberant voice feature; And the acoustic modeling enrichment neural network for learning using the output of the feature mapping enrichment neural network, wherein the acoustic modeling enrichment neural network is stacked directly on the feature mapping enrichment neural network and the combined deepening neural network ensemble based acoustic model is connected Can be re-learned.

According to the embodiments, by estimating the phoneme probability of the acoustic model optimized for the test environment based on the deepened neural network ensemble structure, the combined learned deepened neural network ensembles for speech recognition in reverberation environments having excellent recognition performance in various reverberation environments And a speech recognition method using the same.

In addition, according to the embodiments, an acoustic model and a speech recognition method using the deep-reverberated neural network model can be more effective in a reverberation environment by using the combined deep-learning neural network model as an ensemble to generate one deep- .

FIG. 1 is a block diagram schematically illustrating a speech recognition system using an acoustic model based on a deepening neural network ensemble according to an embodiment.
FIG. 2 is a flowchart illustrating a speech recognition method using a deepening neural network ensemble-based acoustic model according to an exemplary embodiment.
FIG. 3 is a block diagram schematically illustrating a speech recognition system using an enhanced-neural network ensemble-based acoustic model of a combined learning structure according to another embodiment.
4 is a diagram for explaining input and output of a feature mapping deepening network according to another embodiment.
5 is a diagram illustrating a reverberation simulation environment in accordance with one embodiment.
FIG. 6 is a graph illustrating a comparison of the word-false-recognition rate between the conventional background model and the proposed ensemble model in various reverberation environments according to an embodiment.
7 shows a maximum likelihood ratio graph for reverberation time estimated for speech in a reverberation environment according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the embodiments described may be modified in various other forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more fully describe the present invention to those skilled in the art. The shape and size of elements in the drawings may be exaggerated for clarity.

The following embodiments relate to a speech recognition method in a reverberation environment using a combined deepened neural network ensemble acoustic model. In order to effectively model speech signals of various reverberation environments, an acoustic model ensemble structure using reverberation time estimation and a combined learning method The ensemble structure of the deepening neural network model is used.

Here, the acoustic model (Acoustic Model) is a technique for classifying a speech signal into phoneme data, and is an essential element of speech recognition. For example, the acoustic model uses voice recognition to facilitate convenient device manipulation, to allow the speaker to transmit unique information, to increase convenience, and to provide personalized services such as security, finance, and medical services. In the case of the translator and the search service, the acoustic model can perform efficient real-time information processing using the speech recognition system. The acoustic model suitable for the reverberant environment is widely studied because it can improve speech recognition performance in a real environment. The input signals distorted by the reverberation are limited enough to be modeled as a simple deepening neural network based acoustic model.

Accordingly, the embodiments can estimate the phoneme probability of the models for each reverberation environment using the combined learning deepening neural network, which eliminates the distortion of the voice characteristics extracted from the input signals, and the combined learning structure that integrates the acoustic modeling deepening neural network, The phoneme can be classified by combining using the ensemble of the phoneme probability of each model.

According to the embodiments, the feature vectors extracted from the speech signal are modeled for each reverberation environment through the combined structure deepening neural network, and based on the model, the reverberation time estimation using the maximum likelihood in the test environment is used, By ensembling probability, it is possible to perform speech recognition with excellent performance even in various reverberation environments.

The ensemble of the deepening neural network ensemble-based acoustic model according to an embodiment will be described below.

FIG. 1 is a block diagram schematically illustrating a speech recognition system using an acoustic model based on a deepening neural network ensemble according to an embodiment.

The speech recognition system using the deepened neural network ensemble-based acoustic model according to an embodiment can utilize an acoustic model ensemble structure using reverberation time estimation to effectively model speech signals of various reverberation environments. By estimating the phoneme probability of the acoustic model optimized for the test environment based on the deepened neural network ensemble based acoustic model, it is possible to provide a speech recognition method having excellent recognition performance in various reverberation environments.

In particular, the speech recognition system using the deepened neural network ensemble-based acoustic model according to an exemplary embodiment calculates different weights based on the actual reverberation environment estimation using the maximum likelihood method, and calculates different weights using the learned acoustic model for each reverberation environment An ensemble model to be applied to combining can be used.

Referring to FIG. 1, the speech recognition system 100 using the deepened neural network ensemble-based acoustic model according to an embodiment may include a learning unit 110 and a test unit 120.

The learning unit 110 learns the deepening neural network ensemble based acoustic model and may include a feature vector extracting unit 112, a reverberation feature classifying unit 114, and a multi-depth neural network learning unit 115 . The learning unit 110 may further include an audio input unit 111 and an enhanced neural network learning unit 113 according to an embodiment of the present invention.

The feature vector extracting unit 112 can extract a feature vector by receiving speech signals in a plurality of reverberation environments in a learning step.

The reverberation feature classifying unit 114 can separate the features of the reverberation environment of the feature vector extracted in the learning step.

The multiple enrichment neural network learning unit 115 can learn the feature vectors extracted in the learning step for each of the reverberation environments through the deepening neural network.

The test unit 120 estimates the phoneme probability of the acoustic model optimized based on the deepened neural network ensemble-based acoustic model previously learned in the learning unit 110, thereby providing a speech recognition method having excellent recognition performance in various reverberation environments can do. The test unit 120 may include a feature vector extracting unit 122, a reverberation time predicting unit 123, a weight determining unit 124, an ensemble model 125, and a phoneme classifying unit 126. The test unit 120 may further include an audio input unit 121 and a background model 127 according to an exemplary embodiment of the present invention.

The feature vector extracting unit 122 can extract the feature vector from the input speech signal.

The reverberation time predicting unit 123 may estimate the reverberation time from the voice signal and select two acoustic models having the greatest likelihood ratios among the deepening neural network ensemble-based acoustic models to calculate the reverberation time through the maximum likelihood Time can be estimated.

The weight determining unit 124 may calculate a weight based on the reverberation time.

The ensemble model 125 may combine the feature vectors using an ensemble of acoustic models based on the deepened neural network ensembles previously learned for each reverberation environment.

Here, the deepened neural network ensemble-based acoustic model that has been learned in advance can estimate the phoneme probability for each reverberation environment in a plurality of reverberation environments.

The ensemble model 125 may estimate the phoneme probability by passing the feature vector through an ensemble model based on the deepened neural network ensemble-based acoustic model previously learned for each reverberation environment.

The phoneme classifying unit 126 classifies the phonemes and recognizes the phonemes. The phoneme classifying unit 126 classifies the weights calculated by the weight determining unit 124 into the ensemble of the learned neural network ensemble- It can be applied to bonding.

The phoneme classifying unit 126 may classify the phoneme by calculating a posterior probability of the deepening neural network ensemble-based acoustic model using the phoneme probability and the weight.

An example of a deepening neural network ensemble based acoustic model according to an embodiment will be described in more detail below.

FIG. 2 is a flowchart illustrating a speech recognition method using a deepening neural network ensemble-based acoustic model according to an exemplary embodiment.

The method of extracting feature vectors from an input speech signal includes a step 210 of extracting a feature vector from an input speech signal, a step of extracting a feature vector from each of the feature vectors, (240) using an ensemble of acoustic models based on the deepened neural network ensemble based on the reverberation environment, and classifying the phonemes and recognizing the speech (250). Here, the deepened neural network ensemble-based acoustic model that has been learned in advance can estimate the phoneme probability for each reverberation environment in a plurality of reverberation environments.

In addition, a speech recognition method using a deep-learning neural network ensemble-based acoustic model for speech recognition in a reverberation environment according to an exemplary embodiment includes a step 220 of estimating a reverberation time from the speech signal, (Step 230).

At this time, the phonemes can be classified and the calculated weight values for recognizing speech can be applied to the ensemble combination of the learned deepened neural network ensemble-based acoustic model.

In this manner, the embodiments can estimate the phoneme probability of the acoustic model optimized for the test environment based on the deepened neural network ensemble structure, and thereby provide a combined deepened neural network ensemble based on speech recognition in reverberation environments having excellent recognition performance in various reverberation environments And a speech recognition method using the same.

Each step of the speech recognition method using an acoustic model based on the deepening neural network ensemble will be described in detail below. The speech recognition method using the deepened neural network ensemble-based acoustic model according to an embodiment can be more specifically explained using the speech recognition system using the deepened neural network ensemble-based acoustic model according to the embodiment described in FIG.

The speech recognition system 100 using the deepened neural network ensemble-based acoustic model according to an embodiment may include a learning unit 110 and a test unit 120.

The learning unit 110 includes a feature vector extractor 112, a reverberation feature classifier 114, and a multi-depth neural network learning unit 115. The feature vector extractor 112, . The learning unit 110 may further include an audio input unit 111 and an enhanced neural network learning unit 113 according to an embodiment of the present invention.

The test unit 120 estimates the phoneme probability of the acoustic model optimized based on the deepened neural network ensemble-based acoustic model learned in advance by the learning unit 110. The test unit 120 includes a feature vector extractor 122, 123, a weight determining unit 124, an ensemble model 125, and a phoneme classifying unit 126. The test unit 120 may further include an audio input unit 121 and a background model 127 according to an exemplary embodiment of the present invention.

First, the speech recognition method using the deepening neural network ensemble-based acoustic model according to an embodiment may include the step of learning the deepening neural network ensemble-based acoustic model.

In the learning step, the feature vector extracting unit 112 may extract a feature vector by receiving speech signals in a plurality of reverberation environments. The reverberation feature classifying unit 114 may classify the feature of the extracted reverberation environment Can be separated. Thereafter, the multi-depth neural network learning unit 115 can learn the extracted feature vectors for each of the reverberation environments through the deepening neural network.

The speech recognition method using the deepened neural network ensemble-based acoustic model according to an exemplary embodiment can use an acoustic model ensemble structure using reverberation time estimation to effectively model speech signals of various reverberation environments. Then, by estimating the phoneme probability of the acoustic model optimized for the test environment based on the deepened neural network ensemble-based acoustic model, a speech recognition method having excellent recognition performance in various reverberation environments can be provided.

In step 210, the feature vector extraction unit 122 of the test unit 120 can extract the feature vector from the input speech signal.

Meanwhile, in step 220, the reverberation time predicting unit 123 may estimate the reverberation time from the voice signal. More specifically, the reverberation time predicting unit 123 may estimate the reverberation time through the maximum likelihood by selecting two acoustic models having the greatest likelihood ratios among the deepening neural network ensemble-based acoustic models.

Also, in step 230, the weight determining unit 124 may calculate a weight based on the reverberation time.

In step 240, the ensemble model 125 may combine the feature vectors using an ensemble of acoustic models based on the deepened neural network ensembles previously learned for each reverberation environment.

Here, the deepened neural network ensemble-based acoustic model that has been learned in advance can estimate the phoneme probability for each reverberation environment in a plurality of reverberation environments.

The ensemble model 125 may estimate the phoneme probability by passing the feature vector through an ensemble model based on the deepened neural network ensemble-based acoustic model previously learned for each reverberation environment.

More specifically, the phoneme classifying unit 126 classifies the weight value calculated by the weight determining unit 124 into the previously learned deepened phoneme It can be applied to ensemble combination of acoustic model based on neural network ensemble.

The phoneme classifying unit 126 may classify the phoneme by calculating a posterior probability of the deepening neural network ensemble-based acoustic model using the phoneme probability and the weight.

개의 음향 모델의 앙상블을 이용한 앙상블 음향 모델의 최종 사후 확률(posterior probability)은 다음 수학식 1과 같이 표현될 수 있다.Learned for different reverberation environments

The final posterior probability of the ensemble acoustic model using the ensemble of the four acoustic models can be expressed by the following equation (1).

와

은 n 번째 음향 모델의 출력 확률과 가중치(weight)를 나타내며, 가중치

은 잔향 시간 추정에 의해서 결정될 수 있다.Where n is the acoustic model index, k is the Hidden Markov Model (HMM) state, and x is the voice feature vector index.

Wow

Represents the output probability and the weight of the nth acoustic model, and the weight

Can be determined by reverberation time estimation.

The reverberation can be mathematically modeled by a simple noise reduction curve as shown in Equation (2).

,

,

는 각각 실제 진폭, 감소율, 단위 계단 시퀀스를 의미할 수 있다. 또한

는 샘플링 단위,

는 확률 변수 시퀀스를 나타낼 수 있다.From here

,

,

May denote the actual amplitude, rate of decay, and unit step sequence, respectively. Also

Is a sampling unit,

Can represent a random variable sequence.

는

에 대해

개의 독립적 확률 변수로 모델링 되므로, 최대 우도법을 잔향 시간 추정에 사용 가능하다. 따라서 로그 우도비 함수는 다음 수학식 3과 같이 나타낼 수 있다.And

The

About

Since it is modeled as independent random variables, maximum likelihood method can be used for reverberation time estimation. Therefore, the log-likelihood ratio function can be expressed by the following equation (3).

이며, 감소율

는 로그 우도비의 최대값을 찾음으로써 추정 가능하고 다음 수학식 4와 같이 구할 수 있다.At this time,

, And the reduction rate

Can be estimated by finding the maximum value of the log-likelihood ratio and can be obtained by the following equation (4).

은 수학식 5와 같이 잔향 시간으로 변환할 수 있다.From here

Can be converted into a reverberation time as shown in Equation (5).

과

이 실제 잔향 시간에 가장 가깝다. 두 개의 음향 모델 각각에 대한 가중치는 수학식 3을 이용하여 구한 우도비의 비율로 결정되며 다음 수학식 6과 같이 표현될 수 있다. Thus, among the N acoustic models learned for different reverberation environments, two acoustic models with the greatest likelihood ratios

and

This is the closest to the actual reverberation time. The weight for each of the two acoustic models is determined by the ratio of the likelihood ratios obtained using Equation (3), and can be expressed as Equation (6).

는 우도비가 가장 큰 음향 모델

의 가중치이고,

는 우도비가 두 번째로 큰 음향 모델

의 가중치가 될 수 있다. 이 때,

이며, N 은 5가 될 수 있다.
From here

The acoustic model with the largest likelihood ratio

Lt; / RTI >

Is the second largest acoustic model

Can be a weight of < / RTI > At this time,

, And N can be five.

The ensemble of the deepening neural network ensemble-based acoustic model of the combined learning structure according to another embodiment will be described below.

FIG. 3 is a block diagram schematically illustrating a speech recognition system using an enhanced-neural network ensemble-based acoustic model of a combined learning structure according to another embodiment.

Deepening of the Combined Learning Structure According to Another Embodiment The speech recognition system using the neural network ensemble-based acoustic model can use an ensemble structure of a deeply-enriched deepened neural network model using reverberation time estimation to effectively model the voice signals of various reverberation environments have. By using the combined deepened neural network model as an ensemble to generate one deepened neural network that integrates reverberation and acoustic models, a more effective model can be constructed for the reverberant environment.

In particular, the speech recognition system using the deepening neural network ensemble-based acoustic model of the combined learning structure according to another embodiment calculates the different weights based on the actual reverberation environment estimation using the maximum likelihood method, An ensemble combining model can be used that applies to the combination of the two models.

개의 음향 모델을 구성할 때, 기존의 음향 모델 학습법과 달리 결합 학습법을 이용할 수 있다. 결합 학습법은 잔향 음성 특징을 잔향이 없는 음성 특징으로 맵핑(mapping)시키는 특징 맵핑 심화신경망과 음향 모델링 심화신경망을 결합하는 구조이다.Referring to FIG. 3, the speech recognition system using the deepening neural network ensemble-based acoustic model of the combined learning structure according to another embodiment is a system that learns about different reverberation environments

Unlike the conventional acoustic model learning method, the combined learning method can be used when constructing the acoustic model. The joint learning method combines a feature mapping deepening neural network and an acoustic modeling deepening neural network that maps reverberant speech features to reverberant speech features.

Deepening of the Combined Learning Structure According to Another Embodiment The speech recognition system 300 using the neural network ensemble-based acoustic model may include a learning unit 310 and a testing unit 320.

The learning unit 310 learns the deepening neural network ensemble based acoustic model and may include a feature vector extracting unit 312, a reverberation feature classifying unit 314, and a multiple combining learning unit 315. In addition, the learning unit 310 may further include an audio input unit 311 and an association learning unit 313 according to an embodiment. The joint learning unit 313 may include one feature mapping deepening network and an acoustic modeling deepening network.

The learning unit 310 can construct a learned acoustic model for different reverberation environments using a structure that combines a feature mapping deepening neural network that maps to a reverberation-free speech feature and an acoustic modeling deepening neural network .

The feature vector extractor 312 may extract a feature vector by receiving speech signals in a plurality of reverberation environments in a learning step.

The reverberation feature classifying unit 314 can separate the characteristics of the reverberation environment of the feature vector extracted in the learning step.

The multiple combination learning unit 315 may include a feature mapping deepening neural network 315a and an acoustic modeling deepening neural network 315b.

Feature mapping deepening neural network 315a may map the extracted feature vectors to reverberation-free voice features. And the acoustic modeling deepening neural network 315b can be learned using the output of the feature mapping deepening neural network 315a.

The acoustic modeling deepening neural network 315b is stacked directly on the feature mapping deepening neural network 315a and the combined deepening neural network ensemble based acoustic models can be re-learned.

The test unit 320 estimates the phoneme probability of the acoustic model optimized based on the deepened neural network ensemble-based acoustic model previously learned in the learning unit 310 to provide a speech recognition method having excellent recognition performance in various reverberation environments can do. The test unit 320 may include a feature vector extracting unit 322, a reverberation time predicting unit 323, a weight determining unit 324, an ensemble model 325, and a phoneme classifying unit 326. In addition, the test unit 320 may further include an audio input unit 321 and a background model 327 according to an exemplary embodiment of the present invention.

The feature vector extracting unit 322 can extract the feature vector from the input speech signal.

The reverberation time predicting unit 323 can estimate the reverberation time from the voice signal. The reverberation time predicting unit 323 selects two acoustic models having the greatest likelihood ratios among the deepening neural network ensemble-based acoustic models and calculates the reverberation time using the maximum likelihood Time can be estimated.

The weight determination unit 324 can calculate a weight based on the reverberation time.

The ensemble model 325 may combine the feature vectors using an ensemble of acoustic models based on the deepened neural network ensembles previously learned for each reverberation environment.

Here, the deepened neural network ensemble-based acoustic model that has been learned in advance can estimate the phoneme probability for each reverberation environment in a plurality of reverberation environments.

The ensemble model 325 may estimate the phoneme probability by passing the feature vector through an ensemble model based on the deepened neural network ensemble-based acoustic model previously learned for each reverberation environment.

The phoneme classifying unit 326 classifies the phonemes and recognizes the phonemes. The phoneme classifying unit 326 classifies the weights calculated by the weight determining unit 324 into an ensemble of the learned neural network ensemble- It can be applied to bonding.

The phoneme classifying unit 326 may classify the phoneme by calculating a posterior probability of the deepening neural network ensemble-based acoustic model using the phoneme probability and the weight.

The reverberation time estimation using maximum likelihood method is used to combine learned models of various reverberation environments by weights and an ensemble structure that combines learned models by applying combined learning method instead of conventional acoustic model learning method The ensemble structure can be used to verify speech recognition performance.

Meanwhile, the speech recognition method using the deepening neural network ensemble-based acoustic model of the combined learning structure according to another embodiment may be performed using the speech recognition system using the deepening neural network ensemble-based acoustic model of the combined learning structure according to another embodiment . Deepening of the Combined Learning Structure According to Another Embodiment The speech recognition method using the neural network ensemble based acoustic model is similar to the speech recognition method using the deepened neural network ensemble based acoustic model according to the embodiment described in FIG. 2, .

The method of recognizing speech using the deepening neural network ensemble-based acoustic model according to another embodiment may include the step of learning the deepening neural network ensemble-based acoustic model.

In the learning step, the feature vector extracting unit 312 may extract a feature vector by receiving speech signals in a plurality of reverberation environments. The reverberation feature classifying unit 314 may classify the feature of the extracted reverberation environment Can be separated.

Thereafter, the feature mapping deepening neural network 315a may map the extracted feature vectors to reverberation-free speech features. And the acoustic modeling deepening neural network 315b can be learned using the output of the feature mapping deepening neural network 315a.

This acoustic modeling deepening neural network 315b is stacked directly on the feature mapping deepening neural network 315a, and the combined deepening neural network ensemble based acoustic models can be linked and re-learned. At this time, the multiple combination learning unit 315 may include the feature mapping deepening neural network 315a and the acoustic modeling deepening neural network 315b.

4 is a diagram for explaining input and output of a feature mapping deepening network according to another embodiment.

(C) a characteristic of a voice having a reverberation time of 0.5 s; (d) a characteristic of a reverberation time of 0.7 s (E) a feature mapping deepening neural network output with a reverberation time of 0.3 s, (f) a feature deepening neural network output with a reverberation time of 0.5 s, and (g) a voice with a reverberation time of 0.7 s Of-feature deepening neural network output.

The blur caused by reverberation while passing through the feature mapping layer in all reverberation environments is reduced and the boundary between frames can be restored.

개의 심화신경망 모델은 앙상블 구조에 적용되고, 앙상블 가중치는 수학식 7과 같이 나타낼 수 있다. Next, the acoustic modeling layer can be learned using the output of the feature mapping layer. The acoustic modeling deepening neural network can be stacked directly on the feature mapping deepening neural network, and the combined deepening neural network can be linked and re - learned. In this way,

The deepened neural network model is applied to the ensemble structure, and the ensemble weight can be expressed by Equation (7).

는 우도비가 가장 큰 음향 모델

의 가중치이고,

는 우도비가 두 번째로 큰 음향 모델

의 가중치가 될 수 있다. 이 때,

이며,

은 5가 될 수 있다. From here

The acoustic model with the largest likelihood ratio

Lt; / RTI >

Is the second largest acoustic model

Can be a weight of < / RTI > At this time,

Lt;

Can be five.

The final posterior probability of the jointly learned ensemble model can be expressed as Equation 8, and this final posterior probability can be used for phoneme classification.

와

은 n 번째 음향 모델의 출력 확률과 가중치(weight)를 나타내며, 가중치

은 잔향 시간 추정에 의해서 결정될 수 있다.
Where n is the acoustic model index, k is the Hidden Markov Model (HMM) state, and x is the voice feature vector index.

Wow

Represents the output probability and the weight of the nth acoustic model, and the weight

Can be determined by reverberation time estimation.

Experiments can be conducted in various reverberation environments to verify the performance of the structure of this embodiment.

5 is a diagram illustrating a reverberation simulation environment in accordance with one embodiment.

Referring to FIG. 5, for training, 3,320 utterances are used for each environment at five reverberation times of 0.3 s, 0.4 s, 0.5 s, 0.6 s, and 0.7 s, and the same five reverberation times 376 utterances are available for each. For the experiment, we can use 192 utterances for each 50 reverberation environments at intervals of 0.01 s from 0.3 s to 0.7 s of reverberation time. As a feature vector, a 72-dimensional filterbank feature can be used, with a frame length of 25 ms and a frame shift of 10 ms. In addition, 2,021 tied-state triphone HMMs are available. The speech recognition performance test can use Kaldi.

Table 1 shows the word error rate comparisons of the single model and the base model in various reverberation environments.

Referring to Table 1, it is possible to compare the word error rate (%) performance of the basic models for each reverberation environment with the existing single acoustic model in various reverberation environments.

The background model is a conventional single acoustic model, the base acoustic model is an acoustic model for each reverberation environment, and the base joint model is an acoustic model for each reverberation environment, It means a model learned about the environment. In Table 1, the best performance was shown in bold.

The background model and each basic acoustic model are composed of seven hidden layers and one softmax layer, and each hidden layer can be composed of 2,048 nodes.

The basic combining model can be composed of a feature mapping layer composed of 3 hidden layers made up of 2,048 nodes, seven hidden layers made up of 2,048 nodes, and one softmax layer.

It can be seen that the performance of the basic models constructed for each reverberation environment is superior to that of the conventional single acoustic model in all reverberation environments. In particular, it can be seen that the performance of the combined learning model is excellent in general reverberation environments.

FIG. 6 is a graph illustrating a comparison of the word-false-recognition rate between the conventional background model and the proposed ensemble model in various reverberation environments according to an embodiment.

Referring to FIG. 6, the word error rate can be compared with the existing single acoustic model, the proposed ensemble acoustic model, and the combined learned ensemble model. The graph shows the word recognition rate and the reverberation time as the y-axis and the x-axis, respectively, and the lower the graph, the higher the recognition accuracy.

The original background model represents the existing single acoustic model and the deep background model represents the single acoustic model increased to 3,390 nodes and 10 hidden layers to match the ensemble model and computational complexity, An ensemble model represents an ensemble structure of basic acoustic models, and an ensemble joint model represents an ensemble structure of combined learning models.

The proposed ensemble model and the combined ensemble model in all noise situations show excellent performance. Despite the same amount of computation, the recognition performance of the ensemble model is superior to that of the deep background model, and the performance in the general reverberation environment can be further improved by applying the joint learning method to the ensemble structure.

7 shows a maximum likelihood ratio graph for reverberation time estimated for speech in a reverberation environment according to an embodiment.

Referring to FIG. 7, the maximum likelihood ratio graph for the estimated reverberation time for the voice in the reverberant environment has a higher probability that the maximum likelihood ratio and the reverberation time are located on the y axis and the x axis, respectively, can do.

The graph shows the maximum likelihood ratios for each of the five reverberation times, (a) the maximum likelihood ratio for speech with a reverberation time of 0.61 s, (b) the maximum likelihood ratio for speech with a reverberation time of 0.63 s, c) Maximum likelihood ratio for speech with reverberation time of 0.67 s.

In all graphs, the probability of a reverberation time of 0.6 s and the probability of 0.7 s is the highest among five reverberation times. The probability of 0.6 s is higher for a voice with a reverberation time of 0.61 s and 0.63 s, and 0.7 s for a voice with a reverberation time of 0.61 s. The reverberation environment estimate is generally accurate and can be used as an ensemble weight.

The existing single acoustic model structure degrades speech recognition performance in reverberation environments not used for learning. Embodiments can enhance the accuracy of speech recognition by applying the ensemble structure and the combining learning method to various devices and systems by making them robust to various reverberation environments.

The acoustic model can be applied to speech recognition, which can improve recognition performance in the presence of real reverberation and noise. In addition, it can be applied to various devices and systems to provide convenient operation and service using voice recognition in various environments.

Therefore, the combined neural network ensemble-based acoustic model for speech recognition in reverberation environments and speech recognition method using the deep-neural network ensemble-based acoustic model for voice recognition in reverberation environments can be applied to mobile phone terminals, wireless communication companies, voice call services such as KakaoTalk, It can be applied to speech signal processing service in various practical environments as well as speech recognition service such as speech recognition,

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing apparatus may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

A speech recognition method using an acoustic model based on a coupled deepened neural network ensemble for speech recognition in a reverberant environment,
Extracting a feature vector from an input speech signal;
Combining the feature vectors using an ensemble of acoustic models based on the deepened neural network ensembles previously learned for each reverberation environment; And
Recognizing speech by classifying phonemes
Lt; / RTI >
Wherein the pre-learned deepening neural network ensemble-
Estimating phoneme probability for each reverberation environment in multiple reverberant environments
A method for speech recognition using an acoustic model based on a combined deepened neural network ensemble for speech recognition in a reverberant environment. The method according to claim 1,
Estimating a reverberation time from the voice signal; And
Calculating a weight based on the reverberation time
Further comprising:
The step of recognizing speech by classifying the phonemes comprises:
And applying the calculated weight to the ensemble combination of the learned deepened neural network ensemble-based acoustic model
A method of speech recognition using an acoustic model based on combined deepened neural network ensembles for speech recognition in reverberation environments. 3. The method of claim 2,
Combining the feature vectors using an ensemble of acoustic models based on the deepened neural network ensembles previously learned for each reverberation environment,
And estimating the phoneme probability by passing the feature vector through an ensemble model based on the deepened neural network ensemble-based acoustic model previously learned for each reverberation environment
A method for speech recognition using an acoustic model based on a combined deepened neural network ensemble for speech recognition in a reverberant environment. The method of claim 3,
The step of recognizing speech by classifying the phonemes comprises:
And classifying the phoneme by calculating a posterior probability of the deepening neural network ensemble based acoustic model using the phoneme probability and the weight value
A method for speech recognition using an acoustic model based on a combined deepened neural network ensemble for speech recognition in a reverberant environment. 3. The method of claim 2,
Estimating a reverberation time from the speech signal,
Selecting two acoustic models having the greatest likelihood ratios among the acoustic models based on the deepening neural network ensemble and estimating the reverberation time through maximum likelihood
A method for speech recognition using an acoustic model based on a combined deepened neural network ensemble for speech recognition in a reverberant environment. The method according to claim 1,
A step of learning an acoustic model based on the deepened neural network ensemble
Further comprising:
The step of learning the deepened neural network ensemble-
Extracting a feature vector by receiving speech signals in a plurality of reverberation environments in a learning step;
Separating characteristics of the reverberation environment of the extracted feature vector; And
Learning the extracted feature vectors for each of the reverberation environments through the deepening neural network
A method of speech recognition using an acoustic model based on combined deepened neural network ensembles for speech recognition in reverberation environments. The method according to claim 1,
A step of learning an acoustic model based on the deepened neural network ensemble
Further comprising:
The step of learning the deepened neural network ensemble-
Extracting a feature vector by receiving speech signals in a plurality of reverberation environments in a learning step;
Separating characteristics of the reverberation environment of the extracted feature vector;
Passing the feature mapping enrichment neural network to map the extracted feature vector to a reverberant voice feature;
Learning the acoustic modeling deepening neural network using the output of the feature mapping deepening neural network; And
Wherein the acoustic modeling enrichment neural network is stacked directly on the feature mapping enrichment neural network and the combined deepening neural network ensemble based acoustic model is re-
A method of speech recognition using an acoustic model based on combined deepened neural network ensembles for speech recognition in reverberation environments. A speech recognition system using a combined deepened neural network ensemble based acoustic model for speech recognition in a reverberant environment,
A feature vector extracting unit for extracting a feature vector from an input speech signal;
An ensemble model that combines the feature vectors using an ensemble of acoustic models based on deepened neural network ensembles previously learned for each reverberation environment; And
A phoneme classifier for classifying phonemes and recognizing speech
Lt; / RTI >
Wherein the pre-learned deepening neural network ensemble-
Estimating phoneme probability for each reverberation environment in multiple reverberant environments
A Speech Recognition System Using Combined Learned Neural Network Ensemble Based Acoustic Model for Speech Recognition in Reverberation Environment. 9. The method of claim 8,
A reverberation time predicting unit for estimating a reverberation time from the voice signal; And
A weight determining unit for calculating a weight based on the reverberation time,
Further comprising:
Wherein the phoneme classifying unit comprises:
And applying the calculated weight to the ensemble combination of the learned deepened neural network ensemble-based acoustic model
A Speech Recognition System Using Combined Learned Neural Network Ensemble Based Acoustic Model for Speech Recognition in Reverberation Environment. 10. The method of claim 9,
In the ensemble model,
And estimating the phoneme probability by passing the feature vector through an ensemble model based on the deepened neural network ensemble-based acoustic model previously learned for each reverberation environment
A Speech Recognition System Using Combined Learned Neural Network Ensemble Based Acoustic Model for Speech Recognition in Reverberation Environment. 11. The method of claim 10,
Wherein the phoneme classifying unit comprises:
And classifying the phoneme by calculating a posterior probability of the deepening neural network ensemble based acoustic model using the phoneme probability and the weight value
A Speech Recognition System Using Combined Learned Neural Network Ensemble Based Acoustic Model for Speech Recognition in Reverberation Environment. 10. The method of claim 9,
Wherein the reverberation time predicting unit comprises:
Selecting two acoustic models having the greatest likelihood ratios among the acoustic models based on the deepening neural network ensemble and estimating the reverberation time through maximum likelihood
A Speech Recognition System Using Combined Learned Neural Network Ensemble Based Acoustic Model for Speech Recognition in Reverberation Environment. 9. The method of claim 8,
A learning unit for learning the deepening neural network ensemble-based acoustic model;
Further comprising:
Wherein,
A feature vector extractor for extracting a feature vector by receiving speech signals in a plurality of reverberation environments in a learning step;
A reverberation feature classifier for classifying a feature of a reverberation environment of the extracted feature vector; And
And a multi-depth neural network learning unit for learning the extracted feature vectors for each of the reverberation environments through a deep-
A Speech Recognition System Using Combined Learned Neural Network Ensemble Based Acoustic Model for Speech Recognition in Reverberation Environment. 9. The method of claim 8,
A learning unit for learning the deepening neural network ensemble-based acoustic model;
Further comprising:
Wherein,
Feature Mapping to Reverberation-Free Speech Feature Deepening Neural Network and Acoustic Modeling Deepening By constructing a learned acoustic model for different reverberation environments using a structure that combines neural networks
A Speech Recognition System Using Combined Learned Neural Network Ensemble Based Acoustic Model for Speech Recognition in Reverberation Environment. 15. The method of claim 14,
Wherein,
A feature vector extractor for extracting a feature vector by receiving speech signals in a plurality of reverberation environments in a learning step;
A reverberation feature classifier for classifying a feature of a reverberation environment of the extracted feature vector;
The feature mapping enrichment neural network mapping the extracted feature vector to a reverberant voice feature; And
Wherein the acoustic modeling enrichment neural network
/ RTI >
Wherein the acoustic modeling enrichment neural network is stacked directly on the feature mapping enrichment neural network and the combined deepening neural network ensemble based acoustic model is re-learned
A Speech Recognition System Using Combined Learned Neural Network Ensemble Based Acoustic Model for Speech Recognition in Reverberation Environment.

Images