KR20120056086A - Method for adapting acoustic model and Voice recognition Apparatus using the method - Google Patents

Abstract

PURPOSE: An acoustic model adapting method and a voice recognizing device using the same are provided to eliminate a re-study burden of a user about a quantized acoustic model by an embedded voice recognizing machine. CONSTITUTION: An extracting unit(110) extracts features from a waveform corresponding to a voice. The extracting unit generates quantized data. A probability measuring unit(120) applies the quantized data, an adapted network, and a quantized acoustic model to fixed point-applied high-speed computation. The probability measuring unit calculates Gaussian occupancy probability. An adaption unit(130) updates the acoustic model. A voice recognizing unit(150) recognizes the extracted features using the updated acoustic model.

Description

Method for adapting acoustic model and Voice recognition Apparatus using the method

The present invention relates to an acoustic model adaptation method and a speech recognition apparatus using the same.

As the performance of mobile devices improves, voice recognizers are increasingly being used in embedded systems. However, since the speech recognizer must deal with many parameters and large vocabulary of the acoustic model, techniques to reduce the computational cost of the recognition process have been studied. These techniques include high-speed computational techniques using fixed-point, subspace distribution clustering HMM (SDCHMM), which is divided into subspaces of the parameters of the acoustic model for low-capacity memory in mobile embedded environments. Include model quantization techniques.

In general, the speech recognizer varies greatly in recognition performance depending on the surrounding environment and the speaker. In particular, speech recognizers that operate on personalized mobile devices such as smartphones require speaker adaptation techniques to further improve performance.

Among the speaker adaptation techniques, MAP (maximum a posteriori) and MLLR (maximum likelihood linear regression) are representative techniques that use relatively low computational cost. Adaptations should be made taking into account the structure of the model. In addition, the speech recognizer must introduce a high-speed computation technique to practically reduce the computation time required for adaptation.

An object of the present invention is to provide a method for adapting a quantized acoustic model and a speech recognition apparatus using the same.

Speech recognition apparatus using an acoustic model adaptation method according to an embodiment of the present invention for solving the above problems is

An extraction unit which extracts a feature from a waveform corresponding to the voice signal and generates quantized data based on the extracted feature; A probability measurer for applying a quantization data, an adaptive network, and a quantized acoustic model to a fast operation to which a fixed point is applied to calculate a Gaussian occupation probability; An adaptor adapted to update the acoustic model using the Gaussian occupation probability and the adaptive technique; And a speech recognition unit for recognizing the extracted feature using an updated acoustic model and outputting a sentence corresponding to the recognition result.

According to an embodiment of the present invention, the speech recognition apparatus uses an acoustic model adaptation method, eliminating the user's re-learning burden for the quantized acoustic model in an embedded speech recognizer such as a smart phone, and effectively adapting the adaptation with a low calculation cost. can do.

In addition, according to an embodiment of the present invention, the acoustic model adaptation method may be adapted online with a small amount of computation, and may gradually improve performance in a mobile device environment such as a smart phone that can continuously collect and use adaptation data of one user. Can be.

In addition, the speech recognition device using the acoustic model adaptation method can automatically generate the FST network from the recognized sentences and use it for the unsupervised adaptation. You can see the learning effect of adapting the spoken words to the speaker.

1 is a configuration diagram schematically showing a voice recognition device according to an embodiment of the present invention.
2 is a block diagram illustrating a probability measuring unit and an adapting unit according to an exemplary embodiment of the present invention.
3 is a block diagram illustrating an adaptive network generator according to an exemplary embodiment of the present invention.
4 is a flowchart illustrating an acoustic model adaptation method according to an exemplary embodiment of the present invention.
5 is a flowchart illustrating a method of generating an adaptive network according to an embodiment of the present invention.

The present invention will now be described in detail with reference to the accompanying drawings. Here, the repeated description, well-known functions and configurations that may unnecessarily obscure the subject matter of the present invention, and detailed description of the configuration will be omitted. Embodiments of the present invention are provided to more completely describe the present invention to those skilled in the art. Accordingly, the shape and size of elements in the drawings may be exaggerated for clarity.

First, an acoustic model adaptation method and a calculation method applied in a speech recognition apparatus using the same will be described.

Fast Gaussian Operation for Scalar Quantization

) 연산을 기본 단위로 수행한다.In general, a speech recognition device has one Gaussian log density (Equation 1)

) Perform the operation in basic units.

번째 가우시안 d-차수인 평균(

)과 분산(

)에 대한 관측데이터(

)의 연산은 수학식 2와 같은 양자화된 매개변수(

)의 테이블 연산

로 단순화 할 수 있다.here,

Mean of the first Gaussian d-order (

) And variance (

Observations data for

) Can be calculated using the quantized parameter (

Table operations

Can be simplified.

In addition, in the case of scalar quantization of the average and variance values of 32-bit floats into 5-bit and 3-bit index values, 64-bit data obtained by combining the average and variance of the acoustic model Gaussian is converted into 8-bit index data. have.

As shown in Equation 2, when a calculation table between actual quantized averages, variances, and observations is created and stored as a fixed-point, the Gaussian log density operation can be obtained by summing the fixed-points. It becomes possible.

2. Scalar Quantization Method

The scalar quantization method expresses the mean and the variance of the existing model vector using only the mean and the variance corresponding to the scalar value, and expresses all the acoustic models by the combination of connected structures. In this case, in order to quantize the mean and the variance into 32 and 8, respectively, the following process is required.

First, all the mean and variance values of the acoustic model are collected by order, then average subtraction and variance normalization to produce zero mean and fixed variance, and then add all orders to perform k-mean algorithm. do. At this time, Lloyd-max algorithm is performed on k-mean together and sorted to exclude 0.5% of small and large values. Repeat the above procedure to fix the quantized mean. The variances of all orders are set to be the same by considering the variances of the observed data, and then nonlinear quantization values are obtained by PCA method. In this process, by generating and using shift and scale values according to the orders of the mean and the variance, respectively, all orders can be represented with scalar quantization values.

3. Adaptation method of link structure of scalar quantization model

Adaptation methods of vector quantized acoustic models such as SDCHMM include a codebook adaptation method and a link structure adaptation method.

The codebook adaptation method is relatively expensive to compute by directly updating a relatively small number of quantized codeword parameters.However, in order to prevent conflicts between model parameters that share a codeword, the global translator board ( The drawback is that you cannot do anything but adaptation-based adaptation. On the other hand, the link structure adaptation method can update the model parameters by introducing a general adaptation method, but the quantization error remains because the index value is modified to quantize to the nearest codeword value. However, since the adaptation effect by modifying the index value is larger than the disadvantages, the link structure adaptation technique is also applied to the scalar quantized acoustic model adaptation of the present invention.

번째 인덱스 코드워드인

과

에 대해

state,

가우시안,

차수로 적응된 모델 매개변수

와

의 링크구조를 갱신하는 식으로서 거리함수,

를 이용하여 가장 가까운 값으로 링크구조의 인덱스를 바꾼다. 여기서, 코드워드인 평균과 분산값은 각각

차수를 갖는 scale 상수

,

와 shift 상수

,

를 가지고 각 차수의 특성을 반영한다.In order to introduce a link structure adaptation method into a scalar quantization model, the following equation is used. Equations 3 and 4 represent the scalar quantized acoustic model.

Index codeword

and

About

state,

Gaussian,

Model Parameters Adapted to Order

Wow

To update the link structure of the distance function,

Change the index of the link structure to the nearest value using. Here, the mean and the variance of the codewords are respectively

Scale constants with degree

,

And shift constant

,

Reflect the characteristics of each order.

의

차원 값을 양자화하여 연결하는 인덱스 방법을 나타낸다. Next, Equation 5 is the input feature vector

of

An indexing method of quantizing and connecting dimension values is shown.

,

는 수학식 6과 같은 고정소수점(fixed-point) 링크구조 MAP적응 알고리즘으로 변환하여 이용할 수 있다. Scalar Quantized Acoustic Model

,

May be converted into a fixed-point link structure MAP adaptation algorithm as shown in Equation (6).

는 적응데이터 반영 비율을 정하는 실험 상수이고,

는 Baum-Welch 알고리즘을 통해 얻은 t시간의 s state, g 가우시안 고정소수점 사후확률이다. here

Is an experimental constant that determines the ratio of adaptive data reflection,

Is the s state, g, Gaussian fixed-point posterior probability of t time obtained using the Baum-Welch algorithm.

을 얻는다.In addition, the MLLR method, which is widely used as an acoustic model adaptation method, is a transformation-based algorithm using Baum's auxiliary function as the objective function. The average value of the scalar quantized acoustic model is adapted to the method by linear transformation,

Get

는

변환 행렬이고,

는 (n+1) 확장된 평균값

이다. 변환 행렬은 최우도(maximum likelihood)를 지향하는 방향으로 구하게 되는데 목적함수 Baum의 보조함수에 넣으면 수학식 8과 같이 단순화 하여 표현할 수 있다.here

Is

Transformation matrix,

Is the (n + 1) extended mean value

to be. The transformation matrix is obtained in the direction toward the maximum likelihood. When the transformation matrix is put in the auxiliary function of the objective function Baum, it can be simplified and expressed as in Equation (8).

과

는 각각 기존 음향 모델과 새 음향 모델을 나타낸다.

는 t 시간에

회귀 클래스에 속하는 s state, g 가우시안의 고정소수점 사후 확률이다. 또한,

는 분산행렬에 대한 양자화 링크된 값이다. here

and

Represent the existing and new acoustic models, respectively.

In t time

Fixed-point posterior probability of s state, g Gaussian, belonging to the regression class. Also,

Is the quantized linked value for the variance matrix.

에 대한 원소로 편미분하여 정리하면 변환 벡터

에 대해 수학식 9와 같이 요약할 수 있다.Transform Equation 8

Convert by Partial Differential into Elements for Transformation Vectors

Can be summarized as in Equation (9).

4. High-speed computation method using fixed point

In mobile embedded devices, operations vary greatly depending on the same bit float and integer types. Expressions that derive the same value also have a problem that the expressions developed by float multiply operations show more computation time than the expressions developed by addition of integer types. In order to solve this problem, Gaussian probabilistic parts with a lot of sub-point operations reduce the computation cost by applying fixed-point instead of float-point.

In order to update the acoustic model using the adaptation data, a three-step adaptation process is mainly used.

The first step is to obtain data statistics, which is a process of obtaining Gaussian posterior probabilities from adaptive data.

The second step is to find the parameters of the adaptation algorithm and then to adapt the model parameters.

The third is to update the existing acoustic model with the adapted model parameters.

In the three-step adaptation process, the first and second processes require a relatively large amount of computational cost. The second step, the adaptive algorithm, is applied as a fixed-point MAP / fixed-point MLLR method that enables a low-cost method to performance. In addition, the BW algorithm of the first process, which includes the most operations according to the adaptive data length, is constructed as follows to eliminate redundant operations and minimize cost.

① Obtain Gaussian occurrence probability from feature vector: Find the probability of occurrence by using table operation and fixed-point log add operation of Equation 2.

② Forward & Backward Algorithm: The summation between occurrence probabilities obtained in ① is summed by fixed point log sum operation.

③ Obtain Gaussian posterior probability: Combine the values obtained in ① and ② and convert them to fixed point.

Compared to the Viterbi algorithm, the BW method, which calculates the elaborate Gaussian posterior probability, consumes a lot of computation because of the large log sum. Logarithmic operations themselves are difficult to quantize because they have a wide range, but log-sum additions such as Equation 10 are constrained by Equation 11.

5. Unsupervised Adaptation Using FST Network

In general, the acoustic model adaptation method includes a map adaptation method and a non-map adaptation method.

The map adaptation method has a correct answer transcription network corresponding to the input adaptation data in advance and uses the acoustic model required by the adaptation algorithm as an acoustic model corresponding to the transcription statement network.

The non-map adaptation method refers to a method of creating a transcription door network with a recognized result of input data without using a transcription door in advance, and using the acoustic model sequence of the network.

Usually, the unsupervised adaptation is performed by reconstructing the 1-best acoustic model sequence with the recognized words to provide an acoustic model necessary for adaptation, but the present invention divides the speech recognized sentence into the vocabulary possessed by the pronunciation dictionary of the speech recognizer. The multi-pronunciation network is composed of a sentence FST (finite state transducer) network, which is converted into a sentence FST network in the unit of Gaussian mixture model (GMM) to automatically provide an acoustic model for adaptation.

This method overcomes the difficulty of expressing various pronunciation variations, which is a disadvantage of the 1-best acoustic model sequence. And because the configuration is used as an FST network, words can accommodate multiple paths that can be composed in sentences.

Hereinafter, the acoustic model adaptation method and the speech recognition apparatus using the same according to an embodiment of the present invention will be described in detail with reference to the above-described calculation methods and the accompanying drawings.

1 is a configuration diagram schematically showing a voice recognition device according to an embodiment of the present invention.

Referring to FIG. 1, the speech recognition apparatus 100 includes a quantization feature extractor 110, a probability measurer 120, an adaptor 130, an acoustic model storage 140, and a speech recognizer 150. do. In addition, the voice recognition device operates in conjunction with an external adaptive network generator 200, but is not limited thereto. Here, the adaptive network generator 200 uses a recognition result of the speech recognition apparatus 100, that is, a finite state transformation in units of a Gaussian mixture model (hereinafter, referred to as a "GMM") corresponding to the adaptive network using sentences. state transducer (FST) network.

The voice recognition device 100 receives a voice, that is, a voice signal from the user.

The quantization feature extractor 110 extracts a feature from a waveform of a voice signal and generates quantization data in a cepstrum region based on the extracted feature. Here, the quantization data includes quantization feature vectors.

The probability measurer 120 applies an adaptive network, a quantized feature vector, and a quantized acoustic model to a fast operation with a fixed point, thereby calculating a Gaussian output probability and an occupancy probability.

The adaptor 130 updates the acoustic model using a Gaussian occupation probability and an adaptation technique. Here, the adaptive technique includes a maximum likelihood linear regression (MLLR) and a maximum a posteriori (MAP) adaptation technique in a fixed-point link structure.

The acoustic model storage unit 140 stores the acoustic model updated by the adaptation unit 130. In this case, the acoustic model corresponds to a scalar quantized acoustic model, and includes a codebook and an index link structure for average and variance.

The speech recognizer 150 recognizes the feature extracted by the quantization feature extractor 110 using the updated acoustic model, and outputs a sentence including a word corresponding to the recognition result.

Next, the probability measuring unit 120 and the adaptor 130 will be described in detail with reference to FIG. 2.

2 is a block diagram illustrating a probability measuring unit and an adapting unit according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the probability measuring unit 120 includes an occurrence probability calculating unit 121, an algorithm performing unit 122, and an occupancy probability calculating unit 123.

The generation probability calculator 121 calculates a Gaussian occurrence probability by applying a quantization feature vector and a scalar quantization acoustic model to a fixed point log-sum method in a GMM network.

The algorithm performing unit 122 calculates the forward-backward probability by applying the Gaussian occurrence probability to the fixed point log summing method in the GMM network.

The occupation probability calculator 123 calculates a Gaussian occupation probability by applying a forward-backward probability to a fixed-point operation. Here, the Gaussian occupation probability corresponds to the Gaussian occupation probability of scalar quantized HMM (hidden markov model).

Next, the adaptor 130 includes a parameter calculator 131 and a model updater 132.

The parameter calculator 131 calculates the fixed-point conversion parameter by applying the Gaussian occupation probability to the MLLR technique in the fixed-point link structure.

The model updater 132 updates the acoustic model by using the calculated fixed-point conversion parameters, and updates the acoustic model again by applying the updated acoustic model and Gaussian occupation probability to the MAP technique in the fixed-point link structure.

Next, the adaptive network generator 200 will be described in detail with reference to FIG. 3.

3 is a block diagram illustrating an adaptive network generator according to an exemplary embodiment of the present invention.

First, the adaptive network generator 200 receives a recognition result, that is, a sentence of the speech recognition apparatus 100.

Referring to FIG. 3, the adaptive network generator 200 includes a pre-searcher 210, a network generator 220, and a network converter 230.

The dictionary search unit 210 searches for a monophone pronunciation corresponding to at least one word included in the received sentence in the multi-pronunciation dictionary, and receives a search result, that is, receives at least one monophone pronunciation per word. .

The network generator 220 generates a sentence FST network using at least one monophone pronunciation per word.

The network converter 230 converts the sentence FST network into a GMM unit FST network using at least one monophone pronunciation.

Specifically, the network converter 230 converts at least one monophone pronunciation included in the sentence FST network into a triphone. Next, the network converter 230 inquires the converted triphones from the tied triphone list, and converts the triphones into tide triphones corresponding to the inquiry result. In addition, the network conversion unit 230 inquires a three-state (3 state) GMM list corresponding to the tide triphone, and converts to a GMM unit FST network corresponding to the inquiry result.

According to an embodiment of the present invention, the adaptive network generation unit 200 performs a search and conversion, and since the corresponding words, monophones, triphones, and GMMs are composed of bit unit indexes, a fast search and The conversion is performed to minimize the computational cost required to generate the GMM sentence FST from the speech recognition sentence.

Next, the acoustic model adaptation method will be described in detail with reference to FIG. 4.

4 is a flowchart illustrating an acoustic model adaptation method according to an exemplary embodiment of the present invention.

First, the voice recognition apparatus 100 according to an embodiment of the present invention operates in conjunction with the adaptive network generator 200 and receives a GMM unit FST network corresponding to the adaptive network from the adaptive network generator 200.

Referring to FIG. 4, the speech recognition apparatus 100 extracts a feature from a waveform of a voice signal received from a user, and generates quantized data based on the extracted feature (S410).

The speech recognition apparatus 100 calculates a Gaussian occurrence probability and an occupancy probability by applying the adaptive network, the quantization data, and the quantized acoustic model to a fast operation with a fixed point.

The speech recognition apparatus 100 updates the acoustic model using the MLLR and the MAP technique in the Gaussian occupation probability and the fixed-point link structure (S430).

The speech recognition apparatus 100 recognizes the feature extracted from the waveform of the speech signal using the updated acoustic model, and outputs a word corresponding to the recognition result (S440).

Next, a method of generating an adaptive network using words corresponding to the recognition result will be described in detail with reference to FIG. 5.

5 is a flowchart illustrating a method of generating an adaptive network according to an embodiment of the present invention.

First, the adaptive network generator 200 receives a recognition result, that is, a sentence of the speech recognition apparatus 100.

Referring to FIG. 5, the adaptive network generator 200 searches for a monophone pronunciation corresponding to at least one word included in a received sentence in a multi-pronunciation dictionary (S510). Next, the adaptive network generator 200 receives a search result, that is, at least one monophone pronunciation per word.

The adaptive network generator 200 generates a sentence FST network using at least one monophone pronunciation per word corresponding to the search result (S520).

The adaptive network generation unit 200 converts at least one monophone pronunciation included in the sentence FST network into a triphone (S530).

The adaptive network generation unit 200 inquires the converted triphone from the tide triphone list, and converts the triphone into a tide triphone corresponding to the inquiry result (S540).

The adaptive network generation unit 200 inquires a three state GMM list corresponding to the tied triphone and converts the GMM unit FST network corresponding to the inquiry result (S550).

In general, the acoustic model adaptation method requires a lot of data, and in the process of converting the basic model, a lot of decimal operations are used for transform calculation and probability of many parameters. Expressions below the decimal point must be computed as float-point type. This computation process places a lot of burden on the embedded environment. Moreover, quantizing the acoustic model to reduce memory usage requires changing the structure of the parameters to be handled. Therefore, in order to reduce the amount of computation, a fixed-point type arithmetic method is used in which a number after the decimal point is changed to an integer.

The present invention has a scalar quantization model that reduces the size of the acoustic model by about one-eighth while maintaining the existing recognition performance, and employs a non-map link structure MAP-MLLR adaptation using a fixed point that can continuously adapt quickly using less adaptive data. It is a way.

Therefore, according to an embodiment of the present invention, the embedded voice recognition device such as a smart phone can effectively adapt the speaker while eliminating the re-learning burden of the user.

As described above, an optimal embodiment has been disclosed in the drawings and specification. Although specific terms have been used herein, they are used only for the purpose of describing the present invention and are not used to limit the scope of the present invention as defined in the meaning or claims. Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible from this. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

100: speech recognition device
110; Quantization Feature Extraction Unit
120; Probability measure
121; Occurrence probability calculation unit
122; Algorithm Execution Unit
123; Occupancy Probability Calculator
130; Adaptation
131; Parameter calculation section
132; Model update department
140; Acoustic model storage unit
150; Speech recognition unit
200; Adaptive network generator
210; Dictionary search section
220; Network generator
230; Network converter

Claims (1)

In the speech recognition device using the acoustic model adaptation method,
An extraction unit for extracting a feature from a waveform corresponding to a voice and generating quantized data based on the extracted feature;
A probability measurement unit for applying a quantization data, an adaptive network, and a quantized acoustic model to a fast operation applied a fixed point to calculate a Gaussian occupation probability;
An adaptor adapted to update the acoustic model using the Gaussian occupation probability and the adaptive technique; And
Speech recognition unit for recognizing the extracted feature using the updated acoustic model, and outputs a sentence corresponding to the recognition result
Speech recognition device comprising a.

Images