KR102449840B1 - Method and apparatus for user adaptive speech recognition - Google Patents

Abstract

A user adaptive voice recognition method and apparatus are disclosed. According to an embodiment, a method for speech recognition extracts an identification vector representing an individual characteristic of a user from speech data, inputs a sub-input vector including the identification vector to a sub-neural network, determines a scaling factor, and determines the scaling factor. and obtaining a recognition result by inputting voice data into a main neural network to which A is applied.

Description

User adaptive speech recognition method and apparatus

The following embodiments relate to a user adaptive voice recognition method and apparatus.

Recently, as a way to solve the problem of classifying input patterns into specific groups, research to apply an efficient pattern recognition method possessed by humans to an actual computer is being actively conducted. As one of these studies, there is a study on artificial neural networks that model the characteristics of human biological nerve cells by mathematical expressions. To solve the problem of classifying input patterns into specific groups, artificial neural networks use algorithms that mimic the human ability to learn. Through this algorithm, an artificial neural network can generate a mapping between input patterns and output patterns, which is expressed as an artificial neural network having the ability to learn. In addition, the artificial neural network has a generalization ability that can generate a relatively correct output for an input pattern that has not been used for learning based on the learned result.

According to an embodiment, a method for speech recognition includes extracting an identification vector representing an individual characteristic of a user from speech data; inputting a sub-input vector including the identification vector into a sub-neural network; determining a scaling factor based on an output of the sub-neural network; inputting the voice data to a main neural network to which the scaling factor is applied; and obtaining a recognition result of the voice data output from the main neural network.

The main neural network may include a recurrent neural network, and the scaling factor may adjust a state of a hidden unit in the recurrent neural network. The main neural network may include a recurrent neural network, weights for the recurrent neural network may be trained in advance, and the scaling factor may scale a hidden state vector of a hidden unit calculated using the weights.

The main neural network may include a deep neural network, and the scaling factor may adjust an output of a hidden layer in the deep neural network. The main neural network may include a deep neural network, weights for the deep neural network may be trained in advance, and the scaling factor may scale an output vector of a hidden layer calculated using the weights.

The individual characteristics of the user may be reflected in the recognition result of the voice data corresponding to the identification vector. The sub-input vector may further include the voice data. The scaling factor may include a first parameter that determines a scaling degree and a second parameter that limits a scaling range.

The sub-neural network is pre-trained to output a component for determination of the scaling factor in response to the input of the sub-input vector, and the main neural network is the recognition in response to the input of the speech data and application of the scaling factor It can be pre-trained to output results. The sub-neural network and the main neural network may be trained together.

According to an embodiment, an apparatus for speech recognition includes a processor; and a memory including computer-readable instructions, wherein when the instructions are executed by the processor, the processor extracts an identification vector representing the individual characteristics of the user from the voice data, and a sub-input including the identification vector A vector is input to a sub-neural network, a scaling factor is determined based on an output of the sub-neural network, the speech data is input to a main neural network to which the scaling factor is applied, and the speech data output from the main neural network to obtain the recognition result of

1 is a diagram illustrating an apparatus for voice recognition according to an embodiment;
2 is a diagram illustrating a main neural network and a sub-neural network according to an embodiment;
3 is a diagram illustrating a concept of a recurrent model according to an embodiment.
4 is a diagram illustrating a speech recognition process in a recurrent neural network according to an embodiment.
5 is a diagram illustrating a speech recognition process in a deep neural network according to an embodiment.
6 is a diagram illustrating an apparatus for voice recognition according to an embodiment.
Fig. 7 shows an apparatus for training according to an embodiment;

Specific structural or functional descriptions disclosed in this specification are merely illustrative for the purpose of describing embodiments according to technical concepts, and the embodiments may be embodied in various other forms and are limited to the embodiments described herein. doesn't happen

Terms such as first or second may be used to describe various elements, but these terms should be understood only for the purpose of distinguishing one element from another element. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as "comprise" are intended to designate that the described feature, number, step, action, component, part, or combination thereof exists, and includes one or more other features or numbers, steps, actions, It should be understood that the possibility of the presence or addition of components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements.

1 is a diagram illustrating an apparatus for voice recognition according to an embodiment. Referring to FIG. 1 , the apparatus 100 for speech recognition includes a main neural network 110 and a sub-neural network 120 . The device 100 may include a memory that stores instructions for at least one operation described below and a processor that executes the instructions, and the main neural network 110 and the sub-neural network 120 are the device 100 . ) in the memory in the form of instructions. Each of the main neural network 110 and the sub-neural network 120 may include a deep neural network or a recurrent neural network.

The main neural network 110 may receive voice data and output a recognition result corresponding to the voice data. Voice data may be referred to as a voice sequence and may include a plurality of voice frames. Each of the plurality of frames may correspond to feature vectors extracted from a voice signal obtained based on a user's voice input. For example, the apparatus 100 may generate voice data by extracting features from a voice signal.

The main neural network 110 may include an acoustic model. In this case, the recognition result output from the main neural network 110 may include the sound score of each frame included in the voice data. The acoustic model is for a speech recognition engine, and the speech recognition engine may include an acoustic model and a decoder. The acoustic model may provide information on which pronunciation each frame of the input voice signal is close to. The decoder may calculate which word or sentence the input voice is close to, based on information provided by the acoustic model.

The apparatus 100 may extract an identity vector from voice data and construct a sub-input vector based on the identification vector. The identification vector may be extracted for every frame of the voice data, or may be extracted from the voice data at a predetermined period, or a single identification vector may be extracted from the voice data. It is assumed below that a single identification vector having a constant value is used for every frame. The identification vector may represent individual characteristics of the user. As an example, the identification vector may be a vector expressing the variability of the GMM supervector created by connecting the average values of each Gaussian when the distribution of the acoustic parameters extracted from the voice is modeled with a Gaussian mixture model (GMM). The sub-input vector represents a vector input to the sub-neural network 120 . For example, the sub-input vector may include an identification vector or may further include voice data together with the identification vector.

The apparatus 100 may input a sub-input vector including the identification vector to the sub-neural network 120 . The sub-neural network 120 may be pre-trained to output a component for determining a scaling factor in response to an input of a sub-input vector. The apparatus 100 may determine a scaling factor based on the output of the sub-neural network 120 , and apply the determined scaling factor to the main neural network 110 .

According to an embodiment, the sub-neural network 120 may include a recurrent neural network. In this case, the sub-input vector at time step t may be composed of a hidden state vector at time step t-1, and voice data and an identification vector at time step t. Since the sub-input vector is generated for each time step, the scaling factor may be determined for each time step of the voice data. The apparatus 100 may determine a scaling factor for each time step while recognizing voice data and apply it to the main neural network 110 .

According to another embodiment, the sub-neural network 120 may include a deep neural network. In this case, the sub-input vector may be composed of an identification vector. In other words, the identification vector may be input to the sub-neural network 120 as a sub-input vector. Since the sub-input vectors have the same value irrespective of the time step, the scaling factor can be determined only once for the speech data. This scaling factor may be referred to as a representative scaling factor. For example, the device 100 may extract an identification vector from voice data before a recognition operation through the main neural network 110 and input it to the sub-neural network 120 , and based on the output of the sub-neural network 120 , Thus, a representative scaling factor may be determined. The device 100 may store the representative scaling factor and apply the representative scaling factor to the main neural network 110 while recognizing voice data.

The apparatus 100 may input voice data to the main neural network 110 to which the scaling factor is applied, and obtain a recognition result of the voice data output from the main neural network 110 . The main neural network 110 may be pre-trained to output a recognition result of speech data in response to an input of speech data and application of a scaling factor. Since the main neural network 110 operates based on the output of the sub-neural network 120 , the main neural network 110 and the sub-neural network 120 can be trained together at the same time.

According to an embodiment, individual characteristics of a user may be reflected in the main neural network 110 . For example, a user's individual characteristics, such as a pronunciation habit or a dialect, may act as a factor that reduces the accuracy of speech recognition. In addition, when the acoustic model is trained in advance based on the individual characteristics of a specific user, the corresponding acoustic model cannot be used by users other than the corresponding user. According to the embodiment, the individual characteristics of the user may be reflected in the recognition result of the voice data corresponding to the identification vector. In addition, the sub-neural network 120 is pre-trained to output the determining component of the scaling factor based on the identification vector, and user adaptive speech recognition can be performed using the sub-neural network 120 .

2 is a diagram illustrating a main neural network and a sub-neural network according to an embodiment. Referring to FIG. 2 , the main neural network 210 includes a first hidden layer 211 and an n-th hidden layer 213 . Although two hidden layers are shown in the main neural network 210 for convenience of explanation in FIG. 2 , the main neural network 210 may include one hidden layer or three or more hidden layers. Also, although not shown in FIG. 2 , the main neural network 210 may further include an input layer and an output layer. The hidden layers of the main neural network 210 may include a plurality of nodes. In the main neural network 210 , each node belonging to an adjacent layer may be connected to each other through synapses, and pre-trained weights may be assigned to the synapses. The main neural network 210 may output a recognition result of the voice data in response to the input of the voice data.

A scaling operation based on a scaling factor may be performed with respect to each hidden layer in the main neural network 210 . For example, a scaling operation based on a first scaling factor may be performed on the first hidden layer 211 , and a scaling operation based on an n-th scaling factor may be performed on the n-th hidden layer 213 . According to an embodiment, when the main neural network 210 includes a recurrent neural network, a state of a hidden unit in the main neural network 210 may be adjusted by a scaling factor. Also, according to an embodiment, when the main neural network 210 includes a deep neural network, an output of a hidden layer in the current neural network may be adjusted by a scaling factor. 2, the main neural network 210 and the sub-neural network 220 are shown as a deep neural network, but the main neural network 210 and the sub-neural network 220 are not limited to a deep neural network, It may be another kind of neural network, such as a current neural network.

Referring to FIG. 2 , the sub-neural network 220 includes a first network 221 and an n-th network 223 . Although two networks are shown in the sub-neural network 220 for convenience of explanation in FIG. 2 , the sub-neural network 220 may include networks corresponding to the number of hidden layers in the main neural network 210 . . Each network in the sub-neural network 220 including the first network 221 and the second network 223 may include an input layer, a hidden layer, and an output layer. Hidden layers of each network in the sub-neural network 220 may include a plurality of nodes. In each corresponding network, each node belonging to an adjacent layer may be connected to each other through synapses, and pre-trained weights may be assigned to the synapses. Each network in the sub-neural network 220 may output a component for determining a scaling factor in response to an input of sub-input vector data.

For example, a first scaling factor may be determined based on an output of the first network 221 , and an n-th scaling factor may be determined based on an output of the n-th network 223 . As will be described below, the scaling factor may be determined by the output of each network in the sub-neural network 220 , or may be determined based on a certain processing of the output of each network in the sub-neural network 220 . When the sub-neural network 220 includes a recurrent neural network, the output of each network in the sub-neural network 220 may mean a hidden state of each network in the sub-neural network 220 . When the sub-neural network 220 includes a deep neural network, an identification vector may be input to the sub-neural network 220 as a sub-input vector, and a scaling vector may be obtained based on the output of the sub-neural network 220 . can The obtained scaling factor may be stored in a storage device such as a memory as a representative scaling factor, and may be applied to the main neural network 210 during a recognition operation of the main neural network 210 .

3 is a diagram illustrating a concept of a recurrent model according to an embodiment. Referring to FIG. 3 , the recurrent model may have a regression loop. A new output Ot may be output from the input Xt by inputting the output of the current model back to the current model. For example, the recurrent model may include node 310 , and the output of node 310 may be input back to node 310 . A recurrent neural network may be implemented based on a recurrent model. Hereinafter, a parameter indicating a value associated with the node 310 may be referred to as a state parameter. For example, the state parameter may include an output value of the node 310 and the like.

For example, the recurrent model may be implemented in long short term memory (LSTM) or gated recurrent units (GRUs). In the current model, the node 310 may include a plurality of gates, and the state parameter of the node 310 may include a vector value indicating the cell state of the node 310 . A vector representing the cell state of the node 310 may be referred to as a state vector. The state vector may be controlled based on the gate of node 310 . The input Xt may represent a frame input from the recurrent model at a time step t in speech data, and the output Ot may represent frame data output from the recurrent model at a time step t.

4 is a diagram illustrating a speech recognition process in a recurrent neural network according to an embodiment. Referring to FIG. 4 , the main neural network may be a recurrent neural network, and may include hidden units 410 and 420 in an unfolded state. Each of the hidden units 410 and 420 may correspond to the above-described node. 4 shows the hidden unit 410 at time step t-1 and the hidden unit 420 at time step t with respect to the main neural network, but the main neural network may include a larger number of hidden units than this. can For example, the extended length of the main neural network may correspond to the length of the voice data or the window size of the voice data. In addition, the hidden units 410 and 420 are included in one layer, and the main neural network may include more layers in addition to the layer including the hidden units 410 and 420 . A hidden state vector of a general recurrent neural network can be expressed by Equation (1).

는 타임 스텝 t에서 히든 상태 벡터를 나타내고,

는 tanh(hyperbolic tangent) 및 ReLU(rectified linear unit)와 같은 비선형 연산을 나타내고,

는 히든 가중치를 나타내고,

은 타임 스텝 t-1에서 히든 상태 벡터를 나타내고,

는 입력 가중치를 나타내고,

는 타임 스텝 t에서 입력 벡터를 나타내고,

는 바이어스를 나타낸다.

,

및

각각은 모든 타임 스텝에 관해 동일한 값을 가질 수 있다.

의 차원은

,

의 차원은

,

의 차원은

,

의 차원은

일 수 있다.

는 리커런트 뉴럴 네트워크에서 히든 유닛의 수를 나타낸다. 아래에서 메인 뉴럴 네트워크에 관해서는 수학식 1의 표기(notation)를 사용하고, 서브 뉴럴 네트워크에 관해서는 수학식 1의 표기에 햇(hat)을 씌워 사용한다.In Equation 1,

denotes the hidden state vector at time step t,

represents non-linear operations such as hyperbolic tangent (tanh) and rectified linear unit (ReLU),

represents the hidden weight,

denotes the hidden state vector at time step t-1,

represents the input weight,

denotes the input vector at time step t,

represents the bias.

,

and

Each can have the same value for all time steps.

the dimension of

,

the dimension of

,

the dimension of

,

the dimension of

can be

denotes the number of hidden units in the current neural network. In the following, the notation of Equation 1 is used for the main neural network, and a hat is used to cover the notation of Equation 1 for the sub-neural network.

According to an embodiment, the sub-neural network may include a recurrent neural network. In this case, the sub-neural network may include layers and hidden units corresponding to the main neural network. According to the present embodiment, a sub-input vector generated for each time step may be input to the sub-neural network. In this case, the sub-input vector may be expressed by Equation (2).

는 타임 스텝 t에서 서브 입력 벡터를 나타내고,

은 타임 스텝 t-1에서 히든 상태 벡터를 나타내고,

는 타임 스텝 t에서 음성 데이터를 나타내고,

는 식별 벡터를 나타낸다. 수학식 2에서

는 일정한 값을 갖는다.In Equation 2,

denotes the sub-input vector at time step t,

denotes the hidden state vector at time step t-1,

represents the voice data at time step t,

denotes an identification vector. in Equation 2

has a constant value.

의 입력에 반응하여, 서브 뉴럴 네트워크는 스케일링 팩터를 결정하기 위한 성분을 출력할 수 있다. 여기에서 해당 성분을 출력한다는 것은 해당 성분을 셀 상태 값으로 저장하는 것을 포함할 수 있다. 해당 성분은 수학식 3으로 나타나는 서브 뉴럴 네트워크의 히든 상태 벡터를 포함할 수 있다.

In response to an input of , the sub-neural network may output a component for determining a scaling factor. Here, outputting the corresponding component may include storing the corresponding component as a cell state value. The corresponding component may include a hidden state vector of the sub-neural network represented by Equation (3).

는 타임 스텝 t에서 히든 상태 벡터를 나타내고,

는 히든 가중치를 나타내고,

는 입력 가중치를 나타내고,

는 tanh 및 relu와 같은 비선형 연산을 나타내고,

는 타임 스텝 t에서 서브 입력 벡터를 나타내고,

는 바이어스를 나타낸다.

,

및

각각은 모든 타임 스텝에 관해 동일한 값을 가질 수 있다.

의 차원은

,

의 차원은

,

의 차원은

일 수 있다.

는 서브 뉴럴 네트워크의 히든 유닛의 수를 나타내고,

는 메인 뉴럴 네트워크의 히든 유닛의 수를 나타내고,

는 음성 데이터의 차원을 나타내고,

는 식별 벡터의 차원을 나타낸다.In Equation 3,

denotes the hidden state vector at time step t,

represents the hidden weight,

represents the input weight,

represents non-linear operations such as tanh and relu,

denotes the sub-input vector at time step t,

represents the bias.

,

and

Each can have the same value for all time steps.

the dimension of

,

the dimension of

,

the dimension of

can be

represents the number of hidden units in the sub-neural network,

represents the number of hidden units in the main neural network,

represents the dimension of the voice data,

denotes the dimension of the identification vector.

는

에 기초하여 결정되므로, 사용자 별로 다르게 얻어질 수 있다.

를 이용하여 메인 뉴럴 네트워크를 스케일링하기 위한 스케일링 팩터

가 결정될 수 있다. 일 실시예에 따르면,

가

로 이용될 수 있다. 이 경우,

와

는 같다. 다른 실시예에 따르면, 성능 향상을 위해

로 저차원의 벡터

가 생성되고,

로 다시 고차원의 벡터

가 생성되고,

가

로 이용될 수 있다. 이 경우,

는

보다 작고, 여기서

는

의 차원을 나타낸다.

는 수학식 4로 나타낼 수 있고,

는 수학식 5로 나타낼 수 있다.

Is

Since it is determined based on , it may be obtained differently for each user.

A scaling factor for scaling the main neural network using

can be determined. According to one embodiment,

go

can be used as in this case,

Wow

is equal to According to another embodiment, in order to improve the performance

low-dimensional vector

is created,

high dimensional vector back to

is created,

go

can be used as in this case,

Is

smaller, where

Is

represents the dimension of

can be expressed by Equation 4,

can be expressed by Equation 5.

는

에 기초한 저차원의 벡터를 나타내고,

는 히든 가중치를 나타내고,

는 타임 스텝 t에서 히든 상태 벡터를 나타내고,

는 바이어스를 나타낸다.

및

각각은 모든 타임 스텝에 관해 동일한 값을 가질 수 있다.

의 차원은

,

의 차원은

일 수 있다.In Equation 4,

Is

represents a low-dimensional vector based on

represents the hidden weight,

denotes the hidden state vector at time step t,

represents the bias.

and

Each can have the same value for all time steps.

the dimension of

,

the dimension of

can be

는

에 기초한 고차원의 벡터를 나타내고,

는

에 기초한 저차원의 벡터를 나타낸다.

가 스케일링 팩터로 이용될 경우, 메인 뉴럴 네트워크의 히든 상태 벡터는 수학식 6으로 나타낼 수 있다.In Equation 5,

Is

represents a high-dimensional vector based on

Is

represents a low-dimensional vector based on .

When is used as a scaling factor, the hidden state vector of the main neural network can be expressed by Equation (6).

는 타임 스텝 t에서 히든 상태 벡터를 나타내고,

는 스케일링 팩터를 나타내고,

는 tanh 및 relu와 같은 비선형 연산을 나타내고,

는 히든 가중치를 나타내고,

은 타임 스텝 t-1에서 히든 상태 벡터를 나타내고,

는 입력 가중치를 나타내고,

는 타임 스텝 t에서 음성 데이터를 나타내고,

는 바이어스를 나타낸다.

,

및

각각은 모든 타임 스텝에 관해 동일한 값을 가질 수 있다.

의 차원은

,

의 차원은

,

의 차원은

,

의 차원은

일 수 있다.

는 리커런트 뉴럴 네트워크에서 히든 유닛의 수를 나타낸다.In Equation 6,

denotes the hidden state vector at time step t,

represents the scaling factor,

represents non-linear operations such as tanh and relu,

represents the hidden weight,

denotes the hidden state vector at time step t-1,

represents the input weight,

represents the voice data at time step t,

represents the bias.

,

and

Each can have the same value for all time steps.

the dimension of

,

the dimension of

,

the dimension of

,

the dimension of

can be

denotes the number of hidden units in the current neural network.

는 스케일링 전의 히든 상태 벡터이자 스케일링 대상이다. 메인 뉴럴 네트워크에서

및

는 미리 훈련되며, 스케일링 팩터

는 미리 훈련된

및

를 이용하여 연산된 스케일링 전의 히든 상태 벡터인

를 스케일링하는 것으로 볼 수 있다. 만약, 스케일링 팩터로

대신

가 사용될 경우, 수학식 6에서

대신

또는

가 대입될 수 있다. 스케일링 대상이

인 경우, 스케일링 대상이

및

인 경우에 비해 연산량이 적은 이점이 있다.In Equation 6,

is a hidden state vector before scaling and is a scaling target. in the main neural network

and

is pre-trained, the scaling factor

is pre-trained

and

The hidden state vector before scaling calculated using

It can be seen as scaling If the scaling factor is

instead

If is used, in Equation 6

instead

or

can be substituted. scaling target

If , the scaling target is

and

Compared to the case of , there is an advantage that the amount of computation is small.

에서

는 스케일링 정도를 결정하고,

는 스케일링 범위를 제한할 수 있다. 스케일링 결과가 과도하게 크거나 과도하게 작으면 오히려 인식의 정확도가 떨어질 수 있는데, 실시예에 따르면

를 통해 스케일링 범위가 적정하게 유지될 수 있다. 예를 들어,

는 아래 수학식 7에 기초하여 연산될 수 있다.In Equation 6,

at

determines the degree of scaling,

may limit the scaling range. If the scaling result is excessively large or excessively small, the recognition accuracy may rather decrease.

Through this, the scaling range can be properly maintained. for example,

can be calculated based on Equation 7 below.

Referring to FIG. 4 , a scaling factor may be determined based on the hidden unit 430 of the sub-neural network, and a hidden state vector of the hidden unit 410 of the main neural network and a hidden state of the main neural network based on the determined scaling factor may be used. A hidden state vector of the unit 420 may be determined. 4 shows that the scaling factor for the hidden unit 420 and the hidden state vector of the hidden unit 420 are determined. However, according to the foregoing, for all hidden units of the main neural network not shown in FIG. A scaling factor and a hidden state vector may be determined. A recognition result may be determined based on an output corresponding to the hidden state vector of the hidden unit 420 and an output corresponding to the hidden state vector of the hidden unit 430 .

According to another embodiment, the sub-neural network may include a deep neural network. In this case, the sub-neural network may include layers corresponding to the main neural network. According to the present embodiment, an identification vector may be input as a sub-input vector to the sub-neural network. The sub-neural network may output a component for determining the scaling factor in response to the input of the identification vector, and the scaling factor may be determined based on the output of the sub-neural network. In this case, since the sub-input vectors have the same value irrespective of the time step, the scaling factor can be determined only once for the speech data. This scaling factor may be referred to as a representative scaling factor. The representative scaling factor may be obtained and stored in advance before the recognition operation of the main neural network, and may be applied to the main neural network while the main neural network recognizes voice data.

The representative scaling factor may be acquired at a predetermined period in the process of recognizing a specific user's voice. For example, the representative scaling vector may be obtained for every utterance, every time point designated by the user, or every period designated by the user. When the representative scaling vector is acquired, it may be stored in a storage device such as a memory, and may be continuously used until the representative scaling vector is acquired again. Therefore, according to the present embodiment, the calculation cost can be reduced compared to the case where the scaling vector is determined for each time step and applied to the main neural network.

5 is a diagram illustrating a speech recognition process in a deep neural network according to an embodiment. The description with reference to FIG. 5 below assumes that the main neural network includes a deep neural network. The sub-neural network may include a deep neural network or a recurrent neural network. Referring to FIG. 5 , the main neural network includes hidden layers 510 and 520 .

The sub-neural network may output a component that determines the scaling factor based on the identification vector. When the sub-neural network includes a deep neural network, the operation of the sub-neural network may be applied with the exception of the cyclical characteristic in the description of FIG. 4 . After the scaling factor is determined, the scaling factor may be applied to the output of each node in the hidden layer. For example, a scaling factor may be applied to the output from node 515 to node 525 . The scaling factor may be determined with respect to all outputs transmitted to all synapses or may be determined for each node. The weights of the main neural network are pre-trained together with the weights of the sub-neural network. The scaling factor may scale the output vector of the hidden layer calculated using weights of the pre-trained main neural network. In response to the application of the scaling factor, individual characteristics of the user may be reflected in the recognition result of the main neural network.

6 is a diagram illustrating an apparatus for voice recognition according to an embodiment. Referring to FIG. 6 , the apparatus 600 for voice recognition includes a processor 610 and a memory 620 . The memory 620 may store a main neural network 621 and a sub-neural network 623 . The processor 610 may obtain data related to the main neural network 621 and the sub-neural network 623 from the memory 620, and process operations related to the main neural network 621 and the sub-neural network 623. have. The processor 610 extracts an identification vector representing an individual characteristic of a user from voice data, inputs a sub-input vector including the identification vector to the sub-neural network, and determines a scaling factor based on the output of the sub-neural network, Voice data may be input to the main neural network to which the scaling factor is applied, and a recognition result of the voice data output from the main neural network may be obtained. According to an embodiment, the memory 620 may further include a language model and a decoder. The processor 610 may generate a final recognition result using a language model and a decoder. In addition, the above may be applied to the device 600 , and a detailed description thereof will be omitted.

7 is a diagram illustrating an apparatus for training according to an embodiment. Referring to FIG. 7 , an apparatus 700 for training includes a processor 710 and a memory 720 . The memory 720 may store a main neural network 721 and a sub-neural network 723 . The processor 710 may acquire a main neural network 721 and a sub-neural network 723 from the memory 720 . The processor 710 may train the main neural network 721 and the sub-neural network 723 based on the training data. Training data 701 may include a training input and a training output. The training input is input data input to the main neural network 721 and the sub-neural network 723 , and may include, for example, voice data. The training output is data mapped to the training input, and may be, for example, a label to be output from the main neural network 721 as the training input is input to the main neural network 721 and the sub-neural network 723 . For example, when the training input is voice data, the training output may be a pronunciation probability of a sentence corresponding to the input voice data.

The processor 710 may train the main neural network 721 and the sub-neural network 723 to produce a training output from the training input. Training the main neural network 721 and the sub-neural network 723 means training the parameters of the main neural network 721 and the sub-neural network 723, the main neural network 721 and the sub-neural network 723 updating , or updating parameters of the main neural network 721 and the sub-neural network 723 .

The processor 710 extracts an identification vector from speech data of the training input and provides a sub-input vector including the identification vector to the sub-neural network 723, and the sub-neural network 723 determines the scaling factor. We can train the sub-neural network 723 to output. The device 700 may apply a scaling factor to the main neural network 721 and train the main neural network 721 to output a recognition result corresponding to the voice data of the training input. Since the main neural network 721 operates based on the output of the sub-neural network 723 , the main neural network 721 and the sub-neural network 723 can be trained together at the same time.

The training input may include various types of speech data. Accordingly, the sub-neural network 723 may output a component for a scaling factor suitable for the characteristics of each user, and the main neural network 721 may output a recognition result suitable for the characteristics of each user.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). Array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Claims (20)

A method for speech recognition, comprising:
extracting an identification vector representing an individual characteristic of a user from voice data;
inputting a sub-input vector including the identification vector into a sub-neural network;
determining a scaling factor based on an output of the sub-neural network;
inputting the voice data to a main neural network to which the scaling factor is applied; and
obtaining a recognition result of the voice data output from the main neural network;
including,
The main neural network includes a recurrent neural network,
Weights for the current neural network are trained in advance, and the scaling factor scales a hidden state vector of a hidden unit computed using the weights.
Way. According to claim 1,
The main neural network includes a recurrent neural network, and the scaling factor adjusts a state of a hidden unit in the recurrent neural network. delete According to claim 1,
The main neural network includes a deep neural network, and the scaling factor adjusts an output of a hidden layer in the deep neural network. According to claim 1,
The method of claim 1, wherein the main neural network includes a deep neural network, weights for the deep neural network are trained in advance, and the scaling factor scales an output vector of a hidden layer computed using the weights. According to claim 1,
The method of claim 1, wherein individual characteristics of the user are reflected in the recognition result of the voice data corresponding to the identification vector. According to claim 1,
The sub-input vector further comprises the speech data. According to claim 1,
The scaling factor includes a first parameter for determining a scaling degree and a second parameter for limiting a scaling range. According to claim 1,
the sub-neural network is pre-trained to output a component for determining the scaling factor in response to an input of the sub-input vector;
wherein the main neural network is pre-trained to output the recognition result in response to the input of the speech data and the application of the scaling factor. According to claim 1,
The method of claim 1, wherein the sub-neural network and the main neural network are trained together. A computer-readable storage medium storing one or more programs comprising instructions for performing the method of any one of claims 1, 2, and 4 to 10. A device for voice recognition, comprising:
processor; and
memory containing computer-readable instructions
including,
When the instruction is executed by the processor, the processor extracts an identification vector representing an individual characteristic of a user from voice data, inputs a sub-input vector including the identification vector to a sub-neural network, and outputs the sub-neural network determine a scaling factor based on , input the voice data to a main neural network to which the scaling factor is applied, and obtain a recognition result of the voice data output from the main neural network,
The main neural network includes a recurrent neural network,
Weights for the current neural network are trained in advance, and the scaling factor scales a hidden state vector of a hidden unit computed using the weights. 13. The method of claim 12,
The main neural network includes a recurrent neural network, and the scaling factor adjusts a state of a hidden unit in the recurrent neural network. delete 13. The method of claim 12,
The main neural network includes a deep neural network, and the scaling factor adjusts an output of a hidden layer in the deep neural network. 13. The method of claim 12,
The main neural network includes a deep neural network, weights for the deep neural network are trained in advance, and the scaling factor scales an output vector of a hidden layer calculated using the weights. 13. The method of claim 12,
The apparatus of claim 1, wherein individual characteristics of the user are reflected in the recognition result of the voice data corresponding to the identification vector. 13. The method of claim 12,
The sub-input vector further comprises the speech data. 13. The method of claim 12,
The scaling factor includes a first parameter for determining a scaling degree and a second parameter for limiting a scaling range. 13. The method of claim 12,
the sub-neural network is pre-trained to output a component for determining the scaling factor in response to an input of the sub-input vector;
and the main neural network is pre-trained to output the recognition result in response to the input of the speech data and the application of the scaling factor.

Images