KR102415805B1 - Method for generating speech recognition model and apparatuses using the same - Google Patents

Abstract

Disclosed is a method for generating a speech recognition model according to one embodiment. The method for generating the speech recognition model according to one embodiment may comprise: a step of acquiring a plurality of candidate speech recognition models based on a combination of cells from a cell-based search space; a step of training for the plurality of candidate speech recognition models to output a text corresponding to the input speech data; and a step of generating a final speech recognition model by performing an additional training for the candidate speech recognition model that satisfies a predetermined condition among the plurality of trained candidate speech recognition models. Therefore, the present invention is capable of representing a lower word error rate compared to an existing method.

Description

Speech recognition model generation method and apparatus using the same

The present specification relates to a method for generating a Neural Architecture Search (NAS)-based speech recognition model and an apparatus using the same.

In the field of speech recognition, a recurrent neural network (RNN)-based speech recognition model is used as an end-to-end model. A recurrent neural network-based speech recognition model takes a very long time to learn and occupies a lot of memory. The transformer-based speech recognition model can solve the problems of the existing recurrent neural network-based model by using a feed forward neural network instead of a recurrent neural network. However, since the transformer has a large amount of parameters, the memory used for training increases exponentially as the structure of the model deepens.

NAS (Neural Architecture Search) is a technology that automates the design of artificial neural networks widely used in the field of machine learning. NAS refers to a technology that defines the unit structure constituting the model in the search space and automates the design of artificial neural networks by selecting the unit structure.

Non-Patent Document 1: [Dong, Linhao, Shuang Xu, and Bo Xu. "Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition." 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).IEEE, 2018.]

A method of generating a speech recognition model according to an embodiment includes: obtaining a plurality of candidate speech recognition models based on a combination of cells found from a cell-based search space; and generating a final speech recognition model by performing additional training on a candidate speech recognition model that satisfies a predetermined condition among the plurality of candidate speech recognition models that have been initially trained to output text corresponding to the input speech data. may include

The acquiring may include: acquiring an encoder cell from the search space; obtaining a decoder cell from the search space; generating the plurality of candidate speech recognition models based on a combination of the encoder cell and the decoder cell.

The cell may include at least one block, and each of the at least one block may include two branches that perform an operation based on different inputs, and a coupling layer that combines outputs of the two branches.

Each of the encoder cell and the decoder cell may include at least one block corresponding to multi-head attention.

For the block corresponding to the multi-head attention, an attention operation may be performed through the remaining elements except for an element less than a predetermined threshold among elements included in the input matrix.

The generating of the final speech recognition model may include determining the trained candidate speech recognition model as the final speech recognition model when the number of previously performed training corresponds to a preset number threshold.

The generating of the final speech recognition model may include, when the number of pre-trained training is less than a preset number threshold, select a candidate speech recognition model whose performance is greater than or equal to a performance threshold among the trained candidate speech recognition models. determining additional training targets; determining a second additional training target from the candidate speech recognition model additionally obtained from the search space; and performing additional training on the first additional training target and the second additional training target.

The apparatus for generating a speech recognition model according to an embodiment obtains a plurality of candidate speech recognition models based on a combination of cells from a cell-based search space, and receives an input speech with respect to the plurality of candidate speech recognition models. A processor for generating a final speech recognition model by training to output text corresponding to data, and performing additional training on a candidate speech recognition model that satisfies a predetermined condition among the plurality of trained candidate speech recognition models. have.

Through the method for generating a voice recognition model according to the present invention, it is possible to reduce the training time and to generate a more lightweight voice recognition model. In addition, the speech recognition model generated through the method for generating a speech recognition model according to the present invention may exhibit a lower word error rate (WER) than the existing method.

1 is a diagram for illustratively explaining the structure of a transformer used for speech recognition of the present invention.
2 is a diagram for explaining in more detail a multi-head attention layer included in a transformer according to an embodiment.
3 is a diagram for describing a method of generating a voice recognition model according to an exemplary embodiment.
4 is a diagram for explaining a method of obtaining a candidate speech recognition model using a cell-based search space according to a method of generating a speech recognition model according to an embodiment.
5 is a flowchart illustrating a method of generating a voice recognition model based on a PDH according to an embodiment.
6 is a diagram illustrating an example of a speech recognition model generated through a method for generating a speech recognition model according to an embodiment.
7 is a block diagram illustrating an apparatus for generating a speech recognition model according to an embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following detailed description of the present invention refers to the accompanying drawings, which show by way of illustration a specific embodiment in which the present invention may be practiced, in order to clarify the objects, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention.

Throughout the detailed description and claims of the present invention, 'learning' or 'learning' is a term referring to performing machine learning through computing according to a procedure, and refers to a mental action such as human educational activity. It is not intended to be, and training is used in the generally accepted sense of machine learning.

Also, throughout this description and claims, the word 'comprise' and variations thereof are not intended to exclude other technical features, additions, components or steps. In addition, 'one' or 'an' is used to mean more than one, and 'another' is limited to at least a second or more.

Other objects, advantages and characteristics of the present invention will become apparent to a person skilled in the art in part from this description and in part from practice of the present invention. The following illustrations and drawings are provided by way of illustration and are not intended to limit the invention. Accordingly, the details disclosed herein with respect to a specific structure or function should not be construed in a limiting sense, but merely provide a representative guideline for those skilled in the art to variously practice the present invention with substantially any suitable detailed structures. It should be interpreted as basic data.

Moreover, the invention encompasses all possible combinations of the embodiments indicated herein. It should be understood that various embodiments of the present invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it should be understood that the position or arrangement of individual components in each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scope equivalents as those claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

In this specification, unless indicated otherwise or clearly contradicted by context, items referred to in the singular encompass the plural unless the context requires otherwise. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for illustratively explaining the structure of a transformer used for speech recognition of the present invention.

The transformer 100 used for speech recognition does not use a cyclic structure and consists only of a feed-forward neural network. The transformer 100 is not difficult to implement and has the advantage of a fast training speed. More specifically, the transformer 100 may include an encoder 110 and a decoder 120 . The encoder 110 may include a multi-head attention layer 111 and a feed-forward network 112 . The feed forward network 112 may be a combination of a convolutional layer (eg, 1D convolutional layer) and an activation layer (eg, ReLU).

Also, the decoder 120 may include multi-head attention layers 121 and 122 and a feed forward network 123 . The feed forward network 123 may be a combination of a convolutional layer (eg, 1D convolutional layer) and an activation layer (eg, ReLU). The multi-head attention layer 122 of the decoder 120 may perform attention based on the output 113 of the encoder 110 .

을 히든 표현(Hidden representation)

로 변환할 수 있고, 디코더(120)는 히든 표현

을 입력으로 하여 출력 대상인 텍스트에 대응하는 출력 시퀀스

를 출력할 수 있다. 트랜스포머(100)의 상세 동작은 비특허문헌 1 :[Dong, Linhao, Shuang Xu, and Bo Xu. "Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition." 2018 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP).IEEE, 2018.]에 개시된 바와 같다.The transformer 100 inputs a feature sequence extracted from the input voice signal through the encoder 110 .

hidden representation

can be converted to , and the decoder 120

An output sequence corresponding to the text to be output with as input

can be printed out. Detailed operation of the transformer 100 is described in Non-Patent Document 1: [Dong, Linhao, Shuang Xu, and Bo Xu. "Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition." 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).IEEE, 2018.].

The encoder 110 illustrated in FIG. 1 is only an exemplary form, and the encoder 110 constituting the transformer 100 is configured by stacking a plurality of encoder layers having a multi-head attention layer and a feed forward layer as sub-layers. It will be understood by those of ordinary skill in the art. Similarly, the decoder 120 constituting the transformer 100 includes two types of multi-head attention layers (a general multi-head attention layer and a multi-head attention layer using the output of the encoder) and a plurality of decoder layers using the feed forward layer as sub-layers. It may be configured by stacking.

In addition, the encoder 110 and the decoder 120 may have a positional encoding added to the input. Positional encoding may include absolute and relative information on the sequence of the input, in order to use the sequence of the sequence for speech recognition.

The transformer 100 is different from the existing recurrent neural network-based speech recognition model in the following two matters.

i) The encoder 110 and the decoder 120 include multi-head attention layers 111 , 121 , 122 and feed forward networks 112 , 123 configured based on a convolutional layer.

ii) At the decoder 120, an attention based on the output 112 of the encoder 110 is performed (in the existing recurrent neural network, an additional attention step is performed between the encoder and the decoder)

2 is a diagram for explaining in more detail a multi-head attention layer included in a transformer according to an embodiment.

The multi-head attention layer may be a layer for calculating a representative value of the input sequence by associating different positions of the input sequence.

The multi-head attention layer may be configured as shown in (a) of FIG. 2 .

(Query),

(Key),

(Value)에 대한 선형 투영(linear projection)을 수행하는 복수의 투영 레이어(211), 헤드의 개수인

개의 어텐션 레이어(212), 어텐션 레이어(212)의 출력을 결합하는 결합 레이어(213), 결합 레이어(213)의 출력을 선형 투영하는 투영 레이어(214)를 포함할 수 있다.According to an embodiment, the multi-head attention layer corresponds to the input

(Query),

(Key),

A plurality of projection layers 211 that perform linear projection on (Value), the number of heads

Attention layer 212 , a coupling layer 213 coupling the output of the attention layer 212 , and a projection layer 214 linearly projecting the output of the coupling layer 213 may be included.

,

,

를 출력에 매핑하는 동작을 수행하고,

,

,

의 값은 모두 벡터에 해당할 수 있다. 어텐션 레이어(212)의 출력은

값들의 가중합에 기초하여 결정되는데, 가중치는

와 이에 대응하는

의 함수로 결정될 수 있다. 보다 구체적으로, 어텐션 레이어(212) 중 어느 하나의 어텐션 레이어에서 수행되는 어텐션 연산은 하기 수학식 1에 기초하여 수행될 수 있다.According to one embodiment, the attention layer 212 is

,

,

perform an action that maps to the output,

,

,

All values of may correspond to vectors. The output of the attention layer 212 is

It is determined based on a weighted sum of values, where the weight is

and corresponding

can be determined as a function of More specifically, the attention operation performed in any one of the attention layers 212 may be performed based on Equation 1 below.

와

의 차원은

이고,

의 차원은

이다.in Equation 1

Wow

the dimension of

ego,

the dimension of

to be.

,

,

를 각각의 차원인

로

회 서로 다른 선형 투영(투영 레이어(211)가 수행)을 수행하고, 투영 결과에 앞서 설명된 수학식 1의 어텐션을 병렬적으로 적용할 수 있다. 수학식 1이 병렬적으로 적용된 결과는 결합 레이어(213)를 통해 concatenation 연산이 수행될 수 있고, 추가적으로, 투영 레이어(214)를 통해 선형 투영이 진행됨으로써 멀티 헤드 어텐션 레이어의 최종 출력이 생성될 수 있다.As shown in Figure 2, the multi-head attention layer does not perform a single attention,

,

,

is each dimension

as

Linear projection (performed by the projection layer 211) may be performed differently each time, and the attention of Equation 1 described above may be applied in parallel to the projection result. As a result of applying Equation 1 in parallel, a concatenation operation may be performed through the combining layer 213, and in addition, a final output of the multi-head attention layer may be generated by linear projection proceeding through the projection layer 214. have.

The operation method of the multi-head attention layer described above may correspond to Equation 2 below.

는 투영 레이어(214)로부터 출력되는 멀티 헤드 어텐션 레이어의 최종 출력,

는 투영 레이어(211)를 통해

에 적용되는 투영 행렬,

는 투영 레이어(211)를 통해

에 적용되는 투영 행렬,

는 투영 레이어(211)를 통해

에 적용되는 투영 행렬,

는 투영 레이어(214)를 통해 적용되는 투영 행렬을 의미할 수 있다.

는 헤드의 개수,

는

의 차원,

는

의 차원,

는

의 차원,

는 트랜스포머의 차원을 의미할 수 있다.

is the final output of the multi-head attention layer output from the projection layer 214,

is through the projection layer 211

projection matrix applied to

is through the projection layer 211

projection matrix applied to

is through the projection layer 211

projection matrix applied to

may mean a projection matrix applied through the projection layer 214 .

is the number of heads,

Is

dimension of,

Is

dimension of,

Is

dimension of,

may mean the dimension of the transformer.

According to an embodiment, each of the attention layers 212 may have the shape shown in (b), and may perform scaled dot-product attention. Each attention layer 212 includes a matrix multiplication layer 221 for performing matrix multiplication, a scale layer 222 for scaling, a mask layer 223 for masking inappropriate values, and a softmax layer ( 224 ) and a matrix product layer 225 .

,

,

는 각각 행렬 형태로 표현되며, 행렬곱 레이어(221, 225)는 모든 행과 열에 포함된 요소들을 곱하는 행렬 곱셈 연산에 기초하여 출력을 결정한다. 행렬곱 레이어(221, 225)은 행렬의 차원이 nxn인 경우,

과 연관된 연산 시간이 소요된다. 하지만,

,

,

의 행렬에 포함된 요소는 모두 유의미한 값을 가지는 것이 아닐 수 있다. 예를 들어, 행렬에 포함된 소정의 요소들은 0이거나, 매우 미세한 값을 포함할 수 있고, 해당 요소는 연산 결과에 미치는 영향이 매우 미미할 수 있다. 이 경우,

,

,

각각의 행렬의 모든 요소에 대한 연산을 수행하는 것은 의미가 높지 않을 수 있고, 이러한 연산을 통해서도 어텐션 결과를 도출하는 데에 유의미한 문제가 발생하지 않을 수 있다. 따라서, 행렬곱 레이어(221, 225)는

,

,

의 행렬에서 소정의 임계치 이하인 요소에 대한 연산을 제외하고, 나머지 요소들에 대한 연산에 기초하여 어텐션 관련 연산을 수행할 수 있다. 보다 구체적으로, 행렬곱 레이어(221, 225)에서는 행렬의 연산 과정에서 소정의 임계값(예를 들어, 10^e-04) 이하의 값들을 제외한 값에서만 연산이 수행할 수 있다. 이를 통해 행렬곱 레이어(221, 225)은 어텐션 적용을 위한 연산에 필요한 메모리를 크게 절감할 수 있는 효과를 제공할 수 있다.According to one embodiment, the previously described

,

,

are each expressed in a matrix form, and the matrix multiplication layers 221 and 225 determine an output based on a matrix multiplication operation that multiplies elements included in all rows and columns. Matrix product layers (221, 225) when the dimension of the matrix is nxn,

The computation time associated with However,

,

,

The elements included in the matrix of may not all have significant values. For example, predetermined elements included in the matrix may be 0 or may include very fine values, and the corresponding elements may have very insignificant influence on the operation result. in this case,

,

,

It may not be meaningful to perform an operation on all elements of each matrix, and there may not be a significant problem in deriving an attention result through such an operation. Accordingly, the matrix product layers 221 and 225 are

,

,

An attention-related operation may be performed based on the operation on the remaining elements, except for the operation on the element that is less than or equal to a predetermined threshold in the matrix of . More specifically, in the matrix multiplication layers 221 and 225 , an operation may be performed only on values excluding values less than or equal to a predetermined threshold (eg, 10^e-04) in the matrix operation process. Through this, the matrix multiplication layers 221 and 225 may provide an effect of greatly reducing the memory required for an operation for applying the attention.

In addition, in the matrix multiplication layers 221 and 225, the complexity of the operation can be significantly reduced by additionally applying the Strassen algorithm for reducing the complexity of the function operation.

According to an embodiment, the mask layer 223 may be selectively included in the attention layer 212 . More specifically, the mask layer 223 may be selectively included in the attention layer 212 to prevent a leftward information flow or to preserve an auto-regressive property of a decoder.

3 is a diagram for describing a method of generating a voice recognition model according to an exemplary embodiment.

Referring to FIG. 3 , the apparatus for generating a speech recognition model may obtain a candidate speech recognition model in a cell-based search space through step 310 . The process of obtaining the candidate speech recognition model may be performed in such a way that an encoder cell and a decoder cell are respectively searched from a search space, and a speech recognition model in which the searched encoder cell and decoder cell are combined is obtained. Each encoder cell and decoder cell can be searched from the search space through any combination of the values of the search fields.

The apparatus for generating a speech recognition model performs additional training on a candidate speech recognition model that satisfies a predetermined condition among a plurality of candidate speech recognition models that have been initially trained to output text corresponding to the input speech data through step 320 . By doing so, a final speech recognition model can be generated.

More specifically, the apparatus for generating a speech recognition model selects a candidate speech recognition model that satisfies a predetermined condition from among candidate speech recognition models that have undergone initial training through a training algorithm referred to as progressive dynamic hurdles (PDH), and recognizes the selected candidate speech. By performing additional training on the model, the final speech recognition model can be generated. PDH may be a method of training a high-performance speech recognition model while saving resources by excluding candidate speech recognition models that do not satisfy a predetermined performance condition from additional training targets during iterative training. . A method of training a speech recognition model using PDH will be described in more detail with reference to FIG. 5 attached below.

4 is a diagram for explaining a method of obtaining a candidate speech recognition model using a cell-based search space according to a method of generating a speech recognition model according to an embodiment.

In the cell-based search space according to an embodiment, the network structure may be searched in units of cells, unlike the existing NAS method of searching all individual network elements. Candidate speech recognition models can be searched using fewer resources than the existing NAS method that requires searching for each detailed individual element through a cell-based search space.

More specifically, the apparatus for generating a speech recognition model according to an embodiment may search for an encoder cell and a decoder cell in a cell-based search space, and obtain a candidate speech recognition model by combining the found encoder and decoder cells.

More specifically, the apparatus for generating a speech recognition model may search for an encoder cell and a decoder cell from a search space through any combination of values for a plurality of search fields described below.

According to an embodiment, each cell 410 retrieved from the cell search space may consist of a plurality of blocks 420 . For example, the encoder cell may consist of 6 blocks, and the decoder cell may consist of 9 blocks.

Each block 420 is a new branch by combining the outputs of two branches, the left branch 432 and the right branch 431, and each branch 431 and 432 that have different hidden states as inputs. It may include a coupling layer that generates an output corresponding to the hidden state.

Each of the branches 431 and 432 has a) a hidden state used as an input, b) the type of transformation layer being performed, c) whether normalization is performed before transformation is performed, d) the dimension of the output, e) used It may be classified based on the type of activation function.

The apparatus for generating a speech recognition model according to an embodiment includes i) a branch-level search field corresponding to a) to e) described above (each search field for a right branch and a left branch may be configured separately) and ii) A cell to be included in the candidate speech recognition model may be searched from the search space based on the block-level search field for the type of the combined layer.

More specifically, the apparatus for generating a speech recognition model may search for a cell to be included in a candidate speech recognition model from a search space through any combination of branch-level search field information and block-level search field information.

For example, in case a), any one of values '0' to 'i-1' may be selected in the process of searching for the i-th block. When '0' is selected, the input of the transformer is determined as the input hidden state of the corresponding branch, and when 'j(0<j<i-1)' is selected, the output of the j-th block is the input hidden state of the corresponding branch. state can be determined.

In case b), it may correspond to a transform to be applied to the branch input, for example, 1D convolutional layer, h Head attention, Gated Linear Unit, no transform applied. Arbitrary transformations can be included, such as identity that does not exist, and that no output is generated.

In c), it may be information on whether to proceed with normalization before the transform layer of b).

In d), it may be information about the dimension of the output of the corresponding branch.

In e), as the type of activation function to be applied to the branch, ReLU, Swish, Leaky_ReLU, none, etc. may be included.

In addition, the type of the combination layer may include addition, multiplication, and concatenation.

In the cell 410 found from the search space, an individual block 420 of the cell 410 may be determined through a process of connecting the individual branches found based on the search field information through a coupling layer. A connection relationship between the individual blocks 420 may be determined based on an order set in the process of determining the input hidden state of the individual blocks 420 . In addition, a hidden state that is not selected in the search process of the cell 410 may be added to the final output of the aforementioned blocks by a summation operation. Finally, the combined individual blocks 420 are connected to the transformer inputs and outputs so that a final candidate speech recognition model can be created.

According to an embodiment, the process of searching each cell 410 may be repeated a predetermined number of times, and it will be understood by those skilled in the art that in this case, an encoder or decoder to which the searched cells are connected may be generated.

According to an embodiment, a candidate speech recognition model may be searched to include a transform layer corresponding to one or more multi-head attentions in the encoder cell. In addition, a candidate speech recognition model may be searched to include a transform layer corresponding to two or more multi-head attentions in the decoder cell. In the case of a decoder cell, a candidate speech recognition model may be searched to include a transform layer corresponding to a general multi-head attention and a transform layer corresponding to a multi-head attention using an encoder input.

According to an embodiment, the apparatus for generating a speech recognition model may generate a candidate speech recognition model such that all blocks included in a searched cell may include one block having the same shape. More specifically, in the case of an encoder cell, 6 blocks are included, each of which is composed of a set of 3 blocks composed of 2 blocks of the same type, and a decoder cell is composed of 4 blocks composed of 2 blocks of the same type. Candidate speech recognition models may be searched for to be composed of a set of blocks. Through this, resources in the process of searching for a candidate speech recognition model may be reduced, and the search speed may be improved.

The search space according to an embodiment may be configured to search only models in which the number of training parameters is less than or equal to a predetermined threshold. For example, the predetermined threshold may be determined to be 3 million less than the number of learning parameters of the existing transformer set as the baseline. Accordingly, candidate speech recognition models requiring learning parameters greater than or equal to a predetermined threshold may be excluded from the search space in advance, and the number of candidate speech recognition models to be searched in the search space may be significantly reduced. Through this, the capacity of the search space can be significantly reduced and resources for searching can be reduced.

5 is a flowchart illustrating a method of generating a voice recognition model based on a PDH according to an embodiment.

이터레이션(Iteration)에 해당하는 초기 훈련을 수행할 수 있다.

이터레이션(Iteration)에 해당하는 훈련이 완료된 상태인 경우, 훈련 대상 후보 음성 인식 모델은 이하 단계(530, 540)에서 설명되는 방식을 통해 결정된 추가 훈련 대상이 단계(510)의 훈련 대상 후보 음성 인식 모델일 수 있다. Referring to FIG. 5 , the apparatus for generating a speech recognition model may perform training on a candidate speech recognition model obtained from a search space in operation 510 . In a situation where training for the candidate speech recognition model is started, the apparatus for generating a speech recognition model obtains c initial candidate speech recognition models from the search space, and for the c initial candidate speech recognition models

Initial training corresponding to iteration may be performed.

When the training corresponding to the iteration is completed, the training target candidate voice recognition model uses the additional training target determined through the method described in steps 530 and 540 below to recognize the training target candidate voice in step 510 . can be a model.

Training the candidate speech recognition model may mean training the candidate speech recognition model to output a text sequence corresponding to the speech signal from the feature sequence extracted from the input speech signal. For example, the feature sequence extracted from the voice signal may be a spectogram or MFCC (Mel-frequency cepstral coefficients) of the voice signal, but is not limited thereto. It may include any features extracted. The output of the candidate speech recognition model may be a text sequence corresponding to the feature.

The apparatus for generating a speech recognition model may determine whether the number of times of training performed on the speech recognition model satisfies a predetermined number of training in operation 520 . More specifically, the apparatus for generating a speech recognition model may determine whether the number of additional training sessions satisfies a predetermined threshold based on Equation 3 in step 520 .

는 t번째 이터레이션(iteration),

는 미리 결정된 최대 훈련 횟수일 수 있다.

is the t-th iteration,

may be a predetermined maximum number of training.

That is, the apparatus for generating a speech recognition model may determine that the predetermined threshold is not satisfied when the total sum of the performed iterations does not satisfy the predetermined maximum number of training times.

When the number of training times does not meet a predetermined threshold, the apparatus for generating a speech recognition model may determine a candidate recognition model having a performance greater than or equal to the threshold among the candidate speech recognition models trained in step 530 as an additional training target. .

More specifically, the apparatus for generating a speech recognition model may evaluate the performance of a trained candidate speech recognition model based on a verification set (eg, information consisting of a feature sequence extracted from a predetermined speech signal and a text sequence corresponding thereto). have. For example, the performance of the trained candidate speech recognition model may be determined as a numerical value calculated based on how many words are correctly recognized from a verification set consisting of a plurality of words, but is not limited thereto. Any method that can be evaluated numerically may be applied. For example, the performance of the trained candidate speech recognition model may be expressed as Equation (4).

는 i번째 이터레이션에 대응되는 훈련이 진행된 후보 음성 인식 모델에 대한 성능 벡터,

는 j번째 훈련된 후보 음성 인식 모델의 성능을 나타낼 수 있으며, c는 i번째 이터레이션에 대응되는 훈련이 진행된 후보 음성 인식 모델의 총 개수로 초기 훈련시 미리 결정된 수일 수 있다.

is a performance vector for the trained candidate speech recognition model corresponding to the i-th iteration,

may represent the performance of the j-th trained candidate speech recognition model, and c may be the total number of candidate speech recognition models trained corresponding to the i-th iteration, which may be a predetermined number during initial training.

The apparatus for generating a speech recognition model may determine an additional training target from among candidate speech recognition models for which training has been completed based on Equation (5).

는 i번째 이터레이션에 대응되는 훈련이 진행된 j번째 후보 음성 인식 모델의 성능,

는 i번째 이터레이션에 대응되는 훈련이 진행된 음성 인식 모델 성능을 평균치로, 수학식 6와 같이 산출될 수 있다.

is the performance of the j-th candidate speech recognition model trained corresponding to the i-th iteration,

is the average value of the performance of the trained voice recognition model corresponding to the i-th iteration, and may be calculated as in Equation (6).

로 표현될 수 있고, 이는 i+1 번째 이터레이션에 대응되는 추가 훈련이 수행될 후보 음성 인식 모델의 집합을 나타낼 수 있다.For example, the apparatus for generating a speech recognition model may determine, as an additional training target, a candidate speech recognition model having better performance than the average performance according to Equation 6 among candidate speech recognition models trained corresponding to the i-th iteration. have. For example, an additional set of training targets is

may be expressed as , which may indicate a set of candidate speech recognition models on which additional training corresponding to the i+1th iteration is to be performed.

The apparatus for generating a speech recognition model may obtain an additional candidate speech recognition model from the search space in step 540 and determine a candidate speech recognition model satisfying a predetermined condition among the obtained candidate speech recognition models as an additional training target.

에 대응되는 초기 훈련을 수행하고, 초기 훈련이 수행된 후보 음성 인식 모델 중 성능이 소정의 조건을 만족하는 후보 음성 인식 모델을 선택하여

의 구성 요소로 결정할 수 있다. 예를 들어, 음성 인식 모델 생성 장치는

(

는 i번째 이터레이션이 진행된 후보 음성 인식 모델 중 추가 훈련이 진행되는 것으로 결정된 후보 음성 인식 모델의 개수를 의미함)의 개수에 대응되는 추가 후보 음성 인식 모델을 서치 스페이스로부터 검색하고, 추가 후보 음성 인식 모델에 대해

에 대응하는 초기 훈련을 수행할 수 있다. 음성 인식 모델 생성 장치는

에 대응되는 훈련이 수행된 후보 음성 인식 모델 중

이상의 성능을 가지는 후보 음성 인식 모델을

의 구성 요소로 결정할 수 있다. 음성 인식 모델 생성 장치는 i)

에 대응되는 훈련이 진행된 후보 음성 인식 모델 중 성능이

보다 큰 후보 음성 인식 모델 및 ii) 서치 스페이스로부터 추가적으로 획득하여 초기 훈련이 수행된 후보 음성 인식 모델 중 성능이

보다 큰 후보 음성 인식 모델을

의 구성 요소로 결정할 수 있다. More specifically, the apparatus for generating a speech recognition model relates to the additionally obtained candidate speech recognition model.

by performing initial training corresponding to , and selecting a candidate speech recognition model whose performance satisfies a predetermined

can be determined as a component of For example, the apparatus for generating a speech recognition model is

(

means the number of candidate voice recognition models that are determined to undergo additional training among the candidate voice recognition models subjected to the i-th iteration) from the search space, and additional candidate voice recognition models corresponding to the number are searched for, and additional candidate voice recognition is performed. about the model

It is possible to perform initial training corresponding to Speech recognition model generation device

Among the candidate speech recognition models trained corresponding to

A candidate speech recognition model with the above performance

can be determined as a component of The apparatus for generating a speech recognition model is i)

Among the candidate speech recognition models trained corresponding to

Larger candidate speech recognition models and ii) the performance of candidate speech recognition models additionally acquired from the search space and subjected to initial training

larger candidate speech recognition models.

can be determined as a component of

에 대응되는 훈련이 완료된 경우, 음성 인식 모델 생성 장치는

에 포함된 후보 음성 인식 모델들을 최종 음성 인식 모델로 결정할 수 있다.According to an embodiment, the apparatus for generating a speech recognition model uses the candidate speech recognition model trained in step 550 as the final speech recognition model when the number of pre-trained trainings meets a predetermined number of training in step 520 . can decide For example, before performing step 520

When the training corresponding to

Candidate speech recognition models included in ? may be determined as final speech recognition models.

6 is a diagram illustrating an example of a speech recognition model generated through a method for generating a speech recognition model according to an embodiment.

Referring to FIG. 6 , a speech recognition model generated through the method for generating a speech recognition model according to an embodiment may be generated in a structure including an encoder 610 and a decoder 620 .

The encoder 610 has a first form 612 corresponding to the structure of a conventional transformer that is configured by a serial stacking of a conventional convolution layer and a multi-head attention layer shown in FIG. 1 and the present application. The second form 611 newly constructed through the search space of the invention may be a combined form. The second form 611 includes a parallel structure generated based on the branch structure of the search space, and includes various transformation layers (eg, a gated linear unit, etc.) not included in the first form 612 . can be

As shown in FIG. 1, the decoder 620 is also newly formed through the first form 622 corresponding to the structure of the existing transformer and the search space of the present invention, which is composed of a serial stacking of a conventional convolution layer and an attention layer. The configured second form 621 may be a combined form. The second form 622 includes a parallel structure based on the branch structure of the search space, and is composed of various transformation layers (eg, Left sep-conv 1D, etc.) not included in the first form 622 . can be

7 is a block diagram illustrating an apparatus for generating a speech recognition model according to an embodiment.

Referring to FIG. 7 , the apparatus 700 for generating a speech recognition model according to an embodiment includes a processor 720 . The apparatus 700 for generating a speech recognition model may further include a memory 710 and a communication interface 730 . The processor 720 , the memory 710 , and the communication interface 730 may communicate with each other through a communication bus (not shown).

The processor 720 includes a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU) or a tensor processing unit (TPU), a cache memory, and a data bus ( data bus) and the like. In addition, the operating system may further include a software configuration of an application for performing a specific purpose.

The processor 720 obtains a plurality of candidate speech recognition models based on a combination of cells searched from the cell-based search space, and selects a predetermined one of a plurality of candidate speech recognition models that have been initially trained to output text corresponding to the input speech data. A final speech recognition model can be generated by performing additional training on a candidate speech recognition model that satisfies the condition of .

The memory 710 may be a volatile memory or a non-volatile memory.

In addition, the processor 720 may execute a program and control the apparatus 700 for generating a voice recognition model. The program code executed by the processor 720 may be stored in the memory 710 . The voice recognition model generating apparatus 700 may be connected to an external device (eg, a personal computer or a network) through an input/output device (not shown) and exchange data. The apparatus for generating a speech recognition model 700 may be mounted on a server.

Based on the description of the above embodiments, those skilled in the art will appreciate that the method and/or processes of the present invention and the steps thereof may be implemented in hardware, software, or any combination of hardware and software suitable for a particular application. point can be clearly understood. The hardware may include general purpose computers and/or dedicated computing devices or specific computing devices or special features or components of specific computing devices. The processes may be realized by one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices, having internal and/or external memory. Additionally, or alternatively, the processes may be configured to process an application specific integrated circuit (ASIC), a programmable gate array, a programmable array logic (PAL) or electronic signals. It may be implemented with any other device or combination of devices. Moreover, the objects of the technical solution of the present invention or parts contributing to the prior arts may be implemented in the form of program instructions that can be executed through various computer components and recorded in a machine-readable recording medium. The machine-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded in the machine-readable recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the machine-readable recording medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM, DVD, Blu-ray, and a magneto-optical medium such as a floppy disk (magneto-optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include any one of the devices described above, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software, or stored and compiled or interpreted for execution on a machine capable of executing any other program instructions. can be created using a structured programming language such as C, an object-oriented programming language such as C++, or This includes not only bytecode, but also high-level language code that can be executed by a computer using an interpreter or the like.

Accordingly, in one aspect according to the present invention, when the above-described method and combinations thereof are performed by one or more computing devices, the methods and combinations of methods may be implemented as executable code for performing respective steps. In another aspect, the method may be implemented as systems performing the steps, the methods may be distributed in various ways across devices or all functions may be integrated into one dedicated, standalone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.

For example, the hardware device may be configured to operate as one or more software modules to perform processing in accordance with the present invention, and vice versa. The hardware device may include a processor, such as an MPU, CPU, GPU, TPU, coupled with a memory such as ROM/RAM for storing program instructions and configured to execute instructions stored in the memory, an external device and a signal It may include a communication unit that can send and receive. In addition, the hardware device may include a keyboard, a mouse, and other external input devices for receiving commands written by developers.

In the above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments, Those of ordinary skill in the art to which the present invention pertains can devise various modifications and variations from these descriptions.

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and not only the claims described below, but also all modifications equivalently or equivalently to the claims described below belong to the scope of the spirit of the present invention. will do it

Such equivalent or equivalent modifications shall include, for example, logically equivalent methods capable of producing the same results as practiced by the methods according to the present invention, the spirit and scope of the present invention. should not be limited by the above examples, and should be understood in the broadest sense permitted by law.

Claims (12)

A method for generating a voice recognition model performed by a voice recognition model generating apparatus, the method comprising:
obtaining a plurality of candidate speech recognition models based on a combination of cells searched from a cell-based search space; and
generating a final speech recognition model by performing additional training on at least one candidate speech recognition model selected from among the plurality of candidate speech recognition models on which initial training has been performed to output text corresponding to input speech data;
A method of generating a speech recognition model, comprising:
According to claim 1,
The obtaining step is
obtaining an encoder cell from the search space;
obtaining a decoder cell from the search space; and
generating the plurality of candidate speech recognition models based on the combination of the encoder cell and the decoder cell;
A method of generating a speech recognition model, comprising:
According to claim 1,
The cell is
at least one block,
Each of the at least one block,
two branches that perform operations based on different inputs; and
A bonding layer that combines the outputs of the two branches.
A method for generating a speech recognition model, comprising:
3. The method of claim 2,
each of the encoder cell and the decoder cell,
A method of generating a speech recognition model, comprising at least one block corresponding to multi-head attention.
5. The method of claim 4,
The block corresponding to the multi-head attention is,
A method of generating a speech recognition model, excluding elements included in the input matrix that are less than or equal to a predetermined threshold, and performing an operation through the remaining elements.
3. The method of claim 2,
The cell is
at least one block,
Each of the at least one block,
A method of generating a speech recognition model having the same shape as another block included in the cell.
According to claim 1,
the at least one candidate speech recognition model is a candidate speech recognition model that satisfies a predetermined condition among the plurality of candidate speech recognition models;
The predetermined condition is that the number of previously performed training is less than a predetermined number of times threshold and the performance of the candidate speech recognition model is greater than or equal to the predetermined performance threshold, the method for generating a speech recognition model.
According to claim 1,
The step of generating the final speech recognition model comprises:
If the number of previously advanced training is less than the preset number threshold,
determining a first additional training target by selecting a candidate speech recognition model whose performance is greater than or equal to a performance threshold from among the trained candidate speech recognition models;
determining a second additional training target from the candidate speech recognition model additionally obtained from the search space; and
performing additional training on the first additional training target and the second additional training target
A method of generating a speech recognition model, comprising:
9. The method of claim 8,
The performance threshold is
The method of generating a speech recognition model, which is determined as an average value of the performance of the trained candidate speech recognition model.
A computer-readable storage medium storing one or more programs including instructions for performing the method of claim 1 .
Obtaining a plurality of candidate speech recognition models based on the combination of cells from a cell-based search space, training the plurality of candidate speech recognition models to output text corresponding to input speech data, A processor for generating a final speech recognition model by performing additional training on at least one candidate speech recognition model selected from among the plurality of trained candidate speech recognition models
Containing, speech recognition model generating device.
According to claim 1,
The method for generating a speech recognition model, characterized in that the final speech recognition model is a candidate speech recognition model having a preset number of training corresponding to a predetermined number threshold.

Images