JP7209330B2 - classifier, trained model, learning method - Google Patents

Description

The present technology relates to a discriminator that outputs a sequence of labels for an input signal, a trained model directed to the discriminator, and a learning method for the discriminator.

In the field of speech recognition, very deep convolutional networks are known to significantly outperform conventional deep neural networks (DNNs).

For speech recognition tasks, very deep residual time-delay neural networks have been proposed (see, for example, Non-Patent Document 1). Unlike less layered time-delay neural networks (TDNN) and feedforward sequential memory networks (FSMN), ultra-deep residual time-delay neural networks can learn longer context dependencies without recursive feedback. Therefore, it is possible to avoid problems such as time delay that may occur when using a BLSTM (bidirectional long short term memory) network. Therefore, application to E2E (end-to-end) training that integrates acoustic model and language model training is considered promising.

A model using a CTC (connectionist temporal classification) framework is known as an effective E2E framework for speech recognition (see, for example, Non-Patent Document 2). The CTC framework focuses on solving the sequence labeling problem that arises between incoming variable-length speech frames and outgoing labels (units such as phones, characters, syllables, etc.). ing. The CTC modeling technique greatly simplifies the acoustic model pipeline. Therefore, the CTC framework does not require frame-level labels or an initial GMM-HMM (Gaussian mixture model and hidden Markov model) model (corresponding to an acoustic model).

The present inventors have previously proposed training a CTC-based E2E model using a very deep residual time-delay structure (e.g., Non-Patent Document 3). reference).

S. Zhang, M. Li, Z. Yan, and L. Dai, "Deep-FSMN for large vocabulary continuous speech recognition," in arXiv preprint (accepted for ICASSP2018) arxiv:1803.05030, 2018. A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber, "Connectionist temporal classification: labeling unsegmented sequence data with recurrent neural networks," in Proc. ICML, 2006. S. Li, X. Lu, R.Takashima, P. Shen, and H. Kawai, "Improving CTC-based acoustic model with very deep residual neural network," in Proc. INTERSPEECH, 2018.

The enormous number of parameters that define an ultradeep model poses the problem of complicating optimization and degrading generalization performance. In our research, a well-tuned ultradeep model for a particular system cannot be directly applied to a system with a different data set. It is not easy to find a structure that exhibits good performance, and it is necessary to conduct numerous experiments on all candidate network structures.

The purpose of this technology is to provide a model capable of providing an appropriate network structure according to the target system.

According to one aspect of the invention, a discriminator is provided that outputs a sequence of labels for an input signal. The discriminator includes an input layer that sequentially generates a first feature vector for each frame of a predetermined time width from an input signal, a plurality of stacked residual blocks following the input layer, and an output side of the plurality of residual blocks. and an output layer connected to . Each of the plurality of residual blocks adjusts weights between the stacked plurality of time delay layers, shortcut paths bypassing the plurality of time delay layers, paths passing through the plurality of time delay layers, and shortcut paths. and an attention module that A plurality of time delay layers has delay elements that provide a predetermined timestep delay to the input. At each time step, the attention module performs an Update weights.

The attention module may include a fully connected layer connected to the output of the corresponding residual block and the shortcut path, and a softmax function connected to the fully connected layer.

The attention module is configured to calculate a first weight for paths through the multiple time delay layers and a second weight for shortcut paths such that the sum of the first weight and the second weight is one. can be

Each of the time delay layers receives, for an input vector, a first internal vector corresponding to a past frame obtained by moving back a time step from the current frame, which is the frame corresponding to the input vector, and a first internal vector corresponding to the past frame by the time step. A second internal vector corresponding to a future frame advanced in time may be generated.

The input signal may be a speech signal, and the discriminator may output a label indicating the speech recognition result for the speech signal.

According to another aspect of the invention, a trained model is provided for operating a computer to output a sequence of labels for an input signal. The trained model includes an input layer that sequentially generates a first feature vector for each frame of a predetermined time width from an input signal, a plurality of stacked residual blocks following the input layer, and outputs of the plurality of residual blocks. and an output layer connected to the side. Each of the plurality of residual blocks adjusts weights between the stacked plurality of time delay layers, shortcut paths bypassing the plurality of time delay layers, paths passing through the plurality of time delay layers, and shortcut paths. and an attention module that A plurality of time delay layers has delay elements that provide a predetermined timestep delay to the input. At each time step, the attention module performs an It is configured to update the weights.

According to still another aspect of the present invention, there is provided a method of learning a discriminator that outputs a sequence of labels for an input signal. The discriminator includes an input layer that sequentially generates a first feature vector for each frame of a predetermined time width from an input signal, a plurality of stacked residual blocks following the input layer, and an output side of the plurality of residual blocks. and an output layer connected to . Each of the plurality of residual blocks includes a plurality of stacked time delay layers and a shortcut path bypassing the plurality of time delay layers. A plurality of time delay layers has delay elements that provide a predetermined timestep delay to the input. The learning method includes a first training step of using a training data set to determine the parameters that define the network of classifiers; and an adding step of adding the adjusting attention module. At each time step, the attention module performs an Configured to update weights. The learning method includes a second training step of determining parameters defining attention modules using a training data set.

The second training step may comprise re-determining the values of all the parameters defining the network of classifiers, including the parameters defining the attention modules.

The second training step may comprise determining only the parameters defining the attention modules, with the parameters determined in the first training step fixed.

The learning method further includes providing an input signal to a discriminator attached with an attention module to remove a portion of the plurality of time delay layers based on changes in weight values calculated by the attention module. You can also try to

According to the present technology, it is possible to provide an appropriate network structure according to the target system.

FIG. 4 is a schematic diagram showing an application example using a trained model according to the present embodiment; FIG. 2 is a schematic diagram for explaining a learning method of the speech recognition system S shown in FIG. 1; FIG. 4 is a diagram for outlining processing contents in a basic CTC-based model according to the present embodiment; FIG. 1 is a schematic diagram showing an example of a network structure of a basic CTC-based model according to this embodiment; FIG. FIG. 4 is a schematic diagram showing the processing structure of a time delay layer included in the basic CTC-based model according to this embodiment; 5 is a schematic diagram showing an example of a network structure equivalent to the network structure shown in FIG. 4, which is equivalent to a network structure composed of three layers of residual blocks; FIG. FIG. 4 is a schematic diagram showing the main part of the network structure of the improved CTC-based model according to this embodiment; 1 is a schematic diagram showing an example of a hardware configuration that implements a speech recognition system S according to this embodiment; FIG. 4 is a flow chart showing a processing procedure of an improved CTC-based model learning method (retraining method) according to the present embodiment; 4 is a flow chart showing a processing procedure of an improved CTC-based model learning method (clipping method) according to the present embodiment; FIG. 4 is a diagram showing an example distribution of data transfer in the improved CTC-based model according to the present embodiment; FIG. 4 is a diagram for explaining a processing procedure of an improved CTC-based model learning method (network reconstruction method) according to the present embodiment; FIG. 5 is a diagram showing an example of temporal changes in scale factors calculated using the improved CTC-based model according to the present embodiment; 4 is a flow chart showing a processing procedure of an improved CTC-based model learning method (network reconstruction method) according to the present embodiment; 4 is a flow chart showing a processing procedure of an improved CTC-based model decoding method according to the present embodiment; FIG. 10 is a diagram showing an example of change in attention score of the improved CTC-based model according to the present embodiment;

Embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are denoted by the same reference numerals, and the description thereof will not be repeated.

[A. Application example]
First, an application example using a trained model according to the present embodiment will be described.

FIG. 1 is a schematic diagram showing an application example using a trained model according to this embodiment. FIG. 1 shows a speech recognition system S as an application example. The speech recognition system S receives an input of a speech signal and outputs a recognition result. More specifically, the speech recognition system S receives an input of a speech signal and extracts a feature vector from time-series data for each predetermined section (hereinafter also referred to as "speech frame"). and a recognition engine 4 that receives input of vectors from the feature amount extraction unit 2 and outputs recognition results such as text.

The feature extraction unit 2 sequentially generates feature vectors for each audio frame from the input audio signal. The feature vector output from the feature amount extraction unit 2 has a predetermined number of dimensions, and reflects the feature amount of the portion corresponding to the corresponding audio frame of the input audio signal. The feature vectors are sequentially output according to the length of the input audio signal. Hereinafter, all or part of such a series of feature vectors will also be collectively referred to as an "acoustic feature sequence".

The recognition engine 4 inputs the feature vector for each speech frame output from the feature amount extraction unit 2 to the trained model and outputs text. Thus, the recognition engine 4 is configured with a trained model according to this embodiment and functions as a decoder. That is, the recognition engine 4 is an E2E framework (integrated with an acoustic model and a language model) for speech recognition, and receives input of speech frames and outputs corresponding text.

FIG. 2 is a schematic diagram for explaining the learning method of the speech recognition system S shown in FIG. Referring to FIG. 2, a training data set 40 consisting of speech signals 42 and corresponding text 44 is provided. A recognition result (text) from the recognition engine 4 is obtained by inputting the speech signal 42 to the feature amount extraction unit 2 and inputting the feature vectors sequentially generated by the feature amount extraction unit 2 to the recognition engine 4 . The network is optimized by sequentially updating the parameters of the network defining the recognition engine 4 based on the error between the recognition result from the recognition engine 4 and the label (text 44) corresponding to the input speech signal 42. be done.

[B. Basic network structure]
The trained model according to the present embodiment implements learning and optimization of the network structure by appropriately adding attention modules to the basic network structure described below. First, the basic network structure according to this embodiment will be described.

(b1: Overview)
In this embodiment, a basic network structure classified as a model using the CTC framework (hereinafter also referred to as "basic CTC-based model") is used. A basic CTC-based model is a discriminator that outputs a sequence of labels for an input signal. In the following, an example in which a speech signal is used as an input signal and the basic CTC-based model outputs a label indicating the speech recognition result for the speech signal will be mainly described. Application is possible.

As a typical example, the basic CTC-based model 1 according to the present embodiment is supplied with features of windows (including 10 to 15 speech frames) that are sequentially set in sentences of an input speech signal. Here, a sentence means a linguistically meaningful break, and usually includes a plurality of speech frames with a predetermined section length.

The output from the basic CTC-based model 1 according to this embodiment is a frame-level sequence called a path (hereinafter also referred to as "CTC output sequence"). The output sequence contains blanks (hereinafter also expressed as "φ") without any CTC label.

FIG. 3 is a diagram for outlining processing contents in the basic CTC based model 1 according to the present embodiment. Referring to FIG. 3, the CTC output is estimated by setting a window (including 10 to 15 speech frames) at the beginning of the sentence of the input speech signal and sliding the window to multiple locations. As shown in FIG. 3, the inputs to the basic CTC-based model 1 have forward paths only. That is, since only past information is required as input, there is no need to wait for the end of input speech.

In the following description, the basic CTC-based model 1 according to this embodiment is also referred to as "VResTD-CTC" (very deep residual time-delay neural network - CTC).

By training the basic CTC-based model 1 according to this embodiment, a learned model that implements the recognition engine 4 of FIG. 1 can be implemented. The feature quantity extraction unit 2 may be designed in advance based on empirical rules and the like.

FIG. 4 is a schematic diagram showing an example of the network structure of the basic CTC-based model 1 according to this embodiment. Referring to FIG. 4, the basic CTC-based model 1 includes feature vectors ( acoustic feature sequence) is input. The basic CTC-based model 1 sequentially outputs texts (subword sequences) corresponding to sequentially input feature vectors.

More specifically, the basic CTC-based model 1 includes a fully connected layer 10 (hereinafter also referred to as "FC" or "FC layers") as an input layer, a plurality of residual blocks 20, and an output layer 30. including.

A fully connected layer 10 as an input layer receives input of feature vectors and generates internal vectors of a required number of dimensions.

A plurality of residual blocks 20 are arranged subsequent to the fully connected layer 10 . A plurality of residual blocks 20 are stacked on each other to form multistage residual blocks 20 .

Each residual block 20 includes a time delay block 22 . The time delay block 22 includes a plurality of stacked time delay layers 24 (also referred to as "TD layers"). Each residual block 20 further combines a shortcut path 26 that bypasses the time-delay block 22 and combines the output of the time-delay block 22 with the output of the shortcut path 26 (also referred to as a "short-cut path"). and adder 29 .

The output layer 30 is connected to the outputs of the plurality of residual blocks 20 and includes a fully connected layer 32 and a mapping function 34 . The fully connected layer 32 is connected to the output node of the residual block 20 at the final stage, normalizes the probabilities of the output feature vectors, and outputs the most probable label. Since the label is output from the output layer 30 for each frame, the label is sequentially output corresponding to the input audio signal. FIG. 4 shows an example of using monophones (phones, characters, syllables, etc.) as labels. A series of labels (sequences of single sounds) that are sequentially estimated for each frame is the CTC output sequence. The estimation results of the basic CTC-based model 1 may include blanks (indicated by “φ” in FIG. 4) for which there is no corresponding label.

A mapping function 34 sequentially determines the corresponding text (subword sequence) from the CTC output sequence.

As described above, in the basic CTC-based model 1 according to the present embodiment, a text (subword sequence) is output as a recognition result for an input speech signal for each frame.

(b2: processing in a single time delay layer 24)
FIG. 5 is a schematic diagram showing the processing structure of the time delay layer 24 included in the basic CTC based model 1 according to this embodiment. Referring to FIG. 5, time delay layer 24 includes two delay elements 241, 242 that provide a delay of a given time step t _i to the input.

Each of the delay elements 241, 242 delays the input by a _timestep ti. The input sequence provided to time delay layer 24 is provided with a delay of time step t _i in delay element 241 . The resulting output with the delay of time step _ti in delay element 241 is further provided to delay element 242 . Delay element 242 provides an additional delay of timestep t _i to the result output from delay element 241 . Such two stages of delay elements generate three types of contexts whose timing differs by time steps _ti .

By using the input frame as the future context, the result output from the delay element 241 as the current context, and the result output from the delay element 242 as the past context, the time step can be expanded substantially in both directions.

As shown in FIG. 5, each of the time delay layers 24 returns the time step _t to the current frame, which is the frame corresponding to the input sequence (input vector), to the past. A past context (first internal vector) corresponding to the frame and a future context (second internal vector) corresponding to the future frame advanced by time step _ti are generated.

In the basic CTC-based model 1 according to the present embodiment, the entire input sequence Hl given to the ^l -th time delay layer 24 can be expressed as the following equation (1).

First, when any h ^l _t is input to the l-th time delay layer 24, using the standard weight matrix W ^l and the bias b ^l for the l-th time delay layer 24, the following (2) It is linearly transformed as shown in the formula.

Next, the deviation e _t ^l at time step t in the l-th time delay layer 24 can be expressed as the following equation (3).

Each output of the time delay layer 24 can be expressed as the following equation (4).

In the above equation (4), an example of using a rectified linear unit (ReLU) as the activation function of the residual block 20 is shown, but not limited to this, any activation function can be used. be able to. In the following description, the normalized linear function is also referred to as "ReLU".

(b3: Processing in residual block 20)
Looking at the entire residual block 20, which consists of a plurality of stacked time delay layers 24, the output is the combination (the result of the adder 29) of the output of the multi-layer transform f _i and the shortcut output bypassing the multi-layer transform f _i . It will be. A multi-layer transform f _i is a series combination function of a time delay layer 24 and an activation function (ReLU).

A network in which multiple residual blocks 20 are stacked on top of each other behaves like an ensemble network.

FIG. 6 is a schematic diagram showing an example of a network structure equivalent to the network structure shown in FIG. The network structure shown in FIG. 6A can be expressed as an equivalent network structure shown in FIG. As shown in FIG. 6(B), a plurality of paths (eight in FIG. 6) passing through different numbers of residual blocks 20 exist in parallel. As a result, the results given all the different timestep delays are finally combined.

In the basic CTC-based model 1 shown in FIGS. 4 and 6, all outputs from the final residual block 20 will include outputs from other residual blocks 20 over time. For example, assuming a basic CTC-based model 1 consisting of three layers of residual blocks 20, the output y ³ _t at a certain time step from the final residual block 20 can be expressed as the following equation (6). can.

[C. Improved network structure]
An improved network structure according to this embodiment will now be described. The improved network structure according to the present embodiment corresponds to the residual blocks 20 that make up the basic CTC-based model 1 shown in FIGS. 4 and 6 above, with improvements added. Hereinafter, it is also referred to as an "improved CTC-based model" in contrast to the "basic CTC-based model". Note that in a context in which the "basic CTC-based model" and the "improved CTC-based model" are not distinguished, they may simply be collectively referred to as the "CTC-based model."

FIG. 7 is a schematic diagram showing the main part of the network structure of the improved CTC-based model according to this embodiment. FIG. 7A shows an example structure of the residual block 20 of the basic CTC-based model according to this embodiment, and FIG. 7B shows the structure of the residual block 20A of the improved CTC-based model according to this embodiment. Give an example.

Comparing FIGS. 7A and 7B, residual block 20A further includes an attention module 28 compared to residual block 20. FIG. The attention module 28 is arranged after the output layer of the residual block 20A. Attention module 28 adjusts the weights for the two paths included in residual block 20A (the path on shortcut path 26 side and the path on time delay block 22 side). By adopting such an attention module 28, the basic CTC-based model 1 can be made to behave more dynamically, so that learning performance and identification performance can be improved.

In the present embodiment, the attention module 28 is arranged after the residual block (time delay layer 24) that implements the time delay, thereby producing remarkable effects as will be described later.

In the following description, an attention score α _t ⁱ (vector quantity) for changing the weight of each route as shown in the following equation (7) is used.

The attention score α _t ⁱ (vector quantity) includes weight α _t ⁱ and weight β _t ⁱ (=1−α _t ⁱ ) as elements. The weight α _t ⁱ means the scale factor for the data transmitted through the shortcut path 26 of the i-th residual block 20A at any time step t, and the weight β _t ⁱ (=1−α _t ⁱ ) is We denote the scale factor for the data transmitted through the time delay block 22 of the i-th residual block 20A at any time step t.

More specifically, as shown in FIG. 7B, the attention module 28 includes a fully connected layer 282, a softmax function 284, and multipliers 286,288.

A fully connected layer 282 of attention module 28 is connected to the output of corresponding residual block 20A and shortcut path 26 . A softmax function 284 is connected to the fully connected layer 282 .

Output path 285 from time delay block 22 is input to multiplier 286 , multiplied by weight β _t ⁱ in multiplier 286 , and output to adder 29 . On the other hand, the shortcut path 26 is input to the multiplier 288 , multiplied by the weight α _t ⁱ in the multiplier 288 , and output to the adder 29 . Note that α _t ⁱ +β _t ⁱ =1. ^Thus , the ^attention _module 28 _assigns an A weight α _t ⁱ (first weight) and a weight β _t ⁱ (second weight) for the shortcut route 26 are calculated.

The weights α _t ⁱ and the weights β _t ⁱ are dynamically changed so that the ratio between the output of the multi-layered transform f _i and the shortcut output bypassing the multi-layered transform f _i included in the output from the residual block 20A is Can be adjusted dynamically.

Thus, attention module 28 adjusts the weights between output path 285 and shortcut path 26 through multiple time delay layers 24 .

The output from the residual block 20A as shown in FIG. 7 can be defined as the following equation (8) by changing the weighting for the relational expression shown in the above equation (5).

In this way, attention module 28 is able to combine the resulting output obtained from the input provided to corresponding residual block 20A through corresponding multiple time delay layers 24 and the input provided to corresponding residual block 20A. Update the weights α _t ⁱ and the weights β _t ⁱ (scale factors) at each timestep based on .

More specifically, the weight α _t ⁱ and the weight β _t ⁱ are calculated using the fully connected layer 282 and the softmax function 284 according to the following equation (9).

[D. Hardware configuration]
Next, an example of a hardware configuration for realizing speech recognition system S using a trained model according to this embodiment will be described.

FIG. 8 is a schematic diagram showing an example of a hardware configuration for realizing speech recognition system S according to this embodiment. The speech recognition system S is typically implemented using an information processing device 500, which is an example of a computer.

Referring to FIG. 8, an information processing apparatus 500 realizing speech recognition system S includes, as main hardware components, a CPU (central processing unit) 502, a GPU (graphics processing unit) 504, a main memory 506, It includes a display 508 , a network interface (I/F) 510 , a secondary storage device 512 , an input device 522 and an optical drive 524 . These components are connected to each other via an internal bus 528 .

CPU 502 and/or GPU 504 is a processor that executes various programs to be described later to perform processing necessary for realizing speech recognition system S according to the present embodiment. A plurality of CPUs 502 and GPUs 504 may be arranged, or may have a plurality of cores.

The main memory 506 is a storage area that temporarily stores (or caches) program codes and work data when the processor (CPU 502 and/or GPU 504) executes processing. ) and SRAM (static random access memory).

A display 508 is a display unit that outputs a user interface related to processing, processing results, and the like, and is configured by, for example, an LCD (liquid crystal display) or an organic EL (electroluminescence) display.

A network interface 510 exchanges data with any information processing device on the Internet or an intranet. As the network interface 510, for example, any communication method such as Ethernet (registered trademark), wireless LAN (local area network), Bluetooth (registered trademark), or the like can be adopted.

The input device 522 is a device that receives instructions, operations, and the like from the user, and includes, for example, a keyboard, mouse, touch panel, pen, and the like. In addition, the input device 522 may include a sound collecting device for collecting audio signals necessary for learning and decoding, and an interface for accepting input of audio signals collected by the sound collecting device. You can stay.

The optical drive 524 reads information stored in an optical disc 526 such as a CD-ROM (compact disc read only memory) or DVD (digital versatile disc) and outputs it to other components via an internal bus 528 . The optical disc 526 is an example of a non-transitory recording medium, and is distributed in a state in which any program is stored in a non-volatile manner. The optical drive 524 reads the program from the optical disk 526 and installs it in the secondary storage device 512 or the like, so that the computer functions as the information processing device 500 . Therefore, the subject of the present invention can also be a program itself installed in secondary storage device 512 or the like, or a recording medium such as optical disc 526 storing a program for realizing the functions and processes according to the present embodiment. .

FIG. 8 shows an optical recording medium such as an optical disk 526 as an example of a non-transitory recording medium, but is not limited to this, a semiconductor recording medium such as a flash memory, a magnetic recording medium such as a hard disk or a storage tape. , and MO (magneto-optical disk) may be used.

Secondary storage device 512 stores programs and data necessary for the computer to function as information processing device 500 . For example, it is composed of a non-volatile storage device such as a hard disk or an SSD (solid state drive).

More specifically, the secondary storage device 512 contains an OS (operating system) (not shown), a training program 514 for realizing learning processing, model definition data 516 for defining a network structure to be learned, It stores network parameters 518 for defining the finished model and a training data set 520 .

Training program 514 is executed by the processor (CPU 502 and/or GPU 504 ) to implement a learning process for determining network parameters 518 . The model definition data 516 includes information for defining components, connection relationships, and the like that constitute the network structure of the basic CTC-based model 1 and improved CTC-based model 1A to be learned. The network parameters 518 include parameters for each element that constitutes the model (network) to be learned. The value of each parameter contained in network parameters 518 is optimized by running training program 514 . The training data set 520 can use, for example, a data set included in CSJ as described later. For example, if the basic CTC-based model 1 and the improved CTC-based model 1A to be trained are directed to a speech recognition task, then the training data set 520 includes speech signals such as lectures and corresponding speech signals. Transcription text indicating the contents of the utterance.

Some of the libraries and functional modules required when the processor (CPU 502 and/or GPU 504) executes programs may be replaced with libraries or functional modules provided as standard by the OS. In this case, the program alone does not include all of the program modules necessary to implement the corresponding functions, but the intended processing can be achieved by installing it under the execution environment of the OS. Even a program that does not include some of such libraries or functional modules can be included in the technical scope of the present invention.

Moreover, these programs may be distributed by being stored in any recording medium as described above and not only being distributed, but also being downloaded from a server device or the like via the Internet or an intranet.

FIG. 8 shows an example in which the information processing apparatus 500 is configured using a single computer. , the information processing device 500 and a speech recognition system S including the information processing device 500 may be realized.

All or part of the functions realized by the processor (CPU 502 and/or GPU 504) executing the program may be realized using a hard-wired circuit such as an integrated circuit. For example, it may be implemented using an ASIC (application specific integrated circuit), an FPGA (field-programmable gate array), or the like.

A person skilled in the art will be able to realize information processing apparatus 500 according to the present embodiment by appropriately using technology suitable for the era in which the present invention is implemented.

For convenience of explanation, the same information processing apparatus 500 is used to perform learning (construction of a CTC-based model) and decoding (speech recognition using a model including the CTC-based model). It may be implemented using different hardware.

[E. Learning method]
Next, a learning method for improved CTC-based model 1A according to the present embodiment will be described.

(e1: Overview)
The CTC-based model according to this embodiment provides an E2E framework and does not require separate training of acoustic and language models. That is, the CTC-based model directly outputs the text corresponding to the input speech signal, and uses a training data set consisting of the speech signal and the corresponding text in the learning process.

The learning process of the CTC-based model according to the present embodiment can use supervised learning in the same way as the general learning process of neural networks. Specifically, arbitrary initial values are set for the parameters of each component that constitutes the CTC-based model. Then, the speech signal (acoustic feature sequence) included in the training data set is sequentially input to the CTC-based model, and the CTC output sequence (text) sequentially output from the CTC-based model and the input speech signal are matched. The error with the text is calculated, and the parameter of each component constituting the CTC-based model is successively updated based on the calculated error.

Through such learning processing, a trained model corresponding to the CTC-based model can be constructed from the training data.

In this embodiment, the speech recognition performance can be improved by appropriately learning the improved CTC-based model 1A. By inputting the feature vectors into the improved CTC-based model 1A, which includes the attention module 28 as described above, the scale factors (α _t ¹ , α _t ² , . . . α _t ⁱ , . . . α _t ^N ).

According to research by the inventors, in each residual block 20A, the weight of the path through which data passes through the time delay block 22 and the weight of the path through which data passes through the shortcut path 26 are determined by the applied system varies depending on

Therefore, in this embodiment, a learning method such as a retrain-based method, a prune-based method, or a network reconstruction method can be employed as described below.

(e2: retraining method)
The retraining method is a method of retraining all parameters (including the attention module 28 parameters) that define the improved CTC-based model 1A, which is an ultra-deep convolutional network. More specifically, a learned model is acquired by training the basic CTC-based model 1, and an attention module 28 is added to the acquired learned model to configure an improved CTC-based model 1A. and run the training again.

Since both the basic CTC-based model 1 and the improved CTC-based model 1A have to be trained, the time required for training is approximately doubled, but the speech recognition performance can be definitely improved.

Note that the scale factors (α _t ¹ , α _t ² , . . . α _t ⁱ , . . . α _t ^N ) change at each time step t.

FIG. 9 is a flow chart showing a processing procedure of a learning method (retraining method) for improved CTC-based model 1A according to the present embodiment. Each step shown in FIG. 9 is typically implemented by the processor (CPU 502 and/or GPU 504 ) of information processing apparatus 500 executing training program 514 .

Referring to FIG. 9, information processing apparatus 500 is supplied with training data set 40 including speech signal 42 and corresponding text 44 (step S100). The information processing apparatus 500 randomly determines initial values of parameters that define the basic CTC based model 1 (step S102).

The information processing device 500 generates a feature vector for each frame from the speech signal 42 included in the training data set 40 (step S104). Then, the information processing apparatus 500 inputs the generated feature vector to the basic CTC-based model 1 to calculate an estimation result (step S106).

The information processing device 500 determines whether or not the calculated estimation result has reached a predetermined number (step S108). If the calculated estimation result has not reached the predetermined number (NO in step S108), the processing from step S104 onward is repeated.

If the calculated estimation results have reached a predetermined number (YES in step S108), information processing apparatus 500 generates a series of calculated estimation results (output sequence) and corresponding text 44 (label sequence). Based on the error between and, it is determined whether or not the convergence condition of the learning process is satisfied (step S110).

If the convergence condition of the learning process is not satisfied (NO in step S110), the information processing device 500 compares a series of estimation results (output sequence) collectively calculated as a mini-batch and the corresponding text 44 (label sequence). Based on the error between the two, the values of the parameters defining the basic CTC based model 1 are updated (step S112), and the processing from step S104 onward is repeated.

On the other hand, if the convergence condition of the learning process is satisfied (YES in step S110), the current parameters are output as the learning result (step S114). That is, the basic CTC-based model 1 defined by the current parameters is output as a trained model.

In the steps S100 to S114 described above, the information processing device 500 performs a first training step of determining the parameters defining the network of the basic CTC-based model 1 (classifier) using the training data set 40 .

Subsequently, the information processing apparatus 500 adds the attention module 28 to the learned basic CTC-based model 1 to generate the improved CTC-based model 1A (step S116). That is, the information processing apparatus 500 performs an additional step of adding an attention module 28 that adjusts the weight between the route passing through the multiple time delay layers 24 and the shortcut route 26 to the basic CTC-based model 1 (discriminator). Run.

The information processing apparatus 500 randomly determines initial values of the parameters of the attention module 28 added to the improved CTC based model 1A (step S118). Then start training again.

Specifically, the information processing device 500 generates a feature vector for each frame from the speech signal 42 included in the training data set 40 (step S120). The information processing apparatus 500 then inputs the generated feature vector to the improved CTC-based model 1A to calculate an estimation result (step S122).

The information processing device 500 determines whether or not the calculated estimation result has reached a predetermined number (step S124). If the calculated estimation result has not reached the predetermined number (NO in step S124), the processing from step S120 onward is repeated.

If the calculated estimation results have reached a predetermined number (YES in step S124), information processing apparatus 500 generates a series of calculated estimation results (output sequence) and corresponding text 44 (label sequence). Based on the error between and, it is determined whether or not the convergence condition of the learning process is satisfied (step S126).

If the convergence condition of the learning process is not satisfied (NO in step S126), the information processing device 500 compares a series of estimation results (output sequence) collectively calculated as a mini-batch and the corresponding text 44 (label sequence). Based on the error between them, the values of the parameters that define the improved CTC-based model 1A are updated (step S128), and the processes from step S120 onward are repeated.

On the other hand, if the convergence condition of the learning process is satisfied (YES in step S126), the current parameters are output as the learning result (step S130). That is, the improved CTC-based model 1A defined by the current parameters is output as a trained model. Then the process ends.

In steps S118-S130 described above, the information processor 500 performs a second training step of determining the parameters defining the attention module 28 using the training data set 40. FIG. In this second training step, the information processing device 500 will again determine the values of all the parameters defining the network of improved CTC-based models 1A (discriminators), including the parameters defining the attention module 28. .

By executing the process of determining all parameters twice in this manner, high identification accuracy can be achieved.

(e3: clipping method)
In the above-described retraining method, all parameters that define the improved CTC-based model 1A (including the parameters of the attention module 28) were updated, but in the clipping method, the parameters of the generated trained model were fixed. Above, only the attention module 28 parameters may be trained with less training data.

By training only the parameters of the attention module 28, the time required for the learning process can be shortened.

FIG. 10 is a flow chart showing a processing procedure of a learning method (clipping method) for improved CTC-based model 1A according to the present embodiment. Each step shown in FIG. 10 is typically implemented by the processor (CPU 502 and/or GPU 504 ) of information processing apparatus 500 executing training program 514 . Among the processes shown in FIG. 10, the same steps as those shown in FIG. 9 are given the same step numbers.

Referring to FIG. 10, information processing apparatus 500 is supplied with training data set 40 including speech signal 42 and corresponding text 44 (step S100). The information processing apparatus 500 randomly determines initial values of parameters that define the basic CTC based model 1 (step S102).

The information processing device 500 generates a feature vector for each frame from the speech signal 42 included in the training data set 40 (step S104). Then, the information processing apparatus 500 inputs the generated feature vector to the basic CTC-based model 1 to calculate an estimation result (step S106).

The information processing device 500 determines whether or not the calculated estimation result has reached a predetermined number (step S108). If the calculated estimation result has not reached the predetermined number (NO in step S108), the processing from step S104 onward is repeated.

If the calculated estimation results have reached a predetermined number (YES in step S108), information processing apparatus 500 generates a series of calculated estimation results (output sequence) and corresponding text 44 (label sequence). Based on the error between and, it is determined whether or not the convergence condition of the learning process is satisfied (step S110).

If the convergence condition of the learning process is not satisfied (NO in step S110), the information processing device 500 determines the error between the calculated series of estimation results (output sequence) and the corresponding text 44 (label sequence). Based on this, the values of the parameters that define the basic CTC-based model 1 are updated (step S112), and the processing from step S104 onward is repeated.

On the other hand, if the convergence condition of the learning process is satisfied (YES in step S110), the current parameters are output as the learning result (step S114). That is, the basic CTC-based model 1 defined by the current parameters is output as a trained model.

In the steps S100 to S114 described above, the information processing device 500 performs a first training step of determining the parameters defining the network of the basic CTC-based model 1 (classifier) using the training data set 40 .

Subsequently, the information processing apparatus 500 adds the attention module 28 to the learned basic CTC-based model 1 to generate the improved CTC-based model 1A (step S116). That is, the information processing apparatus 500 performs an additional step of adding an attention module 28 that adjusts the weight between the route passing through the multiple time delay layers 24 and the shortcut route 26 to the basic CTC-based model 1 (discriminator). Run.

The information processing apparatus 500 randomly determines initial values of the parameters of the attention module 28 added to the improved CTC based model 1A (step S118). Then, training for the attention module 28 is started.

Specifically, the information processing apparatus 500 acquires a reduced training data set 40S that is part of the training data set 40 used for training the basic CTC-based model 1 (step S119). The information processing device 500 then generates a feature vector for each frame from the speech signal 42 included in the reduced training data set 40S (step S121). The information processing apparatus 500 then inputs the generated feature vector to the improved CTC-based model 1A to calculate an estimation result (step S122).

The information processing device 500 determines whether or not the calculated estimation result has reached a predetermined number (step S124). If the calculated estimation result has not reached the predetermined number (NO in step S124), the processing from step S120 onward is repeated.

If the calculated estimation results have reached a predetermined number (YES in step S124), information processing apparatus 500 generates a series of calculated estimation results (output sequence) and corresponding text 44 (label sequence). Based on the error between and, it is determined whether or not the convergence condition of the learning process is satisfied (step S126).

If the convergence condition of the learning process is not satisfied (NO in step S126), the information processing device 500 determines the error between the calculated series of estimation results (output sequence) and the corresponding text 44 (label sequence). Based on this, the values of the parameters defining the attention module 28 included in the improved CTC based model 1A are updated (step S129), and the processing from step S120 onward is repeated.

On the other hand, if the convergence condition of the learning process is satisfied (YES in step S126), the current parameters are output as the learning result (step S130). That is, the improved CTC-based model 1A defined by the current parameters is output as a trained model. Then the process ends.

In steps S118-S130 described above, the information processor 500 performs a second training step of determining the parameters defining the attention module 28 using the training data set 40. FIG. In this second training step, the information processing apparatus 500 fixes only the parameters defining the attention module 28 while fixing the parameters determined in the first training step (parameters defining the basic CTC-based model 1). Execute the decision process.

By fixing the parameters defining the basic CTC-based model 1 in this way, the second training can be achieved in a short time.

(e4: network reconfiguration method)
The average value of the attention score or scale factor (weight) of each residual block 20A indicated by the attention module 28 of the improved CTC-based model 1A according to this embodiment indicates information from all paths capable of transmitting data. Become.

FIG. 11 is a diagram showing an example distribution of data transfer in improved CTC-based model 1A according to the present embodiment. In the example shown in FIG. 11, in the first residual block 20A, data mainly passes through the shortcut path 26, and in the second and subsequent residual blocks 20A, data mainly passes through the time delay block 22. I know you are passing.

The network structure of the improved CTC-based model 1A can be improved by using the state of data transmission in the improved CTC-based model 1A as prior knowledge. Improvements in the network structure can also improve speech recognition performance.

For example, the network structure itself can be tuned by binarizing (“0” or “1”) the weight α _t ⁱ for the i-th residual block 20A. That is, if the weight α _t ⁱ is "1", no data is transmitted to the time delay block 22 of the corresponding residual block 20A, so it can be determined that the time delay block 22 may be deleted. On the other hand, if the weight α _t ⁱ is "0", it can be determined that the shortcut route 26 may be deleted.

As a method for binarizing such weights α _t ⁱ , the weights α _t ⁱ calculated for each time step are applied to feature vectors sequentially generated using a data set consisting of a part of the training data set. can be determined based on whether a representative value (average, maximum, minimum, median, etc.) of exceeds a predetermined threshold. The data set used for binarizing the weights α _t ⁱ can be, for example, the speech portion of the development data set (CSJ-Dev) described later.

FIG. 12 is a diagram for explaining a processing procedure of a learning method (network reconstruction method) for improved CTC-based model 1A according to the present embodiment. First, as shown in FIG. 12(A), the basic CTC-based model 1 is trained using a regular training data set. Subsequently, as shown in FIG. 12(B), an attention module 28 is added to the basic CTC-based model 1, and the attention module 28 is trained using a reduced training data set consisting of a portion of the normal training.

After training the parameters corresponding to the basic CTC-based model 1 and the parameters corresponding to the attention module 28, the improved CTC-based model 1A is input with feature vectors generated from speech parts such as the development data set, and each residual Calculate the change in scale factor over time in the difference block 20A. In each residual block 20A, a temporal change as shown in FIG. 13 can be calculated.

FIG. 13 is a diagram showing an example of temporal changes in scale factors calculated using improved CTC-based model 1A according to the present embodiment. The value of the weight α _t ¹ , which is the scale factor shown in FIG. 13, varies greatly for each input syllable.

Based on the temporal change in the scale factor calculated for each residual block 20A, the data transfer state in each residual block 20A is evaluated. This data transmission state can also be considered to correspond to the stability in each residual block 20A. Residual blocks 20A whose corresponding scale factor changes over time satisfy a predetermined condition are deleted from the improved CTC-based model 1A, as shown in FIG. 12(C).

Finally, all the parameters (including the attention module 28 parameters) that define the improved CTC-based model 1A after some time delay blocks 22 have been removed, depending on the situation, are determined by retraining.

Thus, by giving an input signal to the improved CTC-based model 1A (discriminator) to which the attention module 28 is added, based on the change in the value of the weight α _t ¹ which is the scale factor calculated by the attention module 28, , a process of deleting some of the plurality of time delay layers 24 may be performed.

Here, the conditions for determining whether or not each of the time delay blocks 22 should be deleted are when the absolute value of the weight α _t ⁱ , which is the scale factor for the shortcut path 26, is relatively large, or when the variation in the value is relatively large. For example, when it is relatively large. That is, when the amount of data passing through the shortcut path 26 is relatively large with respect to the target audio signal, or when the variation in the amount of data passing through the shortcut path 26 is relatively large, the residual block 20A It means that the stability is low, and learning and decoding can be made more stable by deleting such a low-stability residual block 20A.

Specific conditions for whether or not to delete the time delay block 22 include the following.

(1) For a specific speech input, the number (or appearing peak) of which the value of the weight α _t ⁱ (scale factor) exceeds a predetermined threshold (typically “0.5”) is greater than or equal to a predetermined number.

(2) A ^{predetermined} _threshold (typically, " 0.5”) is greater than or equal to a predetermined number (eg, 30%).

(3) When the area of the graph indicated by the temporal change of the weight α _t ⁱ (scale factor) is equal to or greater than a predetermined threshold.

(4) For a specific voice input, when the variation width (standard deviation, variance, difference between the maximum value and the minimum value) of the weight α _t ⁱ (scale factor) over time is greater than or equal to a predetermined threshold .

Any criteria other than those mentioned above can be used.
After optimizing the network structure according to the procedure described above, the learning process is executed.

FIG. 14 is a flow chart showing a processing procedure of a learning method (network reconstruction method) for improved CTC-based model 1A according to the present embodiment. Each step shown in FIG. 14 is typically implemented by the processor (CPU 502 and/or GPU 504 ) of information processing apparatus 500 executing training program 514 .

Referring to FIG. 14, information processing apparatus 500 determines parameters of basic CTC-based model 1 using training data set 40 (step S150). The processing of step S150 is substantially the same as steps S100-S114 of the retraining method shown in FIG.

Subsequently, the information processing apparatus 500 adds the attention module 28 to the trained basic CTC-based model 1 to generate the improved CTC-based model 1A (step S152). The information processing apparatus 500 then determines the parameters of the attention module 28 added to the improved CTC based model 1A (step S154). The processing of this step S154 is substantially the same as the processing of steps S118 to S130 of the clipping method shown in FIG.

Subsequently, the information processing device 500 inputs the feature vector generated from the speech portion of the development data set to the improved CTC-based model 1A, and calculates the temporal change of the scale factor in each residual block 20A (step S156). ). Then, the information processing device 500 determines whether or not there is a time delay block 22 included in the improved CTC-based model 1A that should be deleted, based on the temporal change in the scale factor in each residual block 20A. (Step S158). If there is a time delay block 22 to be deleted (YES in step S158), information processing apparatus 500 deletes the target time delay block 22 from improved CTC based model 1A (step S160). If there is no time delay block 22 to be deleted (NO in step S158), the process of step S160 is skipped.

Finally, the information processing apparatus 500 re-determines all parameters of the improved CTC-based model 1A (after the time delay block 22 has been removed depending on the situation) (step S162). The processing of step S160 is substantially the same as steps S120-S130 of the retraining method shown in FIG.

A learned model of the improved CTC-based model 1A is generated by the above procedure.

[F. Decoding method]
Next, a decoding method using the improved CTC-based model 1A according to this embodiment will be explained. Since the improved CTC-based model 1A according to the present embodiment is an E2E framework, only by inputting feature vectors that are sequentially generated from speech signals, corresponding texts (subword sequences) are sequentially output.

FIG. 15 is a flow chart showing the processing procedure of the decoding method for improved CTC-based model 1A according to this embodiment. Each step shown in FIG. 15 is typically implemented by the processor (CPU 502 and/or GPU 504 ) of information processing apparatus 500 executing training program 514 .

Referring to FIG. 15, information processing apparatus 500 generates a feature vector for each frame from an input audio signal (step S200). The information processing apparatus 500 then inputs the generated feature vector to the improved CTC-based model 1A to calculate and output an estimation result (step S202).

The information processing device 500 then determines whether or not the input of the audio signal continues (step S204). If the input of the audio signal continues (YES in step S204), the processing from step S200 onwards is repeated.

On the other hand, if the input of the audio signal has not continued (NO in step S204), the decoding process ends.

[G. Evaluation experiment]
The inventors of the present application conducted evaluation experiments on the performance of the improved CTC-based model 1A according to the present embodiment described above. Evaluation experiments will be described below.

(g1: description of data and tasks)
In the evaluation experiment, the "Corpus of Spontaneous Japanese (CSJ)" provided by the National Institute for Japanese Language and Linguistics was used as training data and evaluation data.

In accordance with findings from previous studies, 240 hours of speech speech included in CSJ was constructed as a training data set (hereinafter also referred to as “CSJ-Train”). The CSJ includes three official evaluation datasets (CSJ-Eval01, CSJ-Eval02, CSJ-Eval03). Each evaluation data set contains speech for 10 lectures. These evaluation datasets were used to evaluate speech recognition results. In addition, a development dataset (CSJ-Dev) consisting of 10 speeches was used for evaluation during training.

Furthermore, a 27.6-hour data set included in CSJ (hereinafter also referred to as “CSJ-Train _small ”) was selected for seed model training for warm-up initialization and parameter tuning.

The number and duration of talks included in these datasets are shown in Table 1 below.

(g2: baseline model)
First, CSJ-Train was used to train a baseline model as an evaluation criterion. As a first baseline model, we take the DNN-HMM-CE (deep neural network and hidden Markov model cross entropy) model. In constructing the DNN-HMM-CE model, first, the GMM-HMM (Gaussian mixture model and hidden Markov model) model corresponding to the acoustic model is trained, and then five hidden layers (each layer has 2048 hidden We trained a DNN model (which corresponds to a language model) consisting of The output layer has approximately 8500 nodes, which correspond to the combined triphone states of the GMM-HMM model. In these trainings, 72-dimensional filterbank features (24-dimensional static +Δ+ΔΔ) were used. The filterbank features are the result of averaging and normalizing for each speaker and consist of 11 frames partitioned (5 past, 5 present, 5 future). The DNN model was trained using standard stochastic gradient descent (SGD) based on the cross entropy loss criterion.

For decoding, a 4-gram word language model (WLM) was trained with 591 hours of text transcripts from the CSJ training dataset. The WLM vocabulary size is 98×10 ³ .

(g3: setting for training of improved CTC-based model 1A)
The improved CTC-based model 1A according to the present invention was trained with 72-dimensional filterbank features (24-dimensional static +Δ+ΔΔ) (unsplit). In this training, we used 263 syllables of Japanese (which is the basic unit of Japanese written language), non-speech noise, speech noise, and blank (φ) as basic acoustic model units.

The target network (tuned using a phone-based seed system trained by CSJ-Train _small ) is defined as follows. That is, from the nine fully connected layers following the input layer, the following fifteen time delay layers 24 (total of the three residual blocks 20A), and the two fully connected layers placed before the softmax output, Become.

Table 2 below shows the change in window size in each of the three stacked residual blocks 20A.

A 27.6-hour dataset included in the CSJ (CSJ-Train _small ) was used to train a seed model according to the cross-entropy loss criterion, and the resulting model parameters were used to initialize the CTC model. The FsAdaGrad algorithm was used for CTC training. Block-wise model update filtering (BMUF) was applied to speed up training with a training dataset containing 240 hours of speech speech (CSJ-Train). The initial value of the learning rate for each frame was set to 0.00001, and the learning rate was automatically adjusted according to the test results for CSJ-Dev. The mini-batch size was set to 2048, and the number of sequences processed in parallel in the same mini-batch was set to 16. The maximum number of epochs was set to 25.

The modified CTC-based model 1A is decoded by feeding the network-computed scaled log-likelihoods to the EESEN decoder.

A cross-entropy model (VResTD-CE), which has the same structure as the improved CTC-based model 1A according to the present embodiment and whose features are set by Microsoft's Computational Network Toolkit (CNTK), was also trained. In this training, we used the same labels as the DNN-HMM-CE model.

(g4: tuning of improved CTC-based model 1A by addition of attention module)
As described above, the improved CTC-based model 1A is constructed by adding the attention module 28 to the basic CTC-based model 1 (VResTD-CTC). A trained model obtained by training all parameters defining the improved CTC-based model 1A (including the parameters of the attention module 28) using CSJ-Train is referred to as " _VResTDM -CTC retain".

The initial learning rate used to obtain _VResTDM -CTC retract was 0.00001. The minibatch size was 2048. Performance was evaluated using CSJ-Dev after each epoch of training was completed. As a result, training was terminated just before the start of the 17th epoch, just before performance degraded.

FIG. 16 is a diagram showing an example of change in attention score of improved CTC-based model 1A according to the present embodiment. 16(A) and (B) show changes in attention score in the leading residual block 20A for the input speech frame, and FIGS. 16(C) and (D) show the final Figure 3 shows the change in attention score in residual block 20A. CSJ-Eval01 was used as an input speech frame.

The two different systems (syllable system and ci-phone system) behave differently when passing through the leading residual block 20A. Specifically, as shown in FIG. 16(A), on a syllable basis, speech segments tend to follow shortcut paths. On the other hand, as shown in FIG. 16(B), such a tendency is not observed in single-tone bass.

As evaluations, on a syllable basis, CSJ-Eval01 had an average attention score for speech segments of 0.6 and an average attention score for blanks of 0.36. On the other hand, on a single note basis, the mean attention scores are all well below those values.

In the final residual block 20A, there is a strong tendency to avoid shortcut paths for both systems. Specifically, the average attention score for speech frames for CSJ-Eval01 was approximately 0.0 for both systems.

Based on these experimental results, an improved CTC-based model 1A was prepared by adjusting the weights for the residual block 20 included in the basic CTC-based model 1 (VResTD-CTC). More specifically, some time-delayed blocks 22 are deleted by binarizing the attention score α _t ⁱ for the speech segment with a threshold value of “0.5”. That is, the trained model generated by the pruning method described above is referred to as "VResTDM-CTC _prune ."

(g5: speech recognition performance)
Next, an example of evaluation results of speech recognition performance of improved CTC-based model 1A according to the present embodiment will be described. Three evaluation data sets (CSJ-Eval01, CSJ-Eval02, and CSJ-Eval03) included in CSJ were used to evaluate speech recognition performance. Speech recognition performance was evaluated by comparison with the baseline models described above (DNN-HMM-CE and VResTD-CE). The evaluation results of this speech recognition performance are shown in Table 3 below.

In the evaluation results described above, the word error rate (WER) of automatic speech recognition (ASR) was used as an evaluation index. WER indicates the error rate of the text output when speech is input to the evaluation target model with respect to the correct text corresponding to the input speech. A smaller WER value indicates higher performance.

According to the evaluation results described above, both _VResTDM -CTC _prune and VResTDM-CTC retract were superior to the baseline model (DNN-HMM-CE) and basic CTC-based model 1 (VResTD-CTC) in all evaluation datasets. A significant improvement is seen in comparison. In addition, _VResTDM -CTC retract exhibited performance equivalent to that of VResTD-CE in two evaluation data sets, and exhibited higher performance in the third evaluation data set.

[H. summary]
According to the improved CTC-based model 1A according to the present embodiment, the weight (first weight) for the route passing through the multiple time delay layers 24 and the weight (second weight) for the shortcut route 26 are set for each time step. can be updated to By updating the weights at each time step in this manner, the entire network can be made to behave dynamically, thereby realizing an appropriate network structure according to the target system.

Further, according to the improved CTC-based model 1A according to the present embodiment, by monitoring temporal changes in weights (scale factors) updated by the attention module 28, the unstable time delay layer 24 and the like can be identified. It is possible to realize high-precision and high-speed learning.

The embodiments disclosed this time should be considered as examples and not restrictive in all respects. The scope of the present invention is indicated by the scope of the claims rather than the description of the above-described embodiments, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 basic CTC based model, 1A improved CTC based model, 2 feature extractor, 4 recognition engine, 10, 32, 282 fully connected layer, 20, 20A residual block, 22 time delay block, 24 time delay layer, 26 shortcut path, 28 attention module, 29 adder, 30 output layer, 34 mapping function, 40,520 training dataset, 40S reduced training dataset, 42 audio signal, 44 text, 241, 242 delay element, 284 softmax function, 285 output path, 286, 288 multiplier, 500 information processing device, 502 CPU, 504 GPU, 506 main memory, 508 display, 510 network interface, 512 secondary storage device, 514 training program, 516 model definition data, 518 network parameter, 522 Input Device, 524 Optical Drive, 526 Optical Disc, 528 Internal Bus, S Speech Recognition System.

Claims (6)

A discriminator that outputs a sequence of labels for an input signal,
an input layer that sequentially generates a first feature vector from the input signal for each frame of a predetermined time width;
a plurality of stacked residual blocks following the input layer;
an output layer connected to the output side of the plurality of residual blocks;
each of the plurality of residual blocks comprising:
a plurality of stacked time delay layers;
a shortcut path bypassing the plurality of time delay layers;
an attention module that adjusts weights between paths passing through the plurality of time delay layers and the shortcut paths;
The plurality of time delay layers have delay elements that provide a delay of a predetermined timestep with respect to the input,
Each of the time delay layers includes, for an input vector, a first internal vector corresponding to a past frame obtained by moving back the current frame, which is the frame corresponding to the input vector, by the time step; generating a second internal vector corresponding to a future frame advanced in time by the time step;
The attention module performs the time step based on a result output obtained by passing an input provided to a corresponding residual block through a corresponding plurality of time delay layers and an input provided to the corresponding residual block. A discriminator that updates the weights each time. The attention module includes:
a fully connected layer connected to the output of the corresponding residual block and the shortcut path;
2. The discriminator of claim 1, comprising a softmax function connected to said fully connected layer. A discriminator that outputs a sequence of labels for an input signal,
an input layer that sequentially generates a first feature vector from the input signal for each frame of a predetermined time width;
a plurality of stacked residual blocks following the input layer;
an output layer connected to the output side of the plurality of residual blocks;
each of the plurality of residual blocks comprising:
a plurality of stacked time delay layers;
a shortcut path bypassing the plurality of time delay layers;
an attention module that adjusts weights between paths passing through the plurality of time delay layers and the shortcut paths;
The plurality of time delay layers have delay elements that provide a delay of a predetermined timestep with respect to the input,
The attention module performs the time step based on a result output obtained by passing an input provided to a corresponding residual block through a corresponding plurality of time delay layers and an input provided to the corresponding residual block. update the weights every
parameters defining the network of classifiers are determined by training with a training data set in the absence of the attention module;
A discriminator , wherein the parameters defining the attention module are determined by training with a training data set with the attention module in place . A trained model for causing a computer to output a sequence of labels for an input signal, said trained model comprising:
an input layer that sequentially generates a first feature vector from the input signal for each frame of a predetermined time width;
a plurality of stacked residual blocks following the input layer;
an output layer connected to the output side of the plurality of residual blocks;
each of the plurality of residual blocks comprising:
a plurality of stacked time delay layers;
a shortcut path bypassing the plurality of time delay layers;
an attention module that adjusts weights between paths passing through the plurality of time delay layers and the shortcut paths;
The plurality of time delay layers have delay elements that provide a delay of a predetermined timestep with respect to the input,
Each of the time delay layers includes, for an input vector, a first internal vector corresponding to a past frame obtained by moving back the current frame, which is the frame corresponding to the input vector, by the time step; generating a second internal vector corresponding to a future frame advanced in time by the time step;
The attention module performs the time step based on a result output obtained by passing an input provided to a corresponding residual block through a corresponding plurality of time delay layers and an input provided to the corresponding residual block. A trained model configured to update the weights every time. A learning method for a discriminator that outputs a sequence of labels for an input signal, comprising:
The discriminator is
an input layer that sequentially generates a first feature vector from the input signal for each frame of a predetermined time width;
a plurality of stacked residual blocks following the input layer;
an output layer connected to the output side of the plurality of residual blocks;
each of the plurality of residual blocks comprising:
a plurality of stacked time delay layers;
a shortcut path that bypasses the plurality of time delay layers;
The plurality of time delay layers have delay elements that provide a delay of a predetermined timestep with respect to the input,
The learning method includes:
a first training step of determining parameters defining the network of classifiers using a training data set;
adding to the discriminator an attention module that adjusts weights between paths passing through the plurality of time-delay layers and the shortcut paths, wherein the attention module gives to corresponding residual blocks: updating the weights at each of the timesteps based on the resulting output obtained from passing the input through the corresponding plurality of time delay layers and the input provided to the corresponding residual block. cage,
a second training step of determining parameters defining said attention module using a training data set. providing an input signal to a classifier to which the attention module is attached to remove a portion of the plurality of time delay layers based on changes in the weight values calculated by the attention module. A learning method according to claim 5.

Images