JP7385900B2 - Inference machine, inference program and learning method - Google Patents

Description

The present technology relates to an inference device, an inference program, and a learning method for realizing a speech recognition task.

In the field of speech recognition, an end-to-end model, which is a neural network that integrates an acoustic model, a language model, and a dictionary (lexicon), has been studied and proposed (non-patent literature). 1 and 2, etc.). Transformer-based automatic speech recognition (ASR) systems are attracting attention as an end-to-end model for speech recognition tasks (see Non-Patent Document 3, etc.). Using a Transformer-based end-to-end model can facilitate the construction and learning of ASR systems.

Non-Patent Documents 4 and 5 disclose research results of acoustic models in Transformer-based end-to-end speech recognition systems for Chinese.

Additionally, Non-Patent Documents 6 and 7 disclose a method for efficiently training a multilingual end-to-end speech recognition system using a single model. More specifically, at the beginning of each utterance, a specific word <Language Mark> indicating which language the utterance is in (for example, <English>, <Mandarin>, <Japanese>, <German>, etc.) Perform learning using the dataset with added . <Language Mark> is treated as a label.

A. Graves and N. Jaitly, "Towards End-to-End speech recognition with recurrent neural networks," in Proc. ICML, 2014. A. W. Chan, N. Jaitly, Q. Le, and O. Vinyals, "Listen, attend and spell: A neural network for large vocabulary conversational speech recognition," in Proc. IEEE-ICASSP, 2016. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in CoRR abs/1706.03762, 2017. S. Zhou, L. Dong, S. Xu, and B. Xu, "A comparison of modeling units in sequence-to-sequence speech recognition with the transformer on Mandarin Chinese," in CoRR abs/1805.06239, 2018. S. Zhou, L. Dong, S. Xu, and B. Xu, "Syllable-based sequence-to-sequence speech recognition with the transformer in mandarin Chinese," in Proc. INTERSPEECH, 2018. S. Zhou, S. Xu, and B. Xu, "Multilingual end-to-end speech recognition with a single transformer on low-resource languages," in CoRR abs/1806.05059, 2018. B. Li and et al., "Multi-dialect speech recognition with a dingle sequence-to-sequence model," in CoRR abs/1806.05059, 2018.

The methods disclosed in Non-Patent Documents 6 and 7 mentioned above perform learning at the character level, and when learning multiple languages at the same time (i.e., using a single model to learn multilingual speech). When a recognition system is constructed and used), there is a problem that the number of tokens becomes enormous and the parameter size becomes huge.

The purpose of this technology is to provide a technology for realizing a multilingual end-to-end speech recognition system using a model with a smaller parameter size.

According to one embodiment, a reasoner is provided that receives input of an audio signal uttered in any one of a plurality of languages and outputs corresponding text. The reasoner receives an input sequence representing the audio characteristics of the audio signal and outputs a representation at a level different from the character level representing characteristics of characters included in the corresponding text, and a predetermined model. and a reconstruction unit that reconstructs a corresponding text from the expression output from the learned model by referring to the correspondence between characters and the characteristics of the characters.

The expression output from the learned model may include information specifying the structure of each character included in the corresponding text.

The information specifying the structure of the character may include information specifying one or more character parts that constitute the corresponding character.

The information specifying the structure of the character may include information specifying the arrangement of the one or more character parts.

The correspondence relationship may define a correspondence relationship between one or more character parts and corresponding characters for each language.

The expression output from the learned model may include information specifying the pronunciation of each character included in the corresponding text.

The information specifying the pronunciation of the character may include information specifying the pronunciation of the corresponding character based on universal features expressing phonological structure.

The information specifying the pronunciation of the character may include information specifying the pronunciation for each character obtained by further decomposing the word included in the corresponding text.

The correspondence relationship may define a correspondence relationship between information specifying pronunciation and corresponding characters for each language.

According to another embodiment, an inference program for implementing the above inference device on a computer is provided.

According to yet another embodiment, a learning method is provided for learning a reasoner that receives an input of an audio signal uttered in any one of a plurality of languages and outputs a corresponding text. The learning method includes the steps of: preparing an audio signal and a corresponding text; generating an expression at a level different from the character level that represents the characteristics of characters included in the text; and representing the audio characteristics of the audio signal. The method includes the step of optimizing parameters defining the inference device based on an error between an inference result obtained by inputting an input sequence to the inference device and a corresponding expression.

According to yet another embodiment, a learning program for causing a computer to execute the above learning method is provided.

According to the present technology, a multilingual end-to-end speech recognition system can be realized using a model with a smaller parameter size.

FIG. 2 is a schematic diagram showing an example of a Transformer according to related technology of the present invention. 1 is a schematic diagram showing an example of a hardware configuration that implements a speech recognition system according to the present embodiment. FIG. 1 is a schematic diagram showing an overview of a speech recognition system according to a first embodiment. FIG. 3 is a diagram for explaining a method of decomposition into character parts in the speech recognition system according to the first embodiment. FIG. 3 is a diagram showing an example of a character-component correspondence table used in the character synthesis section of the speech recognition system according to the first embodiment. FIG. 3 is a schematic diagram for explaining learning processing of the speech recognition system according to the first embodiment. 3 is a flowchart showing the procedure of learning processing of the speech recognition system according to the first embodiment. 3 is a flowchart illustrating the procedure of inference processing of the speech recognition system according to the first embodiment. FIG. 2 is a schematic diagram showing an overview of a speech recognition system according to a second embodiment. FIG. 7 is a schematic diagram for explaining the contents of learning processing and inference processing in the speech recognition system according to the second embodiment. FIG. 7 is a diagram for explaining processing related to universal speech expression in the speech recognition system according to the second embodiment. FIG. 7 is a diagram showing an example of a speech feature correspondence table used in a character conversion section of a speech recognition system according to a second embodiment. 7 is a flowchart showing the procedure of learning processing of the speech recognition system according to the second embodiment. 7 is a flowchart showing the procedure of inference processing of the speech recognition system according to the second embodiment.

Embodiments of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the figures are designated by the same reference numerals, and the description thereof will not be repeated.

[A. overview]
Conventional models used for speech recognition tasks (typically DNN-HMM models) allow only one token to be used as a label for one frame of utterance. In contrast, end-to-end models such as Transformer allow a series of tokens to be associated with one frame of utterance, thereby demonstrating stronger expressive capabilities.

The speech recognition system according to this embodiment executes multilingual speech recognition tasks using an end-to-end model. The speech recognition system according to the present embodiment uses a different level of representation rather than the character level like existing speech recognition systems.

More specifically, the parameter size is reduced by using expressions that focus on similarities between languages. As an example of such similarities between languages, an example (first example) focusing on the structure of ideograms (typically kanji) in which each character represents a meaning, and An example (second embodiment) in which attention is paid to the structure of phonetic characters (or phonetic characters) in which characters represent phonemes or syllables will be exemplified. Note that the technical scope of the present invention is not limited to ideograms and phonetic characters, but includes a speech recognition system that utilizes any similarity between languages.

The first embodiment (ideograms) assumes a case where a single model is used for multiple languages that use similar ideograms (typically kanji), and kanji are A trained model is constructed by considering it as a combination of one or more character parts such as and "tsukuri".

The second embodiment (phonetic characters) assumes a case where a single model is used for multiple languages that use similar phonetic characters, and a character is defined by one or more tonal characteristics. (articulatory features) and construct a trained model.

By employing such a trained model, it is possible to realize a multilingual real-time speech recognition system while suppressing the scale of the model (parameter size). Furthermore, improvement in recognition performance can also be expected.

The details of the speech recognition system according to this embodiment will be explained below.
[B. Transformer]
Any end-to-end model may be used in the speech recognition system according to this embodiment. At present, examples include a model using Transformer, LSTM (Long short-term memory), and a model called BERT. In the following description, a Transformer-based end-to-end model is adopted as a typical example. However, if a new end-to-end model is developed as technology advances, it is obvious that the present invention can also be applied to such a new model.

A general Transformer will be explained below.
FIG. 1 is a schematic diagram showing an example of a Transformer 10 according to the related technology of the present invention. Referring to FIG. 1, Transformer 10 is a trained model and corresponds to a form of neutral network.

The Transformer 10 includes stacked N layers of encoder blocks 20 and M layers of decoder blocks 40. The stacked N layers of encoder blocks 20 are collectively referred to as an encoder 200. The stacked M layers of decoder blocks 40 are collectively referred to as a decoder 400.

Encoder 200 outputs an intermediate sequence from input sequence 2. Decoder 400 outputs output sequence 70 based on the intermediate sequence output from encoder 200 and the previously output output sequence.

The encoder 200 (that is, the first layer among the N layers of encoder blocks 20) receives an input token string generated by an input embedding layer 4, a positional embedding layer 6, and an adder 8. is input. The encoder 200 (that is, the final layer of the N layers of encoder blocks 20) outputs an intermediate sentence representation as a calculation result.

The input embedding layer 4 divides the input sequence 2, such as a sentence, into one or more tokens in predetermined units, and generates a vector of a predetermined dimension indicating the value of each divided token. The positional embedding layer 6 outputs a positional embedding, which is a value indicating in which position in the input sequence 2 each token exists. Adder 8 adds the position embedding from position embedding layer 6 to the sequence from input embedding layer 4 .

Each of the encoder blocks 20 includes an MHA (Multi-head Attention) layer 22, a feed forward layer 26, and addition/regularization (Add & Norm) layers 24 and 28.

The MHA layer 22 calculates Attention for the input token sequence (vector). The addition/regularization layer 24 adds the vector output from the MHA layer 22 to the input token sequence (vector), and then normalizes the resultant using an arbitrary method. The feedforward layer 26 shifts the position (that is, the input time) with respect to the input vector. The addition/regularization layer 28 adds the vector output from the feedforward layer 26 to the vector output from the addition/regularization layer 24, and then regularizes the resultant using an arbitrary method.

The decoder 400 (that is, the first layer among M layers of decoder blocks 40) has an output token string generated by an output embedding layer 14, a positional embedding layer 16, and an adder 18. is input. The decoder 400 (that is, the final layer of the M layers of decoder blocks 40) outputs an output sequence as a calculation result.

The output embedding layer 14 divides the already output sequence (outputs (shifted right)) 12 into one or more tokens in predetermined units, and , generates a vector of a predetermined dimension indicating the value of each divided token. The positional embedding layer 16 outputs a positional embedding, which is a value indicating in which position in the output sequence 12 each token exists. Adder 18 adds the position embedding from position embedding layer 16 to the token string from output embedding layer 14 .

Each of the decoder blocks 40 includes an MMHA (Masked Multi-head Attention) layer 42, an MHA (Multi-head Attention) layer 46, a feed forward (Feed Forward) layer 50, and an addition/regularization (Add & Norm) layer. 44, 48, and 52. That is, the decoder block 40 has a similar configuration to the encoder block 20, but differs in that it includes an MMHA layer 42 and an addition/regularization layer 44.

The MMHA layer 42 performs mask processing on vectors that cannot exist among the previously calculated vectors. The addition/regularization layer 44 adds the vector output from the MMHA layer 42 to the output token sequence (vector), and then regularizes the resultant using an arbitrary method.

The MHA layer 46 calculates Attention for the intermediate sentence representation output from the addition/regularization layer 28 of the encoder block 20 and the vector output from the addition/regularization layer 44 . The basic processing of the MHA layer 46 is similar to that of the MHA layer 22. The addition/regularization layer 48 adds the vector output from the MHA layer 46 to the vector output from the addition/regularization layer 44, and then regularizes the resultant using an arbitrary method. The feedforward layer 50 shifts the position (that is, the input time) with respect to the input vector. The addition/regularization layer 52 adds the vector output from the feedforward layer 50 to the vector output from the MHA layer 46, and then regularizes the resultant using an arbitrary method.

Transformer 10 includes a softmax layer 60 as an output layer. The softmax layer 60 inputs the vector output from the decoder 400 to a softmax function and outputs the result obtained as an output sequence 70.

[C. Hardware configuration]
Next, an example of the hardware configuration for realizing the speech recognition system according to this embodiment will be described.

FIG. 2 is a schematic diagram showing an example of the hardware configuration for realizing the speech recognition system according to the present embodiment. The speech recognition system is typically realized using an information processing device 500, which is an example of a computer.

Referring to FIG. 2, an information processing device 500 that implements a speech recognition system includes a CPU (central processing unit) 502, a GPU (graphics processing unit) 504, a main memory 506, and a display as main hardware components. 508, a network interface (I/F) 510, a secondary storage device 512, an input device 522, and an optical drive 524. These components are connected to each other via an internal bus 528.

CPU 502 and/or GPU 504 are processors that execute processing necessary to implement the speech recognition system according to this embodiment. A plurality of CPUs 502 and GPUs 504 may be arranged, or may have a plurality of cores.

The main memory 506 is a storage area that temporarily stores (or caches) program codes, work data, etc. when the processor (CPU 502 and/or GPU 504) executes processing. ) and volatile memory devices such as SRAM (static random access memory).

The display 508 is a display unit that outputs a user interface related to processing, processing results, and the like, and is configured with, for example, an LCD (liquid crystal display) or an organic EL (electroluminescence) display.

The network interface 510 exchanges data with any information processing device on the Internet or an intranet. As the network interface 510, any communication method such as Ethernet (registered trademark), wireless LAN (local area network), Bluetooth (registered trademark), etc. can be adopted.

The input device 522 is a device that accepts instructions and operations from the user, and includes, for example, a keyboard, a mouse, a touch panel, a pen, and the like. Further, the input device 522 may include a sound collection device for collecting audio signals necessary for learning and decoding, and may include an interface for receiving input of audio signals collected by the sound collection device. It's okay to stay.

The optical drive 524 reads information stored on an optical disc 526 such as a CD-ROM (compact disc read only memory) or a DVD (digital versatile disc), and outputs the information to other components via an internal bus 528. The optical disc 526 is an example of a non-transitory recording medium, and is distributed in a state in which an arbitrary program is stored in a non-volatile manner. The optical drive 524 reads the program from the optical disk 526 and installs it in the secondary storage device 512 or the like, so that the computer functions as the information processing device 500. Therefore, the subject matter of the present invention may be the program itself installed in the secondary storage device 512 or the like, or a recording medium such as the optical disk 526 that stores the program for realizing the functions and processing according to the present embodiment. .

Although FIG. 2 shows an optical recording medium such as an optical disk 526 as an example of a non-transitory recording medium, the present invention is not limited to this, and the present invention is not limited to semiconductor recording media such as flash memory, magnetic recording media such as hard disks, or storage tapes. , a magneto-optical recording medium such as MO (magneto-optical disk) may be used.

The secondary storage device 512 stores programs and data necessary for the computer to function as the information processing device 500. For example, it is configured with a non-volatile storage device such as a hard disk or SSD (solid state drive).

More specifically, the secondary storage device 512 stores, in addition to an OS (operating system) not shown, a learning program 514 for realizing learning processing, and model definition data that defines the structure of a model used in the speech recognition system. 516, a parameter set 518 consisting of a plurality of parameters defining a learned model used in the speech recognition system, an inference program 520, and a training data set 530.

The learning program 514 is executed by the processor (CPU 502 and/or GPU 504) to realize learning processing for determining the parameter set 518. That is, the learning program 514 causes the computer to execute a learning process for learning the inference device (speech recognition system).

The model definition data 516 includes information for defining components included in the model constituting the speech recognition system, connection relationships between the components, and the like.

Parameter set 518 includes parameters for each component that makes up the speech recognition system. Each parameter included in the parameter set 518 is optimized by executing the learning program 514.

The inference program 520 executes inference processing using the model defined by the parameter set 518. That is, the inference program 520 implements an inference device as described later on a computer. Training data set 530 consists of a combination of data as shown in FIG.

A part of the library or function module required when the processor (CPU 502 and/or GPU 504) executes a program may be replaced by a library or function module provided as standard by the OS. In this case, although a single program does not include all of the program modules necessary to implement the corresponding function, it is possible to implement the desired processing by installing it in the execution environment of the OS. Even programs that do not include some of these libraries or functional modules may be included in the technical scope of the present invention.

Furthermore, these programs may not only be stored and distributed in any of the recording media as described above, but may also be distributed by being downloaded from a server device or the like via the Internet or an intranet.

Although FIG. 2 shows an example in which the information processing device 500 is configured using a single computer, the information processing apparatus 500 is not limited to this, and multiple computers connected via a computer network may explicitly or implicitly cooperate. , a trained model constituting a speech recognition system and an inference device using the trained model may be realized.

All or part of the functions realized by the processor (CPU 502 and/or GPU 504) executing a program may be realized using a hard-wired circuit such as an integrated circuit. For example, it may be realized using an ASIC (application specific integrated circuit), an FPGA (field-programmable gate array), or the like.

Those skilled in the art will be able to implement information processing device 500 according to this embodiment by appropriately using techniques appropriate to the era in which the present invention is implemented.

For convenience of explanation, an example is shown in which the same information processing device 500 is used to perform the learning process and the inference process, but the learning process and the inference process may be implemented using different hardware.

[D. First example (ideograms)]
As a first example, a speech recognition system using a single model for multiple languages using ideographic characters such as Chinese characters will be described.

(d1: Overview)
FIG. 3 is a schematic diagram showing an overview of the speech recognition system 100A according to the first embodiment. Referring to FIG. 3, the speech recognition system 100A receives an input sequence 2 indicating speech characteristics and outputs the corresponding text as an output sequence 70. That is, the speech recognition system 100A corresponds to an inference device that receives input of a speech signal uttered in any language among a plurality of languages and outputs the corresponding text.

At the beginning of the output sequence 70, a language label 72 (<TW>, <HK>, <MA>, etc.) indicating which language it is in is added. By adding such a language label 72, the language can be uniquely identified.

The speech recognition system 100A includes a Transformer 10 and a character synthesis section 80.

The Transformer 10 corresponds to a trained model that receives the input sequence 2 representing the audio characteristics of the audio signal and outputs an expression at a level different from the character level, which represents the characteristics of the characters included in the corresponding text. More specifically, the Transformer 10 creates a representation at a different level (hereinafter also referred to as "character part representation 82" or "Decomposed Character representation"), instead of at the character level, which indicates one or more character parts that constitute a kanji. Use. The character part representation 82 includes information specifying the structure of each character included in the corresponding text (details will be described later).

In this specification, a "character component" means an element that constitutes at least a part of a text to be output, and is defined in units that can be arbitrarily determined depending on the language system and the like.

The character synthesis unit 80 corresponds to a reconstruction unit that reconstructs a corresponding text from the expression output from the Transformer 10 (trained model) with reference to the correspondence between predetermined characters and the characteristics of the characters. . More specifically, the character synthesis unit 80 receives input of the character part representation 82 output from the Transformer 10, synthesizes it into characters (kanji) to be output, and outputs it as the output sequence 70.

In the first embodiment, a model is trained using an expression that shows a state in which a kanji is broken down into one or more character parts.

(d2: Character part representation 82)
The character part representation 82 shown in FIG. 3 is typically output as a sequence of data structures as shown below.

(1) <Language label> [Component identification information], [Part identification information], ..., <Delimiter>, [Part identification information], [Part identification information], ...
(2) <Language label> [Structure identification information], [Part identification information], [Part identification information], ..., <Delimiter>, [Part identification information], [Part identification information], ...
<Language label> included in the character part expression 82 includes information for specifying which language it is. As <language label>, <TW> (Taiwan), <HK> (Hong Kong), <MA> (Mandarin Chinese), etc. are used, for example.

[Component identification information] included in the character component representation 82 includes information for specifying character components that constitute the corresponding character. The <delimiter> included in the character part expression 82 means a delimiter between characters to be output, and the character to be output is determined based on the [component specific information] that exists between the <delimiter> and the next <delimiter>. Characters are reorganized. A blank (no output) may be simply used as the <delimiter>. In this way, the character part representation 82 includes information that specifies one or more character parts that constitute the corresponding character.

[Structure identification information] included in the character part expression 82 includes information for specifying a structure related to a combination of character parts forming a corresponding character. For example, in the case where a certain character is composed of two character parts arranged side by side, the [structure specifying information] includes information indicating that the characters are arranged side by side. In this way, the character part representation 82 may include information specifying the arrangement of one or more character parts.

Note that the data structure of the character part representation 82 described above is just an example, and any data structure may be employed as long as it allows characters to be reconstructed. Furthermore, the character part representation 82 may include more information.

(d3: Decomposition into character parts)
Next, an example of a method for decomposing characters into character parts will be described.

FIG. 4 is a diagram for explaining a method of decomposition into character parts in the speech recognition system 100A according to the first embodiment. Referring to FIG. 4, a plurality of character structures 802 are defined, and after determining which structure 802 each character corresponds to, each character is Alternatively, it is decomposed into a plurality of character parts 804.

Therefore, from each character, information on the determined structure 802 and one or more character parts 804 decomposed based on the information on the determined structure 802 is generated (simple decomposition 806).

Further, some characters may be determined to have multiple structures 802, and information on character parts 804 may be generated according to each structure 802 (mixed structure 808).

Regarding the character structure 802, any pattern may be determined based on the structure of Chinese characters, but as a typical example, 12 types of structures 802 may be prepared in advance.

(d4: Character synthesis section 80)
Next, an example of processing in the character synthesis unit 80 (see FIG. 3) of the speech recognition system 100A according to the first embodiment will be described.

As described above, the character part expression 82 consists of part specifying information for specifying one or more character parts that constitute a character to be output. The character synthesis unit 80 reconstructs characters to be output based on one or more pieces of part specifying information defined for each character included in the character part representation 82. The character part expression 82 has a character part correspondence table 84, and characters are reconstructed based on the character part correspondence table 84.

The character parts correspondence table 84 defines the correspondence between one or more character parts and corresponding characters for each language.

FIG. 5 is a diagram showing an example of the character-component correspondence table 84 used in the character synthesis section 80 of the speech recognition system 100A according to the first embodiment. Referring to FIG. 5, character component correspondence table 84 includes a plurality of sets of combination definitions 842 that define combinations of one or more character components and corresponding characters 844.

The character synthesis unit 80 separates the character part expression 82 output from the Transformer 10 at the position of the delimiter and extracts one or more parts specifying information. Then, the character synthesis unit 80 determines a corresponding character by referring to the character-component correspondence table 84 using the extracted one or more pieces of component specifying information as a key. By repeating the character determination process with reference to the character-component correspondence table 84, the text corresponding to the input sequence 2 is output as the output sequence 70.

The character component correspondence table 84 may be prepared for each language. In this case, the character synthesis unit 80 selects the character-component correspondence table 84 of the corresponding language based on the value of the language label included at the beginning of the sequence of character-component expressions 82 output from the Transformer 10.

Further, the character-component correspondence table 84 may include structure identification information (information for specifying a structure related to a combination of character parts constituting a corresponding character) in association with each data. By adding structure identification information, it is possible to distinguish between characters that are composed of the same character parts but that are arranged differently.

By referring to the character-component correspondence table 84 as described above, the output sequence 70 can be generated from the character-component representation 82 output from the Transformer 10.

(d5: learning process)
Next, an example of the learning process of the speech recognition system 100A according to the first embodiment will be described.

FIG. 6 is a schematic diagram for explaining the learning process of the speech recognition system 100A according to the first embodiment. Referring to FIG. 6, a set of input sequence 2 indicating voice features and corresponding text 64 is prepared as a training data set. The text 64 may include a language label indicating which language it is in.

In the learning process, as a preprocess, a character part representation 82 is generated in which each character included in the text 64 is decomposed into one or more character parts. When generating the character-component representation 82, the character-component correspondence table 84 may be referred to as necessary, and the contents of the character-component correspondence table 84 may be updated as appropriate.

Then, the model (Transformer 10) is learned using the set of the input sequence 2 and the corresponding character part representation 82 as training data. As for the model learning method itself, publicly known techniques can be adopted as appropriate.

FIG. 7 is a flowchart showing the procedure of the learning process of the speech recognition system 100A according to the first embodiment. The main steps shown in FIG. 7 are typically realized by the processor (CPU 502 and/or GPU 504) of the information processing device 500 executing the learning program 514.

Referring to FIG. 7, information processing device 500 receives an input of a training data set consisting of a pair of input sequence 2 indicating voice features and corresponding text (step S100). The information processing device 500 generates a character part representation 82 by decomposing each character included in the text of the received training data set into a combination of one or more character parts according to a predetermined rule (step S102). In this way, the information processing device 500 generates an expression at a level different from the character level that indicates the characteristics of characters included in the text. Then, the information processing device 500 generates a training data set consisting of a combination of the input sequence 2 indicating the voice feature and the corresponding character part representation 82 (step S104).

Subsequently, the information processing device 500 initializes the parameters of the Transformer 10 (step S106). Parameter optimization is then performed. That is, the parameters included in the Transformer 10 are optimized using the training data set.

More specifically, the information processing device 500 inputs the input sequence 2 included in the training data set to the Transformer 10 and calculates the output sequence (inference result of the character part representation 82) (step S108). Then, the information processing device 500 calculates error information by comparing the output sequence (inference result) and the corresponding character part representation 82 (correct data) of the training data set (step S110), and calculates the calculated error information. The parameters of the Transformer 10 are optimized based on (step S112).

The information processing device 500 determines whether a predetermined learning process termination condition is satisfied (step S114). If the predetermined learning processing termination condition is not met (NO in step S114), the information processing device 500 selects training data included in the training data set and re-executes the processing from step S108 onwards. .

On the other hand, if the predetermined learning processing termination condition is satisfied (YES in step S114), the information processing device 500 determines the Transformer 10 defined by the parameter values at the time as the learned model. (Step S116). The parameter values at this time are output as a parameter set 518 that defines the trained model. Then, the process ends.

(d6: Inference processing)
FIG. 8 is a flowchart showing the inference processing procedure of the speech recognition system 100A according to the first embodiment. The main steps shown in FIG. 8 are typically realized by the processor (CPU 502 and/or GPU 504) of the information processing device 500 executing the inference program 520.

Referring to FIG. 8, information processing device 500 generates an input sequence by calculating audio features from an input audio signal (step S150). The information processing device 500 inputs the generated input sequence to the Transformer 10 and calculates the character part representation 82 as an output sequence of the inference result (step S152). Subsequently, the information processing device 500 refers to the character-component correspondence table 84 and reconstructs the text from the character-component representation 82 (step S154). This reconstructed text is output as an output sequence.

The information processing device 500 then determines whether or not the audio signal continues to be input (step S156). If the input of the audio signal continues (YES in step S156), the processes from step S150 onwards are repeated.

On the other hand, if the input of the audio signal is not continuing (NO in step S156), the inference process is temporarily terminated.

(d7: Performance evaluation result)
Next, an example of the results of performance evaluation of the speech recognition system 100A according to the first embodiment will be shown.

In the first experimental example, evaluation was performed using training datasets for three languages that use Chinese characters: Taiwan <TW>, Hong Kong <HK>, and Mandarin Chinese <MA>. The speech recognition systems to be evaluated include a speech recognition system (related technology) that processes at the character level (indicated by "(c)" in the table), and a speech recognition system 100A (character) according to the first embodiment. (using parts representation) (indicated by "(r)" in the table).

We also compared the case in which each language was trained alone and the case in which a single model was trained in three languages. For evaluation, part of the dataset for each language was used as test data.

Character error rate (CER%) is used as an evaluation index of recognition performance.

As shown in Table 1, when a character-level speech recognition system is trained in a single language, it shows high performance in the learned language (MA (c), HK (c), TW (c) ). On the other hand, in the speech recognition system 100A according to the first embodiment, the performance is slightly inferior when trained in a single language (MA(r), HK(r), TW(r)).

However, when a single model is trained in three languages, the speech recognition system 100A (MA+HK+TW(r)) according to the first embodiment is less effective than the speech recognition system (MA+HK+TW(c)) according to the related art. It can be seen that the recognition performance is high.

Next, in a second experimental example, a comparison was made with a speech recognition system according to related technology in which learning is performed in character units and word units. At this time, in order to be able to compare with other speech recognition systems, the speech recognition system 100A according to the first embodiment is used with training data in three languages: Taiwan <TW>, Hong Kong <HK>, and Mandarin Chinese <MA>. In addition to the Japanese training dataset, we learned using the Japanese training dataset. As the Japanese training dataset, we used the Corpus of Spontaneous Japanese (CSJ). In Table 2, "E01", "E02", and "E03" mean CSJ-Eval01, CSJ-Eval02, and CSJ-Eval03, respectively.

At this time, for Japanese, we used a character part representation that includes character parts corresponding to kana in addition to kanji.

Furthermore, in Table 2, WPM (Wordpiece Model) is also shown as a comparative example.

As shown in Table 2, the recognition performance of the speech recognition system 100A according to the first example is equal to or higher than the recognition performance of the latest model.

Next, in a third experimental example, parameter sizes of a speech recognition system according to related technology were evaluated. The speech recognition system 100A according to the first embodiment (indicated by "(r)" in the table) and the speech recognition system according to related technology (indicated by "(c)" in the table) are used in Mandarin Chinese <MA> and Learning was performed using the Japanese <JP> training dataset.

A comparison of the state in which the speech recognition system 100A according to the first embodiment and the speech recognition system according to the related technology have been trained to achieve almost the same recognition performance is as shown in Table 3 below.

As shown in Table 3, when comparing models with almost the same character error rate (CER%), the parameter size of the speech recognition system 100A according to the first embodiment is 1/2 that of the speech recognition system according to the related technology. It can be seen that the parameter size is significantly suppressed.

[E. Second example (phonetic characters)]
As a second embodiment, a speech recognition system using a single model for multiple languages having similar pronunciation systems will be described.

(e1: Overview)
FIG. 9 is a schematic diagram showing an overview of a speech recognition system 100B according to the second embodiment. Referring to FIG. 9, speech recognition system 100B receives input sequence 2 indicating speech characteristics and outputs the corresponding text as output sequence 70. That is, the speech recognition system 100B corresponds to an inference device that receives an input of a speech signal uttered in any language among a plurality of languages and outputs a corresponding text.

At the beginning of the output sequence 70, a language label 72 (<MY>, <KH>, <SI>, <NE>, etc.) indicating which language it is in is added. By adding such a language label 72, the language can be uniquely identified.

The speech recognition system 100B includes a Transformer 10 and a character conversion section 90.

The Transformer 10 corresponds to a trained model that receives the input sequence 2 representing the audio characteristics of the audio signal and outputs an expression at a level different from the character level, which represents the characteristics of the characters included in the corresponding text. More specifically, the Transformer 10 uses a different level of expression (hereinafter also referred to as "universal audio representation 92" or "Universal Articulatory representation") instead of the character level. Universal phonetic representation 92 includes information that specifies the pronunciation of each character included in the corresponding text (details will be described later).

The character conversion unit 90 corresponds to a reconstruction unit that reconstructs a corresponding text from the expression output from the Transformer 10 (trained model) by referring to the correspondence between predetermined characters and the characteristics of the characters. . More specifically, the character conversion unit 90 receives the input of the universal speech expression 92 output from the Transformer 10, converts it into characters to be output, and outputs it as the output sequence 70.

In the second embodiment, a model is trained using expressions indicating sounds indicated by characters.
(e2: Universal phonetic expression 92)
Universal phonetic expression 92 is an expression that defines the pronunciation of text. The pronunciation of text is generally defined using the International Pronunciation Alphabet (IPA). Here, phone-sets differ between different languages, but when IPA is used, it is difficult to appropriately define such different phone-sets.

Therefore, the speech recognition system 100B according to the second embodiment uses a universal speech expression 92 based on universal features that express the phonological structure of various languages. Universal features include (1) round/unrounded lips, (2) tongue (low, middle, high), (3) tongue (front, middle, back), (4) presence/absence of voice (vocal fold vibration), (5) ) consonants (airflow), (6) lips, tongue tip, tongue dorsum, and throat sounds. Additionally, other factors such as tone may be added as universal features.

More specifically, as shown in the universal audio table in Table 4 below, a plurality of attributes are defined for each of the three categories. The three categories include consonants(position), consonants(manner), and vowel.

Universal phonetic representation 92 is generated by assigning a combination of one or more attributes to each character.

Universal speech representation 92 is typically output as a sequence of data structures such as the following.

<Language label> [Attribute], [Attribute], ..., <Delimiter>, [Attribute], [Attribute], ...
<Language label> included in the universal speech expression 92 includes information for specifying which language it is.

[Attributes] included in the universal audio expression 92 includes information for specifying universal features defined according to the universal audio table of Table 4. Thus, the universal phonetic representation 92 includes information that specifies the pronunciation of the corresponding character based on the universal features that represent the phonological structure.

The <delimiter> included in the universal phonetic expression 92 means the delimiter between the characters to be output, and the character to be output is determined based on the [attribute] that exists between the <delimiter> and the next <delimiter>. Reconfigured. A blank (no output) may be simply used as the <delimiter>.

Note that the data structure of the universal speech expression 92 described above is just an example, and any data structure may be adopted as long as it allows characters to be reconstructed.

As described above, in the speech recognition system 100B according to the second embodiment, learning processing and inference processing are performed not at the character level but at the level of universal features that define the pronunciation of each character.

(e3: Processing details)
Next, details of processing in the speech recognition system 100B according to the second embodiment will be explained.

FIG. 10 is a schematic diagram for explaining the contents of learning processing and inference processing in the speech recognition system 100B according to the second embodiment. Referring to FIG. 10, a training data set 530 including multilingual audio data 531 and multilingual text data 532 is used in the learning process. The multilingual text data 532 may include a language label indicating which language it is in.

It is input to the Transformer 10 as a voice feature (input sequence) extracted from the multilingual voice data 531.

Further, the universal feature conversion 91 is applied to the multilingual text data 532, and a combination of one or more attributes indicating the pronunciation of each character included in the multilingual text data 532 is output. Language labels included in multilingual text data 532 are also extracted.

A universal speech expression 92 including a language label and a combination of one or more attributes is input to the Transformer 10 as a corresponding label (correct data).

That is, the parameters of the Transformer 10 are optimized based on a set of audio features and a universal audio expression 92, which is generated from a set of multilingual audio data 531 and multilingual text data 532.

On the other hand, in the inference process, the voice features (input sequence) extracted from the multilingual voice data 533 to be recognized are input to the Transformer 10. Transformer 10 outputs universal speech expression 92 as an inference result. The character conversion unit 90 converts the universal speech expression 92 into text data 534 and outputs it as an inference result.

FIG. 11 is a diagram for explaining processing related to universal speech expression in the speech recognition system 100B according to the second embodiment. In FIG. 11, processing is shown in association with the learning processing and inference processing shown in FIG.

Referring to FIG. 11, in the learning process, the text included in multilingual text data 532 is divided into units of words 96, and then further divided into units of characters 97. Finally, one or more attribute combinations 98 are assigned to each character 97. At this time, the audio feature correspondence table 94 is referred to. In this way, the universal phonetic expression 92 includes information that defines the pronunciation for each character 97 that is further decomposed from the word 96 included in the corresponding text.

The speech feature correspondence table 94 defines the correspondence between information specifying pronunciation and corresponding characters for each language. More specifically, the audio feature correspondence table 94 defines the correspondence between each character and one or more attributes.

FIG. 12 is a diagram showing an example of the speech feature correspondence table 94 used in the character conversion section 90 of the speech recognition system 100B according to the second embodiment. Referring to FIG. 12, audio feature correspondence table 94 defines characters and combinations of one or more attributes of universal features corresponding to the characters. The audio feature correspondence table 94 may be prepared for each language.

Referring again to FIG. 11, in the inference process, the speech feature correspondence table 94 is referred to, and the characters 97 are sequentially converted into characters 97 corresponding to attribute combinations 98 included in the inference results corresponding to the input sequence indicating the speech features. Ru. Then, a word 96 is reconstructed from the characters 97 obtained by the conversion and output as an inference result.

A speech recognition system can be constructed and operated through the processing procedure described above.
(e4: learning process)
Next, an example of the learning process of the speech recognition system 100B according to the second embodiment will be described.

FIG. 13 is a flowchart showing the procedure of the learning process of the speech recognition system 100B according to the second embodiment. The main steps shown in FIG. 13 are typically realized by the processor (CPU 502 and/or GPU 504) of the information processing device 500 executing the learning program 514.

Referring to FIG. 13, information processing device 500 receives an input of a training data set consisting of a pair of input sequence 2 indicating voice features and corresponding text (step S200). The information processing apparatus 500 divides the text of the received training dataset into words (step S202), and divides each divided word into characters (step S204). Further, the information processing device 500 determines a combination of one or more attributes of the universal feature for each character (step S206). A universal audio expression 92 as a label is generated from the determined combination of one or more attributes. At this time, the audio feature correspondence table 94 corresponding to the target text language may be referred to. In this way, the information processing device 500 generates an expression at a level different from the character level that indicates the characteristics of characters included in the text.

The information processing device 500 generates a training data set consisting of a combination of the input sequence 2 indicating voice features and one or more corresponding attributes (step S208).

Subsequently, the information processing device 500 initializes the parameters of the Transformer 10 (step S210). Parameter optimization is then performed. That is, the parameters included in the Transformer 10 are optimized using the training data set.

More specifically, the information processing device 500 inputs the input sequence 2 included in the training data set to the Transformer 10 and calculates the output sequence (universal speech expression 92) (step S212). Then, the information processing device 500 calculates error information by comparing the output sequence (inference result) and the corresponding universal speech expression 92 (correct data) of the training data set (step S214), and calculates the calculated error information. The parameters of the Transformer 10 are optimized based on (step S216).

The information processing device 500 determines whether a predetermined learning process termination condition is satisfied (step S218). If the predetermined learning processing termination condition is not met (NO in step S218), the information processing device 500 selects training data included in the training data set and re-executes the processing from step S212 onwards. .

On the other hand, if the predetermined learning process termination condition is satisfied (YES in step S218), the information processing device 500 determines the Transformer 10 defined by the parameter values at the time as the learned model. (Step S220). The parameter values at this time are output as a parameter set 518 that defines the learned model. Then, the process ends.

(e5: Inference processing)
FIG. 14 is a flowchart showing the inference processing procedure of the speech recognition system 100B according to the second embodiment. The main steps shown in FIG. 14 are typically realized by the processor (CPU 502 and/or GPU 504) of the information processing device 500 executing the inference program 520.

Referring to FIG. 14, information processing device 500 generates an input sequence by calculating audio features from the input audio signal (step S250). The information processing device 500 inputs the generated input sequence to the Transformer 10 and calculates the universal speech expression 92 as an output sequence of the inference result (step S252). Subsequently, the information processing device 500 refers to the speech feature correspondence table 94, converts the universal speech expression 92 into characters (step S254), and reconstructs a word from the plurality of converted characters (step S256). Finally, a text consisting of a plurality of reconstructed words is generated (step S258). This generated text is output as an output sequence.

Then, the information processing device 500 determines whether or not the audio signal continues to be input (step S260). If the audio signal continues to be input (YES in step S260), the processes from step S250 onwards are repeated.

On the other hand, if the input of the audio signal is not continuing (NO in step S260), the inference process ends once.

(e6: Performance evaluation results)
Next, an example of the results of performance evaluation of the speech recognition system 100B according to the second embodiment will be shown.

In the second experimental example, we used training data sets for four languages that use kanji: Malaysian <MY>, Khmer <KH>, Sinhalese <SI>, and Nepali <NE>, which are used in Asia. We conducted an evaluation using The speech recognition systems to be evaluated include a speech recognition system (related technology) that processes at the word level (indicated by "(w)" in the table), a speech recognition system that processes at the character level (related technology), and a speech recognition system that processes at the character level (related technology). technology) (indicated by "(c)" in the table), a speech recognition system that processes at the phonetic symbol level according to the International Phonetic Alphabet (IPA) (related technology) (indicated by "(p)" in the table), and A speech recognition system 100B (using universal speech expression) according to the second embodiment (indicated by "(a)" in the table) was adopted.

Table 5 shows changes in parameter sizes when learning was performed for each language alone and for four languages.

As shown in Table 5, it can be seen that in all evaluation examples, the parameter size of the speech recognition system 100B according to the second example is the smallest.

Furthermore, Table 6 shows changes in recognition performance when learning was performed for each language alone and for four languages. Character error rate (CER%) is used as an evaluation index of recognition performance.

As shown in Table 6, the recognition performance of the speech recognition system 100B according to the second embodiment is equal to or higher than the recognition performance of the speech recognition system (related technology) that processes at the phonetic symbol level according to the International Phonetic Alphabet (IPA). It becomes. As shown in Table 5, considering that the parameter size can be significantly reduced, it is possible to realize a multilingual end-to-end speech recognition system using a model with a smaller parameter size by using the universal speech representation. I understand.

[F. Application examples and modifications]
As an application example using the speech recognition system according to this embodiment, an automatic speech translation system or the like may be realized. In this case, this can be realized by further adding a speech synthesis unit that outputs speech corresponding to the text output from the speech recognition system according to the present embodiment.

Further, the first embodiment and the second embodiment described above can also be realized using a single model. In this case, the number of dimensions of the output layer of the Transformer 10 may be set so that both the character part representation and the universal audio representation can be output. In addition, in addition to the first embodiment and/or the second embodiment, it is also possible to add a language in which learning is performed at the character level or word level.

[G. summary]
According to the learning process according to this embodiment, by using a trained model that uses expressions at a level different from the character level, an estimator that can improve recognition performance while suppressing an increase in parameter size is created. realizable. This makes it possible to provide a technique for realizing a multilingual end-to-end speech recognition system using a model with a smaller parameter size.

The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims rather than the description of the embodiments described above, and it is intended that all changes within the meaning and range equivalent to the claims are included.

2 input sequence, 4 input embedding layer, 6, 16 position embedding layer, 8, 18 adder, 10 Transformer, 14 output embedding layer, 20 encoder block, 22, 46 MHA layer, 24, 28, 44, 48, 52 addition・Regularization layer, 26, 50 Feedforward layer, 40 Decoder block, 42 MMHA layer, 60 Softmax layer, 64 Text, 70 Output sequence, 72 Language label, 80 Character synthesis section, 82 Character component representation, 84 Character component support table, 90 character conversion unit, 91 universal feature conversion, 92 universal speech expression, 94 speech feature correspondence table, 96 word, 97,844 character, 98 combination of attributes, 100A, 100B speech recognition system, 200 encoder, 400 decoder, 500 Information processing device, 502 CPU, 504 GPU, 506 main memory, 508 display, 510 network interface, 512 secondary storage device, 514 learning program, 516 model definition data, 518 parameter set, 520 inference program, 522 input device, 524 optics drive, 526 optical disk, 528 internal bus, 530 training data set, 531,533 multilingual audio data, 532 multilingual text data, 534 text data, 802 structure, 804 character parts, 806 simple decomposition, 808 mixed structure, 842 combination Definition.

Claims (6)

An inference device that receives input of an audio signal uttered in any language among a plurality of languages and outputs a corresponding text,
a trained model that receives an input sequence representing audio characteristics of the audio signal and outputs an expression at a level different from a character level representing characteristics of characters included in the corresponding text;
a reconstruction unit that reconstructs a corresponding text from an expression output from the learned model by referring to a correspondence relationship between a predetermined character and a feature of the character ,
An inference device , wherein the expression output from the trained model includes information specifying the structure of each character included in the corresponding text . An inference device that receives input of an audio signal uttered in any language among a plurality of languages and outputs a corresponding text,
a trained model that receives an input sequence representing audio characteristics of the audio signal and outputs an expression at a level different from a character level that represents characteristics of characters included in the corresponding text;
a reconstruction unit that reconstructs a corresponding text from an expression output from the learned model by referring to a correspondence relationship between a predetermined character and a feature of the character ;
An inference device , wherein the expression output from the trained model includes information for specifying which language the corresponding text is in . The inference device according to claim 2 , wherein the expression output from the trained model includes information specifying the pronunciation of each character included in the corresponding text. 4. The reasoning device according to claim 3, wherein the information specifying the pronunciation of the character includes information specifying the pronunciation of the corresponding character based on a universal feature expressing a phonological structure. An inference program for implementing the inference device according to any one of claims 1 to 4 on a computer. A learning method for learning a reasoner that receives an input of an audio signal uttered in any language among multiple languages and outputs a corresponding text, the method comprising:
providing an audio signal and a corresponding text;
generating a representation at a level different from the character level that indicates characteristics of characters included in the text;
optimizing parameters defining the inference device based on an error between an inference result obtained by inputting an input sequence indicating audio characteristics of the audio signal to the inference device and a corresponding expression ;
The learning method, wherein the expression at a level different from the character level includes information specifying the structure of each character included in the corresponding text or information specifying which language the corresponding text is in.

Images