WO2022267451A1

WO2022267451A1 - Automatic speech recognition method based on neural network, device, and readable storage medium

Info

Publication number: WO2022267451A1
Application number: PCT/CN2022/071220
Authority: WO
Inventors: 方明; 魏韬; 马骏; 王少军
Original assignee: 平安科技（深圳）有限公司
Priority date: 2021-06-24
Filing date: 2022-01-11
Publication date: 2022-12-29
Also published as: CN113450805B; CN113450805A

Abstract

The present application relates to artificial intelligence, and provides an automatic speech recognition method and apparatus based on a neural network, an electronic device, and a computer-readable storage medium. The method comprises: performing recognition processing, by means of an acoustic model and a ngram language model in an ASR recognition process, on an audio to be recognized to obtain at least more than two primary recognition results; transmitting the primary recognition results to a rescore process, and performing scoring processing by means of a gpt language model in the rescore process to obtain a gpt language model score; transmitting the gpt language model score to the ASR recognition process, and replacing a ngram language model score in the ASR recognition process; and sorting the sum of the gpt language model score and the acoustic model score in the ASR recognition process, and taking the top recognition result in a sorting result as a final recognition result. The present application mainly aims to solve the problem of data sparsity by using the gpt language model.

Description

Automatic speech recognition method, device and readable storage medium based on neural network

This application claims the priority of the Chinese patent application with the application number 202110706592.9 submitted to the China Patent Office on June 24, 2021, and the invention title is "Automatic Speech Recognition Method, Device and Readable Storage Medium Based on Neural Network", all of which The contents are incorporated by reference in this application.

technical field

The present application relates to the technical field of artificial intelligence, and in particular to a neural network-based automatic speech recognition method, device, electronic equipment, and computer-readable storage medium.

Background technique

In the process of traditional speech recognition, there are two models, namely the acoustic model and the language model; among them, the language model generally adopts the ngram language model, and ngram, a probability model based on tuple statistics, can only capture the statistics before and after the phrase information, it is impossible to learn more in-depth grammatical and semantic information, coupled with this probability calculation method of word frequency statistics, there are problems with too large parameter space and serious data sparseness, especially in high-order ngram models, with the order increase, the ngram model and sparsity will increase exponentially. Even if people propose and work hard to solve the problems of the ngram model itself, such as pruning and regression, they only weaken the problem of the ngram model, and cannot solve the fundamental problems of the ngram language model.

At present, a common solution is to keep the original ngram model unchanged, and after wfst decoding, generate top n ARS recognition results, re-score and reorder the language model of the generated sentences. The inventor realizes that using an ngram model with more corpus, a higher-order ngram model, etc.; but the problem that is often encountered is that using a more complex language model often leads to more recognition delays, and using a simple language model It is often impossible to obtain accurate recognition results.

In order to solve the above problems, a new automatic speech recognition scheme is urgently needed.

technical problem

The present application provides a neural network-based automatic speech recognition method, device, electronic equipment and computer-readable storage medium, the main purpose of which is to solve the problem of data sparsity by using the gpt language model.

In order to achieve the above purpose, the neural network-based automatic speech recognition method provided by the application is applied to electronic equipment, and the method includes:

Through the acoustic model and the ngram language model in the ASR recognition process, the audio to be recognized is jointly recognized and processed, and at least two or more initial recognition results are obtained; wherein, each recognition result includes an acoustic model score, an ngram language model score, and the acoustic The sum of the model score and the ngram language model score;

The initial recognition result is transmitted to the rescore process, and the gpt language model in the rescore process is used for scoring processing to obtain the gpt language model score;

Transmitting the gpt language model score to the ASR recognition process, and replacing the ngram language model score in the ASR recognition process;

Sorting the sum of the gpt language model scores and the acoustic model scores in the ASR recognition process, and using the top recognition result among the sorted results as the final recognition result.

In order to solve the above problems, the application also provides a neural network-based automatic speech recognition device, including:

The initial identification result acquisition module is used to jointly identify and process the audio to be identified through the acoustic model and the ngram language model in the ASR identification process, and obtain at least two initial identification results; wherein each identification result includes an acoustic model score, ngram language model score and the sum of the acoustic model score and the ngram language model score;

The gpt language model score acquisition module is used to transmit the recognition result to the rescore process, and perform scoring processing through the gpt language model in the rescore process to obtain the gpt language model score;

A language model score replacement module, configured to transmit the gpt language model score to the ASR recognition process, and replace the ngram language model score in the ASR recognition process;

The final recognition result acquisition module is used to sort the sum of the gpt language model score and the acoustic model score in the ASR recognition process, and use the top-ranked recognition result among the sorted results as the final recognition result.

In order to solve the above problems, the present application also provides an electronic device, which includes:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can perform the above-mentioned neural network-based automatic speech recognition method. step.

In order to solve the above problems, the present application also provides a computer-readable storage medium, at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in the electronic device to realize the above-mentioned Automatic Speech Recognition Method Based on Neural Network.

In the embodiment of the present application, the acoustic model in the ASR recognition process and the ngram language model are used to jointly identify and process the audio to be recognized, and at least two or more initial recognition results are obtained; the initial recognition results are transmitted to the rescore process, and passed through the rescore process The gpt language model in carries out scoring process, obtains gpt language model score; Described gpt language model score is transmitted to described ASR identification process, and replaces the ngram language model score in the described ASR identification process; Described ASR identification process The sum of the score of the gpt language model and the score of the acoustic model is sorted, and the result of the sorting of the score of the gpt language model and the score of the acoustic model in the ranking result is the final recognition result . The main purpose of this application is to solve the problem of data sparsity by using the gpt language model.

technical solution

Type technical solution description paragraph here.

Beneficial effect

Type benefit description paragraph here.

Description of drawings

Fig. 1 is the schematic flow chart of the neural network-based automatic speech recognition method that an embodiment of the present application provides;

Fig. 2 is the module schematic diagram of the automatic speech recognition device based on neural network provided by an embodiment of the present application;

3 is a schematic diagram of the internal structure of an electronic device implementing a neural network-based automatic speech recognition method provided by an embodiment of the present application;

The realization, functional features and advantages of the present application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

BEST MODE FOR CARRYING OUT THE INVENTION

Type the paragraph describing the best mode for carrying out the invention here.

Embodiments of the present invention

In order to solve the above problems, the present application provides an automatic speech recognition method based on a neural network. Referring to FIG. 1 , it is a schematic flowchart of a neural network-based automatic speech recognition method provided by an embodiment of the present application. The method may be performed by a device, and the device may be implemented by software and/or hardware.

In this embodiment, the automatic speech recognition method based on neural network includes:

S1: Through the acoustic model and the ngram language model in the ASR recognition process, the audio to be recognized is jointly recognized and processed, and at least two or more initial recognition results are obtained; each recognition result includes the acoustic model score, the ngram language model score and all The sum of the acoustic model score and the ngram language model score;

S2: Transmit the initial recognition result to the rescore process, and perform scoring processing through the gpt language model in the rescore process to obtain the gpt language model score;

S3: Transmitting the gpt language model score to the ASR recognition process, and replacing the ngram language model score in the ASR recognition process;

S4: Sort the sum of the scores of the gpt language model and the scores of the acoustic model in the ASR recognition process, and use the top recognition result among the sorted results as the final recognition result.

The above is the neural network-based automatic speech recognition method of the artificial intelligence of the present application. In the neural network-based automatic speech recognition method of the present application, it includes ASR (Automatic Speech Recognition, Automatic Speech Recognition, referred to as "ASR") recognition process and rescore (re-scoring) process. The ASR recognition process is the sound-to-character module, which adopts the traditional GMM-HMM technical route. The modeling unit of the neural network is hmm state, and the input is acoustic features. The ASR recognition is completed through two modules of acoustic model and language model decoding. Before the recognition starts, the training of the acoustic model and the language model is required. The training principle of the acoustic model is the backpropagation of the neural network, which requires a large amount of audio duration and marked text; the training input of the language model is a large amount of text corpus, and the 3gram language is used. Model modeling generates a language model in arpa format, and then uses tools such as arpa2fst to generate a wfst graph of hclg structure, which is used as input for language model decoding.

In step S1, the acoustic model and the ngram language model in the ASR recognition process perform recognition processing on the audio to be recognized, and obtain at least two recognition results, including the following steps:

S111: Convert the audio to be recognized into audio features;

S112: Obtain the posterior probability of each frame in the audio feature according to the audio feature;

S113: According to the posterior probability of each frame, perform viterbi decoding on the wfst graph generated by the ngram language model to generate a lattice graph; and

S114: Obtain top n recognition results according to the lattice graph; wherein, each recognition result includes text, an acoustic model score, an ngram language model score, and a sum of scores of two models (acoustic model, ngram language model).

Wherein, in step S111, converting the audio to be recognized into audio features includes the following steps:

Step S11101: Framing and windowing the to-be-recognized audio speech to obtain standardized audio; and,

Step S11102: Perform feature extraction on the standard audio through the MFCC feature extraction algorithm to obtain audio features of the audio to be recognized.

In step S112, the posteriori probability of each frame in the audio feature will be obtained according to the audio feature, including the following steps:

Step S11201: extracting the audio feature into an audio feature vector sequence;

Step S11202: Input the audio features of the audio feature vector sequence into the pre-trained acoustic model to determine the time boundary of the phoneme state;

Step S11203: According to the time boundary, extract all the frames in the time boundary, take the average value according to the frame length of the speech frame, and use it as the posterior probability of the speech frame.

Wherein, in step S113, according to the posterior probability of each frame, the wfst graph generated by the ngram language model is subjected to viterbi decoding to generate a lattice graph, including the following steps:

Step S11301: modeling the ngram language model to generate a language model in arpa format;

S11302: Use the arpa2fst tool to generate a wfst diagram of the hclg structure;

S11303: Construct a wfst search space according to the Viterbi algorithm (viterbi), the posterior probability, and the wfst graph;

S11304: Find the optimal path with the highest matching probability in the weighted finite-state transducer (wfst) search space, and obtain the text recognition result.

Wherein, each recognition result includes a text, an acoustic model score, a language model score, and a sum of the acoustic model score and the language model score.

In the embodiment of the present application, the scores of the acoustic model and the language model can be extracted from the lattice graph. Sorting the total score of each output of lattice from small to large, backtracking the top1 result is the default ASR result output by the ngram language model, backtracking the top n result can extract nbest information, and output it to the rescore process to complete the re-scoring work.

In step S120, the rescore thread is the re-scoring module, which is designed as a separate process because of the gpu dependency, works on the gpu and is accelerated by TensorRT, and is responsible for decoding the request and response of the thread in addition to completing the reasoning process of the gpt language model . The Rescore module will input text sentences with a fixed batch size each time, and output the gpt language model score corresponding to each text.

Wherein, the described recognition result is transmitted to the rescore process, and processed by the gpt language model in the rescore process, to obtain the gpt language model score, including the following steps:

Step S121: within the preset time, put together the sentences to be rescored (re-scored) into a batch of sentences to be rescored;

Step S122: Perform neural network forward reasoning on batches of sentences to be re-scored through the gpt language model;

Step S123: Accumulate the posterior probability of each word of the sentence to be re-scored, and output it in logarithmic form, so as to obtain the gpt language model score of the sentence to be re-scored.

In a specific embodiment of the present application, for example, suppose the text to be scored is "[CLS] the dog is hairy [SEP]", then the text sequence input to the gpt model is "[CLS] the dog is hairy Hairy" has 5 words in total, and the probability of the current word corresponding to the next word is taken on the output probability matrix. Specifically, in this example, in the output probability distribution of the word "the", the next word of the current word is "dog", Then take the probability item of the word "dog" in the probability distribution item of "the" as its corresponding output probability. After this processing, it is assumed that the output logarithmic probability sequence of the above input sequence is p1 p2 p3 p4 p5, accumulating p1 to p5 is the gpt language model score.

In steps S130 and S140, in the rescore process, the gpt language model score result is returned to the ASR decoding thread, and the ASR decoding thread replaces the ngram score in the total score in the top n sentence with the gpt language model score, and then calculates the new The total score is reordered from small to large, and the reordered top 1 ASR text is used as the final ASR recognition result, that is, the top recognition result of the gpt language model score and acoustic model score sorting results is used as the final recognition result , so as to achieve the purpose of improving the accuracy of ASR.

In the embodiment of this application, GPT rescore is performed on the ASR recognition results. After experimental testing, the accuracy of the overall ASR recognition results is greatly improved, often by about 2 percentage points, and the character accuracy is reduced by about 1 percentage point. . From the perspective of delay, it will only cause a delay of about 50ms, and the impact on the delay of the overall speech recognition system is very limited. The improvement of recognition accuracy is not just a simple drop in character accuracy, but also brings a better ASR experience. There are also upstream systems that rely on ASR recognition results, such as voice customer service robots, intelligent voice assistants, smart speakers, etc., which can indirectly improve the effect of upstream systems, improve service quality, and improve customer satisfaction.

In the embodiment of the present application, the acoustic model in the ASR recognition process and the ngram language model are used to jointly identify and process the audio to be recognized, and at least two or more initial recognition results are obtained; the initial recognition results are transmitted to the rescore process, and passed through the rescore process The gpt language model in carries out scoring process, obtains gpt language model score; Described gpt language model score is transmitted to described ASR identification process, and replaces the ngram language model score in the described ASR identification process; Described ASR identification process The sum of the gpt language model score and the acoustic model score in is sorted, and the recognition result ranked first among the sorted results is taken as the final recognition result. The main purpose of this application is to solve the problem of data sparsity by using the gpt language model.

As shown in FIG. 2 , it is a functional block diagram of the neural network-based automatic speech recognition device of the present application.

The substrate 100 described in this application can be installed in electronic equipment. According to the realized functions, the neural network-based automatic speech recognition device may include: an initial recognition result acquisition module 101 , a gpt language model score acquisition module 102 , a language model score replacement module 103 and a final recognition result acquisition module 104 . The module described in this application can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of the electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The initial recognition result acquisition module 101 is used to jointly identify and process the audio to be recognized through the acoustic model and the ngram language model in the ASR recognition process, and obtain at least two or more initial recognition results; wherein, each recognition result includes an acoustic model score , ngram language model score and the sum of the acoustic model score and the ngram language model score;

The gpt language model score acquisition module 102 is used to transmit the recognition result to the rescore process, and perform scoring processing through the gpt language model in the rescore process to obtain the gpt language model score;

The language model score replacement module 103 is used to transmit the gpt language model score to the ASR recognition process, and replace the ngram language model score in the ASR recognition process;

The final recognition result acquisition module 104 is configured to sort the sum of the gpt language model scores and the acoustic model scores in the ASR recognition process, and use the top-ranked recognition result among the sorted results as the final recognition result.

In the embodiment of the present application, the initial recognition result acquisition module 101 includes an audio feature conversion module, a posterior probability acquisition module, a lattice graph acquisition module and more than two recognition result acquisition modules, wherein,

The audio feature conversion module is used to convert the audio to be identified into an audio feature;

The posterior probability obtaining module is used to obtain the posterior probability of each frame in the audio feature according to the audio feature;

The lattice graph acquisition module is used to perform viterbi decoding on the wfst graph generated by the ngram language model to generate a lattice graph according to the posterior probability of each frame;

The two or more identification result acquisition modules are configured to acquire at least two or more initial identification results according to the lattice graph.

Wherein, the audio feature conversion module includes a standard audio acquisition module and an audio feature acquisition module of the audio to be recognized. in,

The standard audio acquisition module is used to perform frame division and window processing on the audio speech to be recognized to obtain standard audio; and,

The audio feature acquisition module of the audio to be identified is used to extract the features of the standard audio through the MFCC feature extraction algorithm, and acquire the audio features of the audio to be identified.

Wherein, the posterior probability acquisition module is used to acquire the posterior probability of each frame of the audio feature according to the audio feature, including: an audio feature vector sequence acquisition module, a time boundary determination module of a phoneme state, a posteriori Test probability determination module. in,

The audio feature vector sequence acquisition module is used to extract the audio feature into an audio feature vector sequence;

The time boundary determination module of the phoneme state is used to input the audio feature vector sequence audio feature into the pre-trained acoustic model to determine the time boundary of the phoneme state;

The posterior probability determination module is used to extract all frames in the time boundary according to the time boundary, and take an average value according to the frame length of the speech frame as the posterior probability of the speech frame.

Wherein, the lattice diagram generation module includes: arpa format generation module, wfst diagram generation module, wfst search space construction module and text recognition result determination module, wherein,

The arpa format generation module is used to model the ngram language model to generate an arpa format language model;

The wfst graph generation module is used to generate the wfst graph of the hclg structure by using the arpa2fst tool;

A wfst search space construction module is used to construct a wfst search space according to the Viterbi algorithm (viterbi), the posterior probability and the wfst graph;

The character recognition result determination module is used to find the optimal path with the highest matching probability in the wfst search space search space to obtain the character recognition result.

Among them, wfst (weightedfinite-statetransducer), specifically refers to the weighted finite state transducer.

In the embodiment of the present application, each recognition result includes the text, the score of the acoustic model, the score of the language model and the sum of the scores of the two models (acoustic model, language model).

In the gpt language model score acquisition module 102, the rescore thread is the re-scoring module, which is designed as a separate process due to gpu dependence, works on the gpu and uses TensorRT to accelerate, in addition to completing the reasoning process of the gpt language model, it also It is responsible for decoding the request and response of the thread. The Rescore module will input text sentences with a fixed batch size each time, and output the gpt language model score corresponding to each text.

Wherein, in the gpt language model score acquisition module 102, within a preset time, the sentences to be rescored (re-marked) are pieced together into batches of sentences to be re-scored;

Perform neural network forward reasoning on batches of sentences to be re-scored through the gpt language model;

Accumulate the posterior probability of each word of the sentence to be re-scored, and output it in logarithmic form, so as to obtain the gpt language model score of the sentence to be re-scored.

In a specific embodiment of the present application, for example, suppose the text to be scored is "[CLS] the dog is hairy [SEP]", then the text sequence input to the gpt model is "[CLS] the dog is hairy Hairy" has 5 words in total, and the probability of the current word corresponding to the next word is taken on the output probability matrix. Specifically, in this example, in the output probability distribution of the word "the", the next word of the current word is "dog", Then take the probability item of the word "dog" in the probability distribution item of "the" as its corresponding output probability. After this processing, it is assumed that the output logarithmic probability sequence of the above input sequence is p1, p2, p3, p4, p5, Accumulating p1 to p5 is: gpt language model score.

In the language model score replacement module 103 and the final recognition result acquisition module 104, the rescore process returns the gpt language model score result to the ASR decoding thread, and the ASR decoding thread replaces the ngram in the total score in the top n sentence with the gpt language model score Score, and then reorder the new total score from small to large, and use the reordered top 1 ASR text as the final ASR recognition result (the top recognition result in the sorting result is the final recognition result), so as to improve the accuracy of ASR rate purposes.

In the embodiment of the present application, the acoustic model in the ASR identification process and the ngram language model are used to identify and process the audio to be identified, and at least two or more initial identification results are obtained; the initial identification results are transmitted to the rescore process, and Score processing by the gpt language model in the rescore process to obtain the gpt language model score; transfer the gpt language model score to the ASR recognition process, and replace the ngram language model score in the ASR recognition process; The sum of the gpt language model score and the acoustic model score in the ASR recognition process is sorted, and the recognition result ranked first in the sorted results is used as the final recognition result. The main purpose of this application is to solve the problem of data sparsity by using the gpt language model.

As shown in FIG. 3 , it is a schematic structural diagram of an electronic device implementing the neural network-based automatic speech recognition method of the present application.

The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and operable on the processor 10, such as an automatic speech recognition program 12 based on a neural network .

Wherein, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, magnetic disk, CD etc. The storage 11 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1 . The memory 11 can also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart memory card) equipped on the electronic device 1 Media Card, SMC), Secure Digital (Secure Digital, SD) card, flash memory card (Flash Card) and so on. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various data installed in the electronic device 1 , such as the code of a data audit program, but also can be used to temporarily store data that has been output or will be output. The memory may store content that may be displayed by the electronic device or sent to other devices (eg, headphones) for display or playback by other devices. The memory may also store content received from other devices. The content from other devices may be displayed, played, or used by the electronic device to perform any necessary tasks or operations that may be performed by computer processors or other components in the electronic device and/or wireless access point.

In some embodiments, the processor 10 may be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same function or different functions, including one or more Central processing unit (Central Processing unit, CPU), microprocessor, digital processing chip, graphics processor and a combination of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and runs or executes programs or modules stored in the memory 11 (such as data audit program, etc.), and call the data stored in the memory 11 to execute various functions of the electronic device 1 and process data. The electronics may also include a chipset (not shown) for controlling communications between the one or more processors and one or more of the other components of the user device. In particular embodiments, the electronic device may be based on Intel® architecture or ARM® architecture, and the processor and chipset may be from the Intel® processor and chipset family. The one or more processors 104 may also include one or more application specific integrated circuits (ASICs) or application specific standard products (ASSPs) for handling specific data processing functions or tasks.

The bus may be a peripheral component interconnect standard (PCI for short) bus or an extended industry standard architecture (extended industry standard architecture, referred to as EISA) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10 and the like.

Additionally, network and I/O interfaces may include one or more communication interfaces or network interface devices to provide for data transfer between the electronic device and other devices (eg, web servers) via a network (not shown). Communication interfaces may include, but are not limited to: Body Area Network (BAN), Personal Area Network (PAN), Wired Local Area Network (LAN), Wireless Local Area Network (WLAN), Wireless Wide Area Network (WWAN), and the like. User equipment 102 may be coupled to the network via a wired connection. However, the wireless system interface may include hardware or software to broadcast and receive messages using the Wi-Fi Direct standard and/or the IEEE 802.11 wireless standard, the Bluetooth standard, the Bluetooth low energy standard, the Wi-Gig standard, and/or any other wireless standards and/or combinations thereof.

A wireless system may include a transmitter and a receiver or transceiver capable of operating over a wide range of operating frequencies governed by the IEEE 802.11 wireless standard. A communication interface may utilize acoustic, radio frequency, optical, or other signals to exchange data between the electronic device and other devices, such as access points, hosts, servers, routers, reading devices, and the like. Network 118 may include, but is not limited to, the Internet, a private network, a virtual private network, a wireless wide area network, a local area network, a metropolitan area network, a telephone network, and the like.

Displays may include, but are not limited to, liquid crystal displays, light emitting diode displays, or E-Ink™ displays manufactured by E Ink Corp. of Cambridge, Massachusetts, USA. The display can be used to display content to the user in the form of text, images, or video. In certain instances, the display can also operate as a touch screen display, which can enable a user to initiate commands or operations by touching the screen with certain fingers or gestures.

FIG. 3 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 2 does not constitute a limitation to the electronic device 1, and may include fewer or more components, or combinations of certain components, or different arrangements of components.

For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) for supplying power to various components. Preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so that through power management The device implements functions such as charge management, discharge management, and power consumption management. The power supply may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components. The electronic device 1 may also include various sensors, bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

Further, the electronic device 1 may also include a network interface, optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which are usually used in the electronic device 1 Establish a communication connection with other electronic devices.

Optionally, the electronic device 1 may further include a user interface. The user interface may be a display (Display) or an input unit (such as a keyboard (Keyboard)). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, Organic Light-Emitting Diode) touch controller, etc. Wherein, the display may also be appropriately called a display screen or a display unit, and is used for displaying information processed in the electronic device 1 and for displaying a visualized user interface.

It should be understood that the embodiments are only for illustration, and are not limited by the structure in terms of the scope of the patent application.

The neural network-based automatic speech recognition program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can realize:

Specifically, for the specific implementation method of the above instruction by the processor 10, reference may be made to the description of the relevant steps in the embodiment corresponding to FIG. 1 , which will not be repeated here.

Further, if the integrated modules/units of the electronic device 1 are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory).

In an embodiment of the present application, a computer-readable storage medium, at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the above-mentioned neural network-based The step of the automatic speech recognition method, concrete method is as follows:

In the several embodiments provided in this application, it should be understood that the disclosed devices, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. Wherein the computer-readable storage medium may be non-volatile or volatile.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software function modules.

Certain embodiments of the present application are described above with reference to block diagrams and flowchart illustrations of systems and methods and/or computer program products according to exemplary embodiments of the application. It should be understood that one or more blocks of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by computer-executable program instructions. Likewise, some blocks in the block diagrams and flowcharts may not necessarily be executed in the order presented, or may not even be executed at all, according to some embodiments of the present application.

These computer-executable program instructions can be loaded into a general-purpose computer, special-purpose computer, processor, or other programmable data processing device to produce a specific machine, so that the instructions executed on the computer, processor, or other programmable data processing device Create a component that implements one or more functions specified in a flowchart block or blocks. These computer program products can also be stored in a computer-readable memory, which can instruct a computer or other programmable data processing apparatus to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture, which includes implementing the process An instructional component of one or more functions specified in a block or blocks of a diagram. For example, the embodiments of the present application may provide a computer program product, which includes a computer-usable medium having computer-readable program code or program instructions embodied therein, and the computer-readable program code is adapted to be executed to realize One or more functions specified in multiple boxes. Computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operational elements or steps to be executed on the computer or other programmable device. The instructions executed above provide elements or steps for implementing the functions specified in the flowchart block or blocks.

Accordingly, blocks in the block diagrams or flow diagrams support combinations of means for performing the specified functions, combinations of elements or steps for performing the specified functions and program instruction means for performing the specified functions. It should also be understood that each block in the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by a dedicated hardware-based computer system that performs the specified functions, elements, or steps, or by dedicated hardware or A combined implementation of computer instructions.

While certain embodiments of the present application have been described in connection with what are presently considered to be the most practical and varied embodiments, it should be understood that the application is not limited to the disclosed embodiments, but is intended to cover Various modifications and equivalent arrangements within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

Therefore, the embodiments should be regarded as exemplary and not restrictive in all points of view, and the scope of the application is defined by the appended claims rather than the foregoing description, and it is intended that the scope of the present application be defined by the appended claims rather than by the foregoing description. All changes within the meaning and range of equivalents of the elements are embraced in this application. Any reference sign in a claim should not be construed as limiting the claim concerned.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application without limitation. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be Make modifications or equivalent replacements without departing from the spirit and scope of the technical solutions of the present application.

Industrial Applicability

Type the industrial applicability description paragraph here.

Sequence Listing Free Content

Type the sequence listing free content description paragraph here.

Claims

A neural network-based automatic speech recognition method applied to electronic equipment, wherein the method includes:

Through the acoustic model and the ngram language model in the ASR recognition process, the audio to be recognized is jointly recognized and processed, and at least two or more initial recognition results are obtained; wherein, each recognition result includes an acoustic model score, an ngram language model score, and the acoustic The sum of the model score and the ngram language model score;

The initial recognition result is transmitted to the rescore process, and the gpt language model in the rescore process is used for scoring processing to obtain the gpt language model score;

Transmitting the gpt language model score to the ASR recognition process, and replacing the ngram language model score in the ASR recognition process;

Sorting the sum of the gpt language model scores and the acoustic model scores in the ASR recognition process, and using the top recognition result among the sorted results as the final recognition result.
The automatic speech recognition method based on neural network as claimed in claim 1, wherein,

The acoustic model and the ngram language model in the ASR recognition process are used to identify and process the audio to be recognized together, and obtain at least two or more initial recognition results, including the following steps:

converting the audio to be identified into audio features;

Obtaining the posterior probability of each frame in the audio feature according to the audio feature;

According to the posterior probability of each frame, viterbi decoding is performed on the wfst graph generated by the ngram language model to generate a lattice graph; and

According to the lattice graph, at least two or more initial recognition results are obtained.
The neural network-based automatic speech recognition method as claimed in claim 2, wherein, said audio to be recognized is converted into audio features, comprising the steps of:

Framing and windowing the audio to be identified to obtain standardized audio; and

The feature extraction of the standard audio is carried out through the MFCC feature extraction algorithm, and the audio features of the audio to be recognized are obtained.
The neural network-based automatic speech recognition method according to claim 2, wherein said obtaining the posterior probability of each frame in said audio feature according to said audio feature comprises the steps of:

extracting the audio features as a sequence of audio feature vectors;

Input the audio feature vector sequence into the pre-trained acoustic model to determine the time boundary of the phoneme state;

According to the time boundary, extract all frames in the time boundary, and take an average value according to the frame length of the speech frame, as the posterior probability of the speech frame.
The automatic speech recognition method based on neural network as claimed in claim 2, wherein,

According to the posterior probability of each frame, viterbi decoding is performed on the wfst graph generated by the ngram language model to generate a lattice graph, comprising the steps of:

Modeling the ngram language model to generate a language model in arpa format;

Use the arpa2fst tool to generate the wfst diagram of the hclg structure;

Construct a wfst search space according to the viterbi, the posterior probability and the wfst graph;

Finding the optimal path with the highest matching probability in the wfst search space to obtain the character recognition result.
The automatic speech recognition method based on neural network as claimed in claim 1, wherein,

The described recognition result is transmitted to the rescore process, and is processed by the gpt language model in the rescore process, and obtains the gpt language model score, including the steps:

Put together sentences to be re-scored into batches of sentences to be re-scored within the preset time;

Perform neural network forward reasoning on batches of sentences to be re-scored through the gpt language model;

Accumulate the posterior probability of each word of the sentence to be re-scored, and output it in logarithmic form, so as to obtain the gpt language model score of the sentence to be re-scored.
A neural network-based automatic speech recognition device, wherein the device includes:

The initial identification result acquisition module is used to jointly identify and process the audio to be identified through the acoustic model and the ngram language model in the ASR identification process, and obtain at least two initial identification results; wherein each identification result includes an acoustic model score, ngram language model score and the sum of the acoustic model score and the ngram language model score;

The gpt language model score acquisition module is used to transmit the recognition result to the rescore process, and perform scoring processing through the gpt language model in the rescore process to obtain the gpt language model score;

A language model score replacement module, configured to transmit the gpt language model score to the ASR recognition process, and replace the ngram language model score in the ASR recognition process;

The final recognition result acquisition module is used to sort the sum of the gpt language model score and the acoustic model score in the ASR recognition process, and use the top-ranked recognition result among the sorted results as the final recognition result.
The automatic speech recognition device based on neural network as claimed in claim 7, wherein,

The initial recognition result acquisition module includes an audio feature conversion module, a posterior probability acquisition module, a lattice graph acquisition module and more than two recognition result acquisition modules, wherein,

The audio feature conversion module is used to convert the audio to be identified into an audio feature;

The posterior probability obtaining module is used to obtain the posterior probability of each frame in the audio feature according to the audio feature;

The lattice graph acquisition module is used to perform viterbi decoding on the wfst graph generated by the ngram language model to generate a lattice graph according to the posterior probability of each frame;

The two or more identification result acquisition modules are configured to acquire at least two or more initial identification results according to the lattice graph.
An electronic device, wherein the electronic device includes:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can perform the steps of the neural network-based automatic speech recognition method, in,

The automatic speech recognition method based on neural network comprises:

Through the acoustic model and the ngram language model in the ASR recognition process, the audio to be recognized is jointly recognized and processed, and at least two or more initial recognition results are obtained; wherein, each recognition result includes an acoustic model score, an ngram language model score, and the acoustic The sum of the model score and the ngram language model score;

The initial recognition result is transmitted to the rescore process, and the gpt language model in the rescore process is used for scoring processing to obtain the gpt language model score;

Transmitting the gpt language model score to the ASR recognition process, and replacing the ngram language model score in the ASR recognition process;

Sorting the sum of the gpt language model scores and the acoustic model scores in the ASR recognition process, and using the top recognition result among the sorted results as the final recognition result.
The electronic device of claim 9, wherein,

The acoustic model and the ngram language model in the ASR recognition process are used to identify and process the audio to be recognized together, and obtain at least two or more initial recognition results, including the following steps:

converting the audio to be identified into audio features;

Obtaining the posterior probability of each frame in the audio feature according to the audio feature;

According to the posterior probability of each frame, viterbi decoding is performed on the wfst graph generated by the ngram language model to generate a lattice graph; and

According to the lattice graph, at least two or more initial recognition results are obtained.
The electronic device according to claim 10, wherein said converting the audio to be recognized into audio features comprises the steps of:

Framing and windowing the audio to be identified to obtain standardized audio; and

The feature extraction of the standard audio is carried out through the MFCC feature extraction algorithm, and the audio features of the audio to be recognized are obtained.
The electronic device according to claim 10, wherein said obtaining the posterior probability of each frame in said audio feature according to said audio feature comprises the following steps:

extracting the audio features as a sequence of audio feature vectors;

Input the audio feature vector sequence into the pre-trained acoustic model to determine the time boundary of the phoneme state;

According to the time boundary, extract all frames in the time boundary, and take an average value according to the frame length of the speech frame, as the posterior probability of the speech frame.
The electronic device of claim 10, wherein,

According to the posterior probability of each frame, viterbi decoding is performed on the wfst graph generated by the ngram language model to generate a lattice graph, comprising the steps of:

Modeling the ngram language model to generate a language model in arpa format;

Use the arpa2fst tool to generate the wfst diagram of the hclg structure;

Construct a wfst search space according to the viterbi, the posterior probability and the wfst graph;

Finding the optimal path with the highest matching probability in the wfst search space to obtain the character recognition result.
The electronic device of claim 9, wherein,

The described recognition result is transmitted to the rescore process, and is processed by the gpt language model in the rescore process, and obtains the gpt language model score, including the steps:

Put together sentences to be re-scored into batches of sentences to be re-scored within the preset time;

Perform neural network forward reasoning on batches of sentences to be re-scored through the gpt language model;

Accumulate the posterior probability of each word of the sentence to be re-scored, and output it in logarithmic form, so as to obtain the gpt language model score of the sentence to be re-scored.
A computer-readable storage medium, storing a computer program, wherein, when the computer program is executed by a processor, an automatic speech recognition method based on a neural network is implemented, wherein,

Through the acoustic model and the ngram language model in the ASR recognition process, the audio to be recognized is jointly recognized and processed, and at least two or more initial recognition results are obtained; wherein, each recognition result includes an acoustic model score, an ngram language model score, and the acoustic The sum of the model score and the ngram language model score;

The initial recognition result is transmitted to the rescore process, and the gpt language model in the rescore process is used for scoring processing to obtain the gpt language model score;

Transmitting the gpt language model score to the ASR recognition process, and replacing the ngram language model score in the ASR recognition process;

Sorting the sum of the gpt language model scores and the acoustic model scores in the ASR recognition process, and using the top recognition result among the sorted results as the final recognition result.
The computer readable storage medium of claim 15, wherein:

The acoustic model and the ngram language model in the ASR recognition process are used to identify and process the audio to be recognized together, and obtain at least two or more initial recognition results, including the following steps:

converting the audio to be identified into audio features;

Obtaining the posterior probability of each frame in the audio feature according to the audio feature;

According to the posterior probability of each frame, viterbi decoding is performed on the wfst graph generated by the ngram language model to generate a lattice graph; and

According to the lattice graph, at least two or more initial recognition results are obtained.
The computer-readable storage medium according to claim 16, wherein said converting the audio to be recognized into audio features comprises the steps of:

Framing and windowing the audio to be identified to obtain standardized audio; and

The feature extraction of the standard audio is carried out through the MFCC feature extraction algorithm, and the audio features of the audio to be recognized are obtained.
The computer-readable storage medium according to claim 16, wherein said acquiring the posterior probability of each frame in said audio feature according to said audio feature comprises the following steps:

extracting the audio features as a sequence of audio feature vectors;

Input the audio feature vector sequence into the pre-trained acoustic model to determine the time boundary of the phoneme state;

According to the time boundary, extract all frames in the time boundary, and take an average value according to the frame length of the speech frame, as the posterior probability of the speech frame.
The computer readable storage medium of claim 16, wherein:

According to the posterior probability of each frame, viterbi decoding is performed on the wfst graph generated by the ngram language model to generate a lattice graph, comprising the steps of:

Modeling the ngram language model to generate a language model in arpa format;

Use the arpa2fst tool to generate the wfst diagram of the hclg structure;

Construct a wfst search space according to the viterbi, the posterior probability and the wfst graph;

Finding the optimal path with the highest matching probability in the wfst search space to obtain the character recognition result.
The computer readable storage medium of claim 15, wherein:

The described recognition result is transmitted to the rescore process, and is processed by the gpt language model in the rescore process, and obtains the gpt language model score, including the steps:

Put together sentences to be re-scored into batches of sentences to be re-scored within the preset time;

Perform neural network forward reasoning on batches of sentences to be re-scored through the gpt language model;

Accumulate the posterior probability of each word of the sentence to be re-scored, and output it in logarithmic form, so as to obtain the gpt language model score of the sentence to be re-scored.