CN114360510A - Voice recognition method and related device - Google Patents

Abstract

The embodiment of the application discloses a voice recognition method and a related device, which at least relate to a voice recognition technology in artificial intelligence, and the voice data to be recognized is used as input data of a time delay neural network in an acoustic model. When the syllable is identified through the output layer, the syllable information before and after the voice frame can be combined, and the syllable of the voice frame is subjected to auxiliary judgment based on the pronunciation rule so as to output more accurate syllable probability distribution. Moreover, because syllables are generally composed of one or more phonemes, the method has higher fault-tolerant capability, not only obtains more accurate voice recognition results based on syllable probability distribution, but also has low requirements on the quality of voice data to be recognized, and effectively expands the application scenes of the voice recognition technology.

Description

A kind of speech recognition method and related device

technical field

The present application relates to the field of speech recognition, and in particular, to a speech recognition method and related apparatus.

Background technique

Voice recognition technology can provide users with voice content recognition services. This technology can be applied to various scenarios, such as voice-to-text, voice wake-up, and human-computer interaction. In a specific implementation, the acoustic features of the speech data to be recognized may be extracted through an acoustic model, and a corresponding speech recognition result may be determined based on the acoustic features.

The related art mainly uses phone as the modeling unit of the acoustic model. The phone is the smallest phonetic unit divided according to the natural attributes of the voice. It is analyzed according to the pronunciation action in the syllable, and an action constitutes a phoneme.

However, the modeling granularity of phonemes is relatively fine, and this fine-grained speech recognition method has high requirements on the quality of the recognized speech data, and slight pronunciation errors may directly affect the recognition results. As a result, it is difficult for speech recognition technology to adapt to some speech recognition scenarios.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the present application provides a speech recognition method and a related device, which are used to improve the accuracy of speech recognition results and expand the usage scenarios of speech recognition technology.

On the one hand, the embodiment of the present application provides a speech recognition method, the method includes:

Acquiring an acoustic model and the speech data to be recognized, the acoustic model includes a time-delay neural network, and the output layer of the time-delay neural network includes acoustic modeling units corresponding to a plurality of syllables respectively;

The speech data is used as the input data of the time-delay neural network, and the syllable probability distribution corresponding to the speech frames included in the speech data is determined through the time-delay neural network, and the syllable probability distribution is used to identify the speech the probability that the frame corresponds to the plurality of syllables respectively;

The speech recognition result corresponding to the speech data is determined according to the syllable probability distribution.

On the other hand, an embodiment of the present application provides a speech recognition device, the device includes an acquisition unit, a syllable probability distribution determination unit, and a speech recognition result determination unit;

The acquisition unit is used to acquire an acoustic model and the speech data to be recognized, the acoustic model includes a time-delay neural network, and the output layer of the time-delay neural network includes an acoustic modeling unit corresponding to a plurality of syllables respectively;

The syllable probability distribution determining unit is configured to use the speech data as the input data of the time-delay neural network, and determine the syllable probability distributions corresponding to the speech frames included in the speech data through the time-delay neural network. The syllable probability distribution is used to identify the probability that the speech frame corresponds to the multiple syllables respectively;

The speech recognition result determination unit is configured to determine the speech recognition result corresponding to the speech data according to the syllable probability distribution.

On the other hand, an embodiment of the present application provides a computer device, the device includes a processor and a memory:

the memory is used to store program code and transmit the program code to the processor;

The processor is configured to execute the method described in the above aspects according to the instructions in the program code.

On the other hand, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the foregoing aspects.

On the other hand, an embodiment of the present application provides a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method described in the above aspects.

It can be seen from the above technical solutions that, for the speech data to be recognized, an acoustic model including a time-delay neural network is obtained. Taking the speech data as the input data of the time-delay neural network, since the output layer of the time-delay neural network includes acoustic modeling units corresponding to multiple syllables, the time-delay neural network can use syllables as the identification granularity, and Each acoustic modeling unit of the output layer obtains the probability distribution of the syllables corresponding to the speech frames included in the speech data. In the process of speech feature extraction and transmission to the output layer, the time-delay neural network will carry the rich context information of the speech frame in the speech data. When identifying syllables through the output layer, the syllables of the speech frame can be judged based on the pronunciation rules based on the information of the syllables before and after the speech frame, so as to output a more accurate probability distribution of syllables. Moreover, since a syllable is generally composed of one or more phonemes, it has higher error tolerance. Even if the pronunciation error of an individual phoneme occurs in a syllable in the speech data, the impact on the recognition result of the entire syllable is smaller than that of the phoneme. Therefore, even if the quality of the speech data to be recognized is not high, a more accurate determination result of speech recognition can be obtained based on the probability distribution of syllables, which effectively expands the applicable scenarios of the speech recognition technology.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

1 is a schematic diagram of an application scenario of a speech recognition method provided by an embodiment of the present application;

2 is a flowchart of a speech recognition method provided by an embodiment of the present application;

3 is a schematic diagram of a TDNN provided by an embodiment of the present application;

4 is a schematic diagram of an acoustic model provided by an embodiment of the present application;

5 is a schematic diagram of an embodiment of an application scenario of a speech recognition system provided by an embodiment of the present application;

6 is a schematic diagram of an acoustic model provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a speech recognition device according to an embodiment of the present application;

FIG. 8 is a structural diagram of a terminal device provided by an embodiment of the present application;

FIG. 9 is a structural diagram of a server provided by an embodiment of the present application.

Detailed ways

The embodiments of the present application will be described below with reference to the accompanying drawings.

When recognizing speech data, in the related art, the phoneme is used as the modeling unit of the acoustic model. For example, the phoneme sequence corresponding to the keyword "hello" is "n iy3 hh aw3", only the phoneme sequence "n iy3 hh aw3" is recognized in the speech data. iy3 hhaw3" to recognize the keyword "hello". Because the modeling granularity of phoneme is too fine, it has higher requirements on the quality of speech data. If the pronunciation of one of the keywords is not standard, the keyword recognition will fail, resulting in a low accuracy of the speech recognition result. Even in some complex speech recognition scenarios such as noisy background sounds and non-standard user speech, the speech The robustness of recognition is low, making speech recognition technology less applicable to scenarios.

Based on this, the embodiments of the present application provide a speech recognition method, which not only improves the accuracy of speech recognition results, but also has lower requirements on the quality of speech data, effectively expanding the applicable scenarios of speech recognition technology.

The speech recognition methods provided in the embodiments of the present application are implemented based on artificial intelligence. Artificial intelligence (AI) uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use it Knowledge of theories, methods, techniques and applied systems for obtaining optimal results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning, autonomous driving, and smart transportation.

In the embodiments of the present application, the artificial intelligence technology mainly involved includes the above-mentioned voice processing technology and other directions.

The speech recognition method provided in this application can be applied to speech recognition devices with data processing capabilities, such as terminal devices and servers. Among them, the terminal devices involved in this application may specifically be smart phones, tablet computers, notebook computers, PDAs, personal computers, smart watches, smart speakers, vehicle-mounted devices, wearable devices, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. , but not limited to this. The server involved in this application may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, Cloud servers for basic cloud computing services such as cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The terminal device and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application. The number of servers and terminal devices is also not limited.

The aforementioned speech recognition device may have speech processing technology, and the key technologies of speech technology include automatic speech recognition technology, speech synthesis technology, and voiceprint recognition technology. Making computers able to hear, see, speak, and feel is the development direction of human-computer interaction in the future, and voice will become one of the most promising human-computer interaction methods in the future.

In the speech recognition method provided by the embodiment of the present application, the adopted artificial intelligence model mainly involves the application of speech recognition technology, and the speech recognition result corresponding to the speech data to be recognized is determined by the acoustic model.

In order to facilitate the understanding of the technical solutions of the present application, the speech recognition method provided by the embodiments of the present application is described below in combination with an actual application scenario, using a terminal device as a speech recognition device.

Referring to FIG. 1 , this figure is a schematic diagram of an application scenario of a speech recognition method provided by an embodiment of the present application. In the application scenario shown in FIG. 1 , the terminal device is a smart speaker 100 , which is used to recognize the keyword "hello" in the speech data to be recognized.

The smart speaker 100 acquires an acoustic model. The acoustic model includes a time-delay neural network, and the output layer of the time-delay neural network includes acoustic modeling units corresponding to multiple syllables respectively. In the embodiment of the present application, the phoneme is no longer used as the acoustic modeling unit, but the syllable is used as the acoustic modeling unit. Phonemes can also be self-syllabic, that is, a syllable can include one or more phonemes. Taking the keyword "hello" as an example, the corresponding syllable sequence is "ni3 hao3", and the phoneme sequence is "niy3hh aw3". Therefore, compared with phonemes, syllables have higher fault tolerance, even if speech data Pronunciation errors of individual phonemes appear in one syllable, and the impact on the recognition result of the whole syllable is smaller than that of phonemes, so the requirements for the quality of the speech data to be recognized are lowered.

After the smart speaker 100 obtains the speech data to be recognized, it inputs the speech data into the time-delay neural network. The time-delay neural network includes multiple layers of networks. The extended neural network has a wider contextual vision and can capture wider contextual information. The contextual information can reflect the relevant information of the syllables before and after the speech frame in the speech data. Therefore, when identifying syllables through the output layer, it can be combined with The information of the syllables before and after the speech frame is used to assist in the judgment of the syllable to which the speech frame belongs based on the pronunciation rules, so that the output probability distribution of the syllables whose syllable sequence is "ni3 hao3" is more accurate.

The smart speaker 100 determines a more accurate speech recognition result corresponding to the speech data according to a more accurate syllable probability distribution. As a result, not only the accuracy of the speech recognition result is improved, but also the requirement for the quality of speech data is lowered, which effectively expands the applicable scenarios of speech recognition technology.

In the following, a speech recognition method provided by an embodiment of the present application is introduced by using a server as a speech recognition device with reference to the accompanying drawings.

Referring to FIG. 2 , which is a flowchart of a speech recognition method provided by an embodiment of the present application. As shown in Figure 2, the speech recognition method includes the following steps:

S201: Acquire an acoustic model and speech data to be recognized.

In the related art, the Gaussian Mixed Model (GMM)-Hidden Markov Model (HMM) with phoneme as the modeling unit, that is, the GMM-HMM model has become the automatic speech recognition (Automatic Speech Recognition, The mainstream system for ASR) tasks. However, with the accumulation of labeled corpus and the rapid growth of computer computing power, the effect of many newly proposed deep neural network models has greatly surpassed the GMM-HMM model, and has been further improved. Due to the powerful modeling capabilities of these deep neural networks, the output labels of acoustic models also no longer need to be subdivided into phonemes. A lot of research work has found that acoustic models can also use larger modeling granularity, and in most cases can achieve better results.

Based on this, in the embodiments of the present application, phonemes are no longer used as acoustic modeling units, but syllables are used as acoustic modeling units. Among them, syllables are mainly composed of initials, finals and tones, and each syllable can be decomposed into one or more phonemes. See Table 1, and take the keywords "hello" and "world" as examples to illustrate the difference between syllables and phonemes.

Table 1 Differences between syllables and phonemes

Key words syllable sequence phoneme sequence Hello ni3 hao3 n iy3 hh aw3 world shi4 jie4 sh iy4 j iy4 eh4

Compared with phonemes, syllables can carry more contextual information, thereby improving the contextual learning ability of acoustic models and improving the coverage of speech keywords. In addition, syllables are more in line with the human speaking process and are easier to understand. Moreover, since a syllable is generally composed of one or more phonemes, it has higher error tolerance. Even if the pronunciation error of an individual phoneme occurs in a syllable in the speech data, the impact on the recognition result of the entire syllable is smaller than that of the phoneme. As a result, the requirements for the quality of the speech data are reduced, and the applicable scenarios of the speech recognition technology are effectively expanded.

The Acoustic Model (AM) includes a Time Delay Neural Network (TDNN), which will be described later. Voice data, also known as sound files, is data recorded by voice. For example, the sound of waking up a smart speaker can be called voice data.

It can be understood that, in the specific implementation of this application, it involves user-related data such as voice data. When the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain user permission or consent, and the collection of relevant data. , use and processing need to comply with relevant laws, regulations and standards of relevant countries and regions.

It should be noted that the acoustic model and the voice data to be recognized can be obtained at the same time, the voice data to be recognized can also be obtained after the acoustic model is obtained in advance, or the acoustic model can be obtained after the voice data to be recognized is obtained. There is no specific limitation on this.

S202: Use the speech data as input data of the time-delay neural network, and determine the probability distribution of syllables corresponding to speech frames included in the speech data through the time-delay neural network.

The acoustic model can convert the speech input into an acoustically represented syllable output. In this embodiment of the present application, the probability distribution of the syllable to which it belongs is calculated for each speech frame. The acoustic model includes a TDNN, as shown in FIG. 3 , which is a schematic diagram of a TDNN provided by an embodiment of the present application.

As shown in Figure 3, TDNN includes multiple layers of networks, and each layer of the network has a strong ability to abstract speech features. Moreover, TDNN has a wider contextual view, can capture wider contextual information, and has a stronger ability to model speech timing dependence information.

In the embodiment of the present application, the output layer of the TDNN includes acoustic modeling units corresponding to multiple syllables. After the speech data is input to the TDNN, the TDNN can use the syllable as the recognition granularity to determine the corresponding speech frames included in the speech data. Syllable probability distribution. The syllable probability distribution is used to identify the probability that the speech frame corresponds to a plurality of syllables respectively. As shown in Figure 3, in the output layer of TDNN, the probability corresponding to each syllable can be output. It should be noted that, FIG. 3 only shows 4 syllables as an example, but is not limited to 4 syllables. For example, all syllables required for Chinese can be counted, and all syllables can be represented by the output layer, so as to determine the recognition result corresponding to the speech data to be recognized input by the Chinese user.

Since the time-delay neural network will carry the rich context information of the speech frame in the speech data in the process of speech feature extraction and transmission to the output layer, the context information can reflect the relevant information of the syllables before and after the speech frame in the speech data. When identifying syllables through the output layer, it can combine the information of the syllables before and after the speech frame to assist in the judgment of the syllable to which the speech frame belongs based on the pronunciation rules, so as to output a more accurate probability distribution of syllables.

S203: Determine a speech recognition result corresponding to the speech data according to the syllable probability distribution.

The syllable probability distribution is used to identify the probability that the speech frame corresponds to a plurality of syllables respectively. For example, the output layer of TDNN will output the probability corresponding to 100 syllables, then the probability distribution of syllables obtained by TDNN is a 1×100 row vector, each element in the row vector represents the probability corresponding to the syllable, and the corresponding The probability sum is 1. It can be understood that the greater the probability of a certain syllable, the greater the probability that the actual pronunciation of the speech frame is the same as the pronunciation of the syllable. Continuing to take the previous example, if the probability distribution of syllables obtained by speech frame A is [0.1, 0.8, 0.1, 0, ..., 0] (95 0s are omitted in the middle), then the actual pronunciation of speech frame A is the first three The pronunciation of syllable composition is more likely, or the actual pronunciation of speech frame A is the pronunciation of the second syllable. This application does not specifically limit this, and those skilled in the art can perform according to actual application scenarios set up.

After obtaining the syllable probability distribution corresponding to each speech frame in the speech data, the speech recognition result corresponding to the speech data may be further determined according to the syllable probability distribution corresponding to the multiple speech frames respectively. For example, after the user inputs a piece of speech, it is regarded as the speech data to be recognized, and the probability distribution of the syllables corresponding to the speech frames included in the speech data can be obtained through the TDNN, and then the speech recognition result can be determined.

It can be seen from the above technical solutions that, for the speech data to be recognized, an acoustic model including a time-delay neural network is obtained. Taking the speech data as the input data of the time-delay neural network, since the output layer of the time-delay neural network includes acoustic modeling units corresponding to multiple syllables, the time-delay neural network can use syllables as the identification granularity, and Each acoustic modeling unit of the output layer obtains the probability distribution of the syllables corresponding to the speech frames included in the speech data. In the process of speech feature extraction and transmission to the output layer, the time-delay neural network will carry the rich context information of the speech frame in the speech data. When identifying syllables through the output layer, the syllables of the speech frame can be judged based on the pronunciation rules based on the information of the syllables before and after the speech frame, so as to output a more accurate probability distribution of syllables. Moreover, since a syllable is generally composed of one or more phonemes, it has higher error tolerance. Even if the pronunciation error of an individual phoneme occurs in a syllable in the speech data, the impact on the recognition result of the entire syllable is smaller than that of the phoneme. Therefore, even if the quality of the speech data to be recognized is not high, a more accurate determination result of speech recognition can be obtained based on the probability distribution of syllables, which effectively expands the applicable scenarios of the speech recognition technology.

It should be noted that the present application can be applied to voice keyword spotting (Keyword spotting, KWS) tasks such as wake-up interaction and voice keyword detection of audio and video files in smart devices. Due to the limitations of labeled corpus and computing power, Gaussian Mixed Mode (GMM), Deep Neural Networks (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (Recurrent Neural Network) are used. Network, RNN) and other acoustic models are not strong enough, resulting in poor performance in smart devices. In order to cope with the limited computing resources of smart devices and the processing requirements of massive audio and video files, not only the speed of keyword detection is required to be fast enough, but also the detection results are expected to be accurate enough.

Based on this, refer to FIG. 4 , which is a schematic diagram of an acoustic model provided by an embodiment of the present application. As shown in Figure 4, this acoustic model includes not only the TDNN but also the decoder. Since TDNN has a wider contextual view and can capture wider contextual information, it can further increase the ability of acoustic models to model speech timing information, that is, TDNN can process timing information more efficiently, thereby improving keyword detection. accuracy. The decoder is configured to determine whether the speech data includes the speech recognition result of the keyword according to the syllable probability distribution, so as to determine whether to wake up the corresponding terminal device according to the speech recognition result.

For the convenience of description, the following takes the wake-up interaction scenario of the smart device as an example for description, see S2031-S2033:

S2031: Determine a keyword corresponding to the wake-up scene according to the wake-up scene corresponding to the voice data.

It can be understood that, due to different wake-up scenarios, the corresponding keywords may be different, and those skilled in the art can set them according to actual needs. For example, the keyword corresponding to the smart home wake-up scene is "open XX (home equipment)", and the keyword corresponding to the virtual assistant wake-up scene is "XXX (virtual assistant name)", etc. According to the setting of the keyword by those skilled in the art, After acquiring the speech data to be recognized, the keyword corresponding to the wake-up scene is determined according to the corresponding wake-up scene.

It should be noted that, S2031 may be executed after obtaining the probability distribution of syllables, or may be executed after obtaining the speech data to be recognized, which is not specifically limited in this application.

S2032: According to the probability distribution of syllables, determine, by the decoder, a speech recognition result for identifying whether the speech data includes a keyword.

Among them, the decoder is used to decompress the syllable probability distribution into the speech recognition result. Therefore, after obtaining the syllable probability distribution through TDNN, not only can the decoder determine the speech recognition result corresponding to the speech data according to the syllable probability distribution, but also can decode the syllable probability distribution. The processor determines a speech recognition result for identifying whether a keyword is included in the speech data.

As a possible implementation manner, a given keyword may be decoded by a weighted finite state converter (Weighted Finite State Transducer, WFST)-based decoder, and the speech recognition result of whether the keyword is included may be directly output. Among them, WFST provides a unified framework, which can integrate information from acoustic models, pronunciation dictionaries and language models in a unified way to find the optimal path in a huge search space, thereby improving the search speed of the model.

As a possible implementation manner, in the decoding process, the Lattice static decoding strategy based on WFST can be further adopted to find the optimal path in the search space and improve the search speed of the model.

As a possible implementation, in the decoding process, the decoding speed can also be improved by methods such as frame pruning. For example, the probability distribution of the generated syllables is a 1×100 vector, the first three probabilities are 0.1, 0.8 and 0.1 respectively, and the last 97 probabilities are all 0. The last 97 are pruned by frame pruning, and the 1× A vector of 100 becomes a 1×3 vector, so that only the 1×3 vector needs to be decoded, thereby improving the decoding speed.

S2033: If the voice recognition result indicates that the voice data includes a keyword, wake up the terminal device corresponding to the wake-up scene.

It can be seen from the above technical solutions that using syllables instead of phonemes as the modeling unit of the acoustic model, the granularity becomes coarser, which not only reduces the memory occupied, but also requires less computing resources. For phonetic keywords, syllable coverage is higher, error tolerance is higher, and accuracy is higher. At the same time, for the same length of speech data to be recognized, it is faster to detect keywords based on syllables. When dealing with limited computing resource constraints and the processing requirements of massive audio and video files, since the acoustic model includes TDNN and decoder, TDNN can To further increase the ability of the acoustic model to model speech timing information, the decoder can determine whether the speech data includes the speech recognition results of keywords, thereby improving the accuracy of keyword detection. Therefore, the acoustic model provided by the embodiment of the present application not only requires faster keyword detection, but also higher accuracy of the detection result, a better balance between its performance and speed, and a better effect in smart devices .

Regarding the aforementioned S2032, the embodiment of the present application does not specifically limit the manner in which the speech recognition result is determined by the decoder. For example, the speech recognition method for identifying whether the speech data includes a keyword may be determined by the decoder according to the probability distribution of syllables and the matching vocabulary. result.

For example, in the virtual assistant wake-up scenario, the corresponding keyword is "hello world", and a corresponding keyword table is established according to the keyword. After the user inputs a piece of speech, the speech is used as the speech data to be recognized, and the probability distribution of syllables is determined by the acoustic model provided in the embodiment of the present application, and the decoder searches for the keyword table corresponding to "Hello World", and determines the probability distribution of the syllables. Whether the voice data input by the user includes "Hello World". If the voice data includes "Hello World", the decoder may output a result of "including keywords", and further, the virtual assistant may be awakened according to the voice recognition result to provide services for the user, as shown in S2033. The terminal device corresponding to the wake-up scene may be a terminal device such as a smart speaker, a virtual assistant, and smart furniture, which is not specifically limited in this application.

Among them, the keyword table is established with the keyword as the core, including the possible corresponding pronunciation of the keyword. The embodiments of the present application do not specifically limit the method of determining the keyword table. For example, a matching vocabulary for keywords may be constructed with syllables as the granularity according to similar syllables and/or polysyllabic syllables and syllables corresponding to keywords. Enriching the matching vocabulary through similar syllables and/or polysyllabic syllables can effectively improve the coverage of keywords, thereby avoiding missed wakeups and improving the user's sense of use. The similar syllables and the polysyllabic syllables are respectively described below.

Similar syllables: Similar syllables are similar syllables corresponding to the syllables included in the keyword according to the degree of pronunciation similarity. For example, the syllable sequence of the keyword "hello" is "ni3 hao3", when the keyword is read out loud, "you" " may be pronounced once instead of three, so the syllable sequence "ni1 hao3" can be added to the list of collocation keywords as similar syllables.

Polysyllabic syllable: A polysyllabic word is a word with more than one pronunciation. If a keyword has a polysyllabic character, determine the polysyllabic syllable corresponding to the polysyllabic character, and all its pronunciations should be included in the keyword table to prevent leakage caused by the user's accent. wake. Continuing to take the keyword "hello" as an example, the syllable sequence "ni3 hao4" can be added to the collocation keyword table as a similar syllable.

As a possible implementation method, some mispronounced pronunciations will also be added to the keyword table. For example, the keyword "傥" is easily read as "dang (three tones)", so the syllable "dang3" will be included in the key word list. The word "傥" corresponds to the keyword table.

As a possible implementation, TDNN can not only combine the information of the syllables before and after the speech frame, but also combine the characteristics of the speech frame corresponding to at least one frame before and after the i-th speech frame in the speech data to further improve the accuracy of the syllable probability distribution. . Taking the TDNN including N layers of feature extraction layers as an example, for the i-th speech frame in the speech frames included in the speech data, S202 can be implemented through S2021-S2023, as follows:

S2021 : According to the output feature of the j-1th layer of feature extraction layer for the i-th speech frame, determine the speech frame feature of the i-th speech frame through the j-th layer of feature extraction layer.

Each layer of network included in TDNN can be called a feature extraction layer, and each layer of feature extraction layer will perform feature extraction for each frame of speech frame included in the speech data to be recognized. In the following, for the ith frame of speech frame, any two feature extraction layers in the N-layer feature extraction layer, that is, the j-1-th feature extraction layer and the j-th feature extraction layer, are taken as examples for description. Among them, j∈N.

For the i-th speech frame, the feature extraction layer of the j-1th layer will perform feature extraction, so as to output the output features for the i-th speech frame, and then input it to the j-th layer of feature extraction layer, and the j-th layer of feature extraction layer Feature extraction is performed to determine the speech frame features of the i-th speech frame.

S2022: Determine the output feature of the i-th speech frame in the j-th feature extraction layer by using the speech frame feature and the speech frame feature corresponding to at least one frame before and after the i-th speech frame in the speech data.

In order to further improve the context learning capability of the acoustic model, after determining the speech frame features of the i-th speech frame, not only the syllable information before and after the speech frame can be combined, but also the pre- and post-syllabic information of the i-th speech frame in the speech data at least The feature of the speech frame corresponding to one frame, that is, further context information is carried by the context information of the speech frame, to determine the output feature of the i-th speech frame in the j-th feature extraction layer.

Continue to refer to Fig. 3, in this figure, each upper node is connected to the lower three nodes, each node corresponds to a frame of voice frame feature, S2022 can be through the voice frame feature, the i-th frame of voice frame in the voice data corresponding to the previous frame The speech frame feature and the speech frame feature corresponding to the next frame determine the output feature of the i-th speech frame in the j-th feature extraction layer. It can be understood that the speech frame features corresponding to the first three frames of the current frame and the speech frame features corresponding to the next frame can also be considered, and those skilled in the art can set according to actual needs, which are not specifically limited in this application.

As a possible implementation, the N-layer feature extraction layers included in the TDNN can implement the above S2021-S2022, until all the feature extraction layers perform feature extraction for the i-th speech frame, and then determine the i-th frame corresponding to the speech frame. Syllable probability distribution.

S2023: Determine the probability distribution of syllables corresponding to the i-th speech frame according to the output features of the i-th speech frame in the j-th layer feature extraction layer.

Thus, after inputting the speech data to be recognized into the TDNN, the probability distribution of the syllables corresponding to the i-th speech frame can be obtained, thereby obtaining the probability distribution of the syllables corresponding to the speech frames included in the speech data, and then determining the speech recognition corresponding to the speech data. result.

As a possible implementation manner, the following describes the training process of the acoustic model provided by the embodiment of the present application.

S301: Acquire a voice sample belonging to the same language as the voice data.

Speech data belonging to the same language have basically the same pronunciation rules, and the corresponding syllables are basically the same. For example, the syllables corresponding to Chinese are different from the syllables corresponding to English, so speech samples can be obtained based on the language to which the speech data belongs, and the acoustic model can be trained.

S302: According to the speech sample as the input data of the initial acoustic model, obtain the prediction result through the initial time delay neural network included in the initial acoustic model.

Input the speech samples into the initial time delay neural network included in the initial acoustic model to obtain the prediction result. Compared with speech sample labels, prediction results can include two types, correct prediction results and wrong prediction results. It should be noted that the correct prediction result is that the error between the prediction result and the corresponding sample label is less than the threshold, and the wrong prediction result is that the error between the prediction result and the corresponding sample label is greater than or equal to the threshold. The embodiments of the present application do not specifically limit the threshold, and those skilled in the art can determine it according to actual needs.

S303: Determine a loss function according to the prediction result and the sample label of the speech sample.

Among them, the loss function includes a guide weight, which is used to improve the influence of the correct prediction result on the identification path in the initial delay neural network, and reduce the influence of the wrong prediction result on the identification path in the initial delay neural network.

As a possible implementation manner, there may be multiple speech samples input into the initial TDNN. For example, if there are 100 speech samples, 20 can be used as a batch and input into the initial TDNN in 5 times to determine the loss function. If 20 speech samples are input into the initial TDNN, 5 correct prediction results and 15 wrong prediction results are obtained. When determining the loss function, the influence of the recognition paths where the 5 correct prediction results are located is increased by guiding the weight, and the influence of the recognition paths where the 15 incorrect prediction results are located is reduced, so that the subsequent training can be more trustworthy. , so that the influence of the recognition path corresponding to the correct prediction result is getting higher and higher, and the influence of the recognition path corresponding to the wrong prediction result is getting lower and lower, so as to avoid falling into a local optimum.

Among them, the initial TDNN includes a multi-layer network, and one layer of the network includes multiple nodes. Each node obtains different output characteristics from the previous layer of network, so the learned content is different, and finally a variety of identification paths are formed. As shown in Figure 3, if the TDNN model is the initial TDNN model, the first node of the second layer can obtain different output features from the first node, the second node and the third node from the left of the first layer, so that Three paths are formed, continue up to the last layer, and then form multiple identification paths between nodes from the first layer to the last layer. The final prediction results corresponding to different identification paths may be different, and then according to the prediction results and samples Labels, different recognition paths have different influences, such as different weights corresponding to different recognition paths.

As a possible implementation, the loss function of Lattic-free Maximum Mutual Information (LF-MMI) can be added to the frame-level cross-entropy (CE) loss function, that is, using The bi-phone-based LF-MMI model replaces the mono-phone-based nnet3 model and is used to guide the error transfer of the initial TDNN model. Compared with only the CE loss function, this method can make the training faster and more stable. The robustness and coverage of keyword detection are further improved by TDNN. The LF-MMI model introduces blanks to absorb uncertain boundaries, that is, to absorb coincident and meaningless outputs, such as "ni3 hao3 hao3" or "ni3ni3 hao3", etc.

S304: Adjust the model parameters of the initial time-delay neural network in the initial acoustic model according to the loss function to obtain the acoustic model.

The model parameters of the initial delay neural network in the initial acoustic model are adjusted according to the loss function, so as to obtain the acoustic model including the adjusted delay neural network.

As a possible implementation, if the initial acoustic model includes the initial time delay neural network and the initial decoder, the model parameters of the initial time delay neural network and the initial decoder in the initial acoustic model can be adjusted according to the loss function, and the parameters of the initial time delay neural network and the initial decoder can be adjusted according to the loss function. Acoustic models of good time-delay neural networks and decoders.

Therefore, through the guidance weight included in the loss function, the influence of the correct prediction result in the identification path in the initial delay neural network is improved, and the influence of the wrong prediction result in the identification path in the initial time delay neural network is reduced, so as to realize the difference training, The recognition probability of the correct prediction result is made larger, and other probabilities are as small as possible, which not only improves the recognition rate of the trained acoustic model, but also makes the training of the acoustic model faster and more stable.

As a possible implementation, if the number of speech samples used for training is small, the accuracy of the acoustic model may be affected. Therefore, for scenarios where the number of speech samples is insufficient, such as small languages, data augmentation can be used. Increase the number of speech samples. details as follows:

S3031: Acquire to-be-processed voice samples belonging to the same language as the voice data.

S3032: Perform data augmentation according to the speech samples to be processed to obtain augmented samples.

The sample labels of the augmented samples are determined based on the sample labels of the corresponding speech samples to be processed. The embodiments of the present application do not specifically limit the manner of data augmentation, and the following three manners are used as examples for description.

Mode 1: Through the voice coding method, augmented samples with different sampling rates, different channels, and different coding formats are generated for the voice samples to be processed.

Method 2: Through the speech rate and volume perturbation method, a multiplicative factor is randomly generated in a preset interval to change the speed and volume of the speech sample to be processed, and an augmented sample is obtained. The preset interval may be between 0.9 and 1.1, and those skilled in the art may add it according to actual needs, which is not specifically limited in this application.

Method 3: The augmented sample is obtained by adding background music and noise, that is, using various background music and noise of various scenes, and adding it to a relatively clean speech sample.

It should be noted that the augmented samples may be generated by combining one or more of the above three methods.

S3033: Obtain a speech sample according to the to-be-processed speech sample and the augmented sample.

It can be seen from the above technical solutions that due to the great differences between voices, such as the sampling rate, coding rate, channel difference of audio and video, as well as the speaker's speech rate, intonation, environmental interference, etc., these will lead to speech. The difficulty of keyword detection has increased dramatically. If you want to collect a large amount of voice data of a certain scene and mark it, and then obtain a large number of voice samples, it is a very expensive thing, and even in some low-resource and small language scenarios, the number of voice samples is even less. Therefore, in a scenario with a small number of speech samples, by expanding the number of speech samples by means of S3031-S3033, the robustness of the acoustic model in different scenarios can be improved in the case of limited speech samples.

When training an acoustic model, each frame needs to have an output as a label. However, after data augmentation, the speech frames of the augmented samples do not correspond to the sample labels. Based on this, the speech frames of the augmented samples and the sample labels can be aligned one-to-one, that is, a model is used to convert the output sequence One-to-one correspondence with the input features, as follows:

The sample label of the to-be-processed speech sample corresponding to the augmented sample is used as the to-be-determined label of the augmented sample;

The label alignment process is performed according to the speech frame of the augmented sample and the undetermined label, and the sample label of the augmented sample is obtained.

In the related art, it is usually realized by GMM. The score of each frame on the potential label is calculated by GMM, and the final alignment label is output in combination with dynamic programming. However, with more labeled corpus and greater computing power, the effect of the deep neural network model has gradually surpassed the GMM model. In order to obtain better results, the ChainModel-based acoustic model can be used to replace the GMM model, which makes the output label sequence more accurate.

Next, with reference to FIG. 5 and FIG. 6 , for the scenario of keyword recognition, the speech recognition method provided by the embodiment of the present application will be described by taking an acoustic model including a TDNN and a decoder as an example.

Referring to FIG. 5 , this figure is a schematic diagram of an embodiment of an application scenario of a speech recognition system provided by an embodiment of the present application. The speech recognition system includes a feature extraction module 501, a TDNN module 502, a decoding module 503 and a vocabulary generation module 504, which will be described below.

The feature extraction module 501 converts the continuous speech data to be recognized into discrete vector representations through digital signal processing algorithms, and these vectors can effectively represent the speech features corresponding to the speech data to be recognized, thereby facilitating subsequent speech tasks.

As a possible implementation manner, in the process of speech feature extraction, fast fixed-point processing can also be performed, that is, integers are used to simulate floating-point numbers. Specifically, through linear mapping, floating-point values are mapped to integer values according to the dynamic range of the data, such as int8, int16 or int32, etc. Through testing, this method is faster than the kaldi tool (an open source speech recognition tool) Almost twice as fast.

The TDNN module 502 is configured to determine the probability distribution of the syllables corresponding to the speech frames included in the speech data according to the speech features.

The decoding module 503 determines a speech recognition result for determining whether a keyword is included in the speech data.

The decoder and the TDNN can form an acoustic model as shown in Figure 6, and the extracted speech features are input into the TDNN to obtain the syllable probability distribution, and the syllable probability distribution is input to the decoder. At the same time, the speech features are data aligned and input to the decoder, and the decoder determines the speech recognition result used to identify whether the speech data includes keywords according to the probability distribution of syllables and the matching vocabulary.

The vocabulary list generation module 504 is configured to generate the syllable sequence corresponding to the keyword according to the dictionary.

Thus, by cascading the above four modules, the speech recognition system can efficiently detect all keywords and their start and end positions in the speech data to be recognized.

In this keyword recognition scenario, the speech data to be recognized with a duration of 27.5 hours is used as the test set, and Table 2 lists the effects of the two keyword detection methods. It can be seen from the results that, compared with the traditional keyword/fill-based keyword detection system, the keyword detection proposed in the embodiments of the present application has obvious improvements in accuracy and coverage, and the overall F1 increases from 65.99% boosted to 75.57%. This result shows that the speech recognition method provided by the embodiment of the present application can effectively increase the number of recalled keywords, and can also more effectively reduce the number of false alarms of keywords.

Table 2 Performance comparison experimental results

method Accuracy recall F1 Keyword/stuffing-based keyword detection 86.76% 53.24% 65.99% Keyword Detection Based on the Embodiments of the Application 92.00% 64.12% 75.57% boost effect +5.23% +10.89% +9.58%

Among them, F1 is an indicator used in statistics to measure the accuracy of the binary classification model. It takes into account both the precision and recall of the classification model. The F1 score can be regarded as a harmonic average of the model's precision and recall, with a maximum value of 1 and a minimum value of 0.

With respect to the speech recognition method provided by the foregoing embodiments, an embodiment of the present application further provides a speech recognition apparatus.

Referring to FIG. 7 , this figure is a schematic diagram of a speech recognition apparatus provided by an embodiment of the present application. As shown in FIG. 7 , the speech recognition apparatus 700 includes: an acquisition unit 701, a syllable probability distribution determination unit 702, and a speech recognition result determination unit 703;

The obtaining unit 701 is configured to obtain an acoustic model and speech data to be recognized, the acoustic model includes a time-delay neural network, and the output layer of the time-delay neural network includes an acoustic modeling unit corresponding to a plurality of syllables respectively;

The syllable probability distribution determining unit 702 is configured to use the speech data as the input data of the time-delay neural network, and determine the syllable probability distributions corresponding to the speech frames included in the speech data through the time-delay neural network, The syllable probability distribution is used to identify the probability that the speech frame corresponds to the multiple syllables respectively;

The speech recognition result determining unit 703 is configured to determine the speech recognition result corresponding to the speech data according to the syllable probability distribution.

As a possible implementation manner, the acoustic model further includes a decoder, and the speech recognition result determination unit 703 is configured to:

According to the wake-up scene corresponding to the voice data, determine the keyword corresponding to the wake-up scene;

According to the probability distribution of the syllables, determine, by the decoder, a speech recognition result for identifying whether the speech data includes the keyword;

The method also includes:

If the voice recognition result indicates that the voice data includes the keyword, wake up the terminal device corresponding to the wake-up scene.

As a possible implementation manner, the apparatus further includes a matching vocabulary building unit for:

Constructing a matching vocabulary for the keyword with syllables as the division granularity;

The speech recognition result determination unit 703 is used for:

According to the syllable probability distribution and the matching vocabulary, a speech recognition result for identifying whether the keyword is included in the speech data is determined by the decoder.

As a possible implementation manner, the time delay neural network includes N layers of feature extraction layers, j∈N, and for the i-th speech frame in the speech frames included in the speech data, the syllable probability distribution determining unit 702 , for:

According to the output feature of the i-th frame of speech frame according to the feature extraction layer of the j-1 layer, the speech frame feature of the i-th frame of speech frame is determined by the feature extraction layer of the j-th layer;

Determine the output of the i-th voice frame in the j-th feature extraction layer by using the voice frame feature and the voice frame feature corresponding to at least one frame before and after the i-th voice frame in the voice data feature;

The syllable probability distribution corresponding to the i-th speech frame is determined according to the output feature of the i-th speech frame in the j-th feature extraction layer.

As a possible implementation manner, the matching vocabulary building unit is used for:

Determine the similar syllables of the syllables included in the keyword according to the degree of pronunciation similarity;

If there is a polyphonic word in the keyword, determine the polyphonic syllable corresponding to the polyphonic word;

According to at least one of the similar syllables and the polysyllabic syllables, and the syllables corresponding to the keywords, a matching vocabulary for the keywords is constructed with a syllable as a division granularity.

As a possible implementation manner, the apparatus further includes a training unit for:

obtaining a voice sample belonging to the same language as the voice data;

According to the speech sample as the input data of the initial acoustic model, the prediction result is obtained through the initial time delay neural network included in the initial acoustic model;

A loss function is determined according to the prediction result and the sample label of the speech sample, and the loss function includes a guide weight, and the guide weight is used to improve the influence of the correct prediction result on the recognition path in the initial delay neural network , reducing the influence of the wrong prediction result on the identification path in the initial delay neural network;

The model parameters of the initial time-delay neural network in the initial acoustic model are adjusted according to the loss function to obtain the acoustic model.

As a possible implementation manner, the training unit is used for:

Acquiring to-be-processed voice samples belonging to the same language as the voice data;

Data augmentation is performed according to the to-be-processed speech sample to obtain an augmented sample, and the sample label of the augmented sample is determined based on the sample label of the corresponding to-be-processed speech sample;

The speech sample is obtained according to the to-be-processed speech sample and the augmented sample.

As a possible implementation manner, the training unit is also used for:

Using the sample label of the to-be-processed speech sample corresponding to the augmented sample as the to-be-determined label of the augmented sample;

Perform label alignment processing on the speech frame of the augmented sample and the undetermined label to obtain the sample label of the augmented sample.

It can be seen from the above technical solutions that, for the speech data to be recognized, an acoustic model including a time-delay neural network is obtained. Taking the speech data as the input data of the time-delay neural network, since the output layer of the time-delay neural network includes acoustic modeling units corresponding to multiple syllables, the time-delay neural network can use syllables as the identification granularity, and Each acoustic modeling unit of the output layer obtains the probability distribution of the syllables corresponding to the speech frames included in the speech data. In the process of speech feature extraction and transmission to the output layer, the time-delay neural network will carry the rich context information of the speech frame in the speech data. When identifying syllables through the output layer, the syllables of the speech frame can be judged based on the pronunciation rules based on the information of the syllables before and after the speech frame, so as to output a more accurate probability distribution of syllables. Moreover, since a syllable is generally composed of one or more phonemes, it has higher error tolerance. Even if the pronunciation error of an individual phoneme occurs in a syllable in the speech data, the impact on the recognition result of the entire syllable is smaller than that of the phoneme. Therefore, even if the quality of the speech data to be recognized is not high, a more accurate determination result of speech recognition can be obtained based on the probability distribution of syllables, which effectively expands the applicable scenarios of the speech recognition technology.

The aforementioned voice recognition device can be a computer device, which can be a server or a terminal device, and the aforementioned voice recognition device can be built into a server or a terminal device. The computer equipment provided by the embodiments of the present application will be introduced from a perspective. Among them, FIG. 8 is a schematic structural diagram of a server, and FIG. 9 is a schematic structural diagram of a terminal device.

Referring to FIG. 8, FIG. 8 is a schematic structural diagram of a server provided by an embodiment of the present application. The server 1400 may vary greatly due to different configurations or performance, and may include one or more central processing units (Central Processing Units, CPU) 1422 (eg, one or more processors) and memory 1432, one or more storage media 1430 (eg, one or more mass storage devices) storing applications 1442 or data 1444. Among them, the memory 1432 and the storage medium 1430 may be short-term storage or persistent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the CPU 1422 may be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the server 1400 .

Server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and/or, one or more operating systems 1441, such as Windows Server ^™ , Mac OS X ^TM , Unix ^TM , Linux ^TM , FreeBSD ^TM and many more.

The steps performed by the server in the above embodiment may be based on the server structure shown in FIG. 8 .

Among them, the CPU 1422 is used to perform the following steps:

Acquiring an acoustic model and the speech data to be recognized, the acoustic model includes a time-delay neural network, and the output layer of the time-delay neural network includes acoustic modeling units corresponding to a plurality of syllables respectively;

The speech data is used as the input data of the time-delay neural network, and the syllable probability distribution corresponding to the speech frames included in the speech data is determined through the time-delay neural network, and the syllable probability distribution is used to identify the speech the probability that the frame corresponds to the plurality of syllables respectively;

The speech recognition result corresponding to the speech data is determined according to the syllable probability distribution.

Optionally, the CPU 1422 may also execute the method steps of any specific implementation manner of the speech recognition method in the embodiment of the present application.

Referring to FIG. 9 , FIG. 9 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. FIG. 9 is a block diagram showing a partial structure of a smart phone related to a terminal device provided by an embodiment of the present application. The smart phone includes: a radio frequency (Radio Frequency, RF for short) circuit 1510 , a memory 1520 , an input unit 1530 , and a display unit 1540, a sensor 1550, an audio circuit 1560, a wireless fidelity (Wireless Fidelity, WiFi for short) module 1570, a processor 1580, a power supply 1590 and other components. Those skilled in the art can understand that the structure of the smart phone shown in FIG. 9 does not constitute a limitation on the smart phone, and may include more or less components than the one shown, or combine some components, or arrange different components.

The following describes the various components of the smart phone in detail with reference to FIG. 9 :

The RF circuit 1510 can be used for receiving and sending signals during the process of sending and receiving information or talking. In particular, after receiving the downlink information of the base station, it is processed by the processor 1580; in addition, it sends the designed uplink data to the base station. Generally, the RF circuit 1510 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA for short), a duplexer, and the like. In addition, RF circuitry 1510 may also communicate with networks and other devices via wireless communications. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM for short), General Packet Radio Service (GPRS for short), Code Division Multiple Access (Code Division Multiple Access) Division Multiple Access (CDMA for short), Wideband Code Division Multiple Access (WCDMA for short), Long Term Evolution (LTE for short), E-mail, Short Messaging Service (Short Messaging Service, SMS for short), etc. .

The memory 1520 can be used to store software programs and modules, and the processor 1580 implements various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1520 . The memory 1520 may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program (such as a sound playback function, an image playback function, etc.) required for at least one function, and the like; Data created by the use of the smartphone (such as audio data, phonebook, etc.), etc. Additionally, memory 1520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The input unit 1530 may be used to receive input numerical or character information, and generate key signal input related to user settings and function control of the smartphone. Specifically, the input unit 1530 may include a touch panel 1531 and other input devices 1532 . The touch panel 1531, also known as a touch screen, can collect the user's touch operations on or near it (such as the user's finger, stylus, etc., any suitable object or accessory on or near the touch panel 1531). operation), and drive the corresponding connection device according to the preset program. Optionally, the touch panel 1531 may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the touch controller. To the processor 1580, and can receive the command sent by the processor 1580 and execute it. In addition, the touch panel 1531 can be realized by various types of resistive, capacitive, infrared, and surface acoustic waves. Besides the touch panel 1531 , the input unit 1530 may also include other input devices 1532 . Specifically, other input devices 1532 may include, but are not limited to, one or more of physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, joysticks, and the like.

The display unit 1540 may be used to display information input by the user or information provided to the user and various menus of the smartphone. The display unit 1540 may include a display panel 1541 . Optionally, the display panel 1541 may be configured in the form of a liquid crystal display (LCD for short), an organic light-emitting diode (OLED for short). Further, the touch panel 1531 may cover the display panel 1541. When the touch panel 1531 detects a touch operation on or near it, it transmits it to the processor 1580 to determine the type of the touch event, and then the processor 1580 determines the type of the touch event according to the touch event. Type provides corresponding visual output on display panel 1541. Although in FIG. 9, the touch panel 1531 and the display panel 1541 are used as two independent components to realize the input and input functions of the smartphone, in some embodiments, the touch panel 1531 and the display panel 1541 may be integrated And realize the input and output functions of smart phones.

The smartphone may also include at least one sensor 1550, such as light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1541 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1541 and the display panel 1541 when the smartphone is moved to the ear. / or backlight. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, and can be used for applications that recognize the posture of smartphones (such as horizontal and vertical screen switching, related games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can be configured on smartphones, here No longer.

The audio circuit 1560, the speaker 1561, and the microphone 1562 can provide an audio interface between the user and the smartphone. The audio circuit 1560 can convert the received audio data into an electrical signal, and transmit it to the speaker 1561, and the speaker 1561 converts it into a sound signal for output; on the other hand, the microphone 1562 converts the collected sound signal into an electrical signal, which is converted by the audio circuit 1560 After receiving, it is converted into audio data, and then the audio data is output to the processor 1580 for processing, and then sent to, for example, another smartphone via the RF circuit 1510, or the audio data is output to the memory 1520 for further processing.

WiFi is a short-distance wireless transmission technology. The smartphone can help users to send and receive emails, browse web pages, and access streaming media through the WiFi module 1570. It provides users with wireless broadband Internet access. Although FIG. 9 shows the WiFi module 1570, it can be understood that it is not a necessary component of a smart phone, and can be completely omitted as required within the scope of not changing the essence of the invention.

The processor 1580 is the control center of the smart phone, using various interfaces and lines to connect various parts of the entire smart phone, by running or executing the software programs and/or modules stored in the memory 1520, and calling the data stored in the memory 1520. , perform various functions of the smartphone and process data. Optionally, the processor 1580 may include one or more processing units; preferably, the processor 1580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc. , the modem processor mainly deals with wireless communication. It can be understood that, the above-mentioned modulation and demodulation processor may not be integrated into the processor 1580.

The smart phone also includes a power supply 1590 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, so as to manage charging, discharging, and power consumption management functions through the power management system.

Although not shown, the smart phone may also include a camera, a Bluetooth module, etc., which will not be repeated here.

In this embodiment of the present application, the memory 1520 included in the smart phone may store program codes, and transmit the program codes to the processor.

The processor 1580 included in the smart phone can execute the speech recognition method provided by the above embodiment according to the instructions in the program code.

Embodiments of the present application further provide a computer-readable storage medium for storing a computer program, where the computer program is used to execute the speech recognition method provided by the foregoing embodiments.

Embodiments of the present application also provide a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the speech recognition methods provided in various optional implementations of the above aspects.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, the execution includes: The steps of the above method embodiments; and the aforementioned storage medium may be at least one of the following media: read-only memory (English: Read-only Memory, abbreviation: ROM), RAM, magnetic disk or optical disk and other various storage media medium of program code.

It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. place. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for related parts. The device and system embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Substitutions should be covered within the protection scope of this application. Moreover, on the basis of the implementation manners provided by the above aspects, the present application may further combine to provide more implementation manners. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (12)

1. a speech recognition method, is characterized in that, described method comprises: Acquiring an acoustic model and the speech data to be recognized, the acoustic model includes a time-delay neural network, and the output layer of the time-delay neural network includes acoustic modeling units corresponding to a plurality of syllables respectively; The speech data is used as the input data of the time-delay neural network, and the syllable probability distribution corresponding to the speech frames included in the speech data is determined through the time-delay neural network, and the syllable probability distribution is used to identify the speech the probability that the frame corresponds to the plurality of syllables respectively; The speech recognition result corresponding to the speech data is determined according to the syllable probability distribution. 2. The method according to claim 1, wherein the acoustic model further comprises a decoder, and the determining the speech recognition result corresponding to the speech data according to the syllable probability distribution comprises: According to the wake-up scene corresponding to the voice data, determine the keyword corresponding to the wake-up scene; According to the probability distribution of the syllables, determine, by the decoder, a speech recognition result for identifying whether the speech data includes the keyword; The method also includes: If the voice recognition result indicates that the voice data includes the keyword, wake up the terminal device corresponding to the wake-up scene. 3. The method according to claim 2, wherein the method further comprises: Constructing a matching vocabulary for the keyword with syllables as the division granularity; The determining, by the decoder, according to the syllable probability distribution, the speech recognition result for identifying whether the speech data includes the keyword includes: According to the syllable probability distribution and the matching vocabulary, a speech recognition result for identifying whether the keyword is included in the speech data is determined by the decoder. 4. The method according to claim 1, wherein the time delay neural network comprises N layers of feature extraction layers, j∈N, for the i-th speech frame in the speech frames included in the speech data, the The voice data is used as the input data of the time-delay neural network, and the probability distribution of syllables corresponding to the speech frames included in the voice data is determined by the time-delay neural network, including: According to the output feature of the i-th frame of speech frame according to the feature extraction layer of the j-1 layer, the speech frame feature of the i-th frame of speech frame is determined by the feature extraction layer of the j-th layer; Determine the output of the i-th voice frame in the j-th feature extraction layer by using the voice frame feature and the voice frame feature corresponding to at least one frame before and after the i-th voice frame in the voice data feature; The syllable probability distribution corresponding to the i-th speech frame is determined according to the output feature of the i-th speech frame in the j-th feature extraction layer. 5. The method according to claim 3, wherein the method further comprises: Determine the similar syllables of the syllables included in the keyword according to the degree of pronunciation similarity; If there is a polyphonic word in the keyword, determine the polyphonic syllable corresponding to the polyphonic word; The described construction of a matching vocabulary for the keyword with syllables as the division granularity, including: According to at least one of the similar syllables and the polysyllabic syllables, and the syllables corresponding to the keywords, a matching vocabulary for the keywords is constructed with a syllable as a division granularity. 6. The method of claim 1, wherein the method further comprises: obtaining a voice sample belonging to the same language as the voice data; According to the speech sample as the input data of the initial acoustic model, the prediction result is obtained through the initial time delay neural network included in the initial acoustic model; A loss function is determined according to the prediction result and the sample label of the speech sample, and the loss function includes a guide weight, and the guide weight is used to improve the influence of the correct prediction result on the recognition path in the initial delay neural network , reducing the influence of the wrong prediction result on the identification path in the initial delay neural network; The model parameters of the initial time-delay neural network in the initial acoustic model are adjusted according to the loss function to obtain the acoustic model. 7. The method according to claim 6, wherein the acquiring a voice sample belonging to the same language as the voice data comprises: Acquiring to-be-processed voice samples belonging to the same language as the voice data; Data augmentation is performed according to the to-be-processed speech sample to obtain an augmented sample, and the sample label of the augmented sample is determined based on the sample label of the corresponding to-be-processed speech sample; The speech sample is obtained according to the to-be-processed speech sample and the augmented sample. 8. The method according to claim 7, wherein the method further comprises: Using the sample label of the to-be-processed speech sample corresponding to the augmented sample as the to-be-determined label of the augmented sample; Perform label alignment processing on the speech frame of the augmented sample and the undetermined label to obtain the sample label of the augmented sample. 9. A speech recognition device, characterized in that the device comprises an acquisition unit, a syllable probability distribution determination unit and a speech recognition result determination unit; The acquisition unit is used to acquire an acoustic model and the speech data to be recognized, the acoustic model includes a time-delay neural network, and the output layer of the time-delay neural network includes an acoustic modeling unit corresponding to a plurality of syllables respectively; The syllable probability distribution determining unit is configured to use the speech data as the input data of the time-delay neural network, and determine the syllable probability distributions corresponding to the speech frames included in the speech data through the time-delay neural network. The syllable probability distribution is used to identify the probability that the speech frame corresponds to the multiple syllables respectively; The speech recognition result determination unit is configured to determine the speech recognition result corresponding to the speech data according to the syllable probability distribution. 10. A computer device, characterized in that the computer device comprises a processor and a memory: the memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the speech recognition method according to any one of claims 1-8 according to the instructions in the program code. 11. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the speech recognition method according to any one of claims 1-8. 12. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the speech recognition method of any one of claims 1-8.

Images