TW201828279A - Voice recognition method and device capable of precisely recognizing the information related to the client end when recognizing voice signal sampled from the user end - Google Patents

Abstract

The present invention discloses a voice recognition method, which includes using a preset voice knowledge source to generate a search space, which includes user end preset data, for decoding voice signals; extracting feature vector sequence of the voice signal to be recognized; computing probability of the feature vector corresponding to the basic unit of the search space; performing decoding operation in the search space by using the probability as an input, in order to obtain a word sequence corresponding to the feature vector sequence. The present invention further provides a voice recognition device, and another voice recognition method and device. According to the method provided by the present invention, since the user end preset data is included when generating the search space for decoding, the information related the client end can be relatively and accurately identified when the voice signal collected by the user end is recognized, so as to increase accuracy of voice recognition, and improve user experience.

Description

 Speech recognition method and device  

The present application relates to a speech recognition technology, and more particularly to a speech recognition method and apparatus. This application also relates to another speech recognition method and apparatus.

Speech is the acoustic expression of language. It is the most natural, effective and convenient means for human beings to exchange information. It is also a kind of support for human thinking. Automatic Speech Recognition (ASR) usually refers to the process of converting a person's spoken language into a corresponding output text or command through recognition and understanding of speech. The core framework is: based on the statistical model modeling, based on the feature sequence O extracted from the speech signal to be recognized, the following Bayesian decision criterion is used to solve the optimal word sequence corresponding to the speech signal to be recognized. *:W*=argmaxP(O|W)P(W)

In a specific implementation, the process of solving the optimal word sequence is called a decoding process (a module that implements a decoding function is generally called a decoder), that is, searching in a search space composed of a plurality of knowledge sources such as a pronunciation dictionary and a language model. The best word sequence shown in the above formula.

With the development of technology, the computing power and storage capacity of hardware have been greatly improved. The speech recognition system has been gradually applied in the industry, and various applications using voice as a human-computer interaction medium have appeared on the user-side device. For example, the dialing application on the smart phone, the user only needs to give a voice indication (such as: "call Zhang three"), then the phone dialing function can be automatically realized.

The current voice recognition application usually adopts two modes, one is based on the mode of the client and the server, that is, the user collects the voice, uploads it to the server via the network, and the server recognizes the voice as text through decoding, and then returns the voice. To the client. The reason for adopting this mode is that the computing power of the client is relatively weak, and the memory space is limited, and the server has obvious advantages in both aspects; but this mode, if there is no network access In the environment, the client cannot complete the voice recognition function. In response to the above problem, a second voice recognition application mode that relies only on the user end appears. In this mode, by reducing the scale, the model and the search space originally stored on the server are placed locally on the client device, and the user terminal itself The operation of collecting voice and decoding is completed.

In practical applications, whether the first mode or the second mode, when using the above-mentioned general framework for voice recognition, it is generally impossible to effectively identify the content of the voice signal related to the local information of the user equipment, for example, in the address book. The name of the contact person leads to low recognition accuracy, which causes inconvenience to the user and affects the user experience.

The embodiment of the present invention provides a voice recognition method and device, which solves the problem that the existing voice recognition technology has low recognition accuracy of local related information at the user end. Another embodiment of the present application provides a voice recognition method and apparatus.

The present application provides a voice recognition method, including: generating a search space for decoding a voice signal by using a preset voice knowledge source; extracting a feature vector sequence of the voice signal to be recognized; and calculating a feature The vector corresponds to the probability of searching for the basic unit of the space; the input is performed at the probability, and a decoding operation is performed in the search space to obtain a sequence of words corresponding to the sequence of feature vectors.

Optionally, the search space includes: a weighted finite state converter.

Optionally, the search space basic unit includes: a context-dependent triphone; the preset knowledge source includes: a pronunciation dictionary, a language model, and a triphone state linkage field table.

Optionally, the generating, by using the preset voice knowledge source, a search space for decoding the voice signal that includes the preset information of the UE, including: by using a replacement label, to a pre-generated at least language model based Adding user preset information corresponding to the preset theme category in the weighted finite state converter, and obtaining a single weighted finite state converter based on the triphone state tied field table, the pronunciation dictionary, and the language model; wherein the language The model is pre-trained in the following manner: The preset named entity in the text for training the language model is replaced with a label corresponding to the preset theme category, and the language model is trained using the text.

Optionally, the method for replacing the label is used to add the user preset information corresponding to the preset theme category to the pre-generated at least language-based weighted finite state converter, and obtain a triphone-based state binding column. A single weighted finite state converter of the bit table, the pronunciation dictionary, and the language model, including: adding a user preset corresponding to the preset theme category to the pre-generated language model-based weighted finite state converter by replacing the label Information; the weighted finite state converter to which the user preset information is added, combined with a pre-generated weighted finite state converter based on the triphone stateline field table and the pronunciation dictionary, to obtain the single weighted limited State converter.

Optionally, the text used to train the language model refers to text for the preset theme category.

Optionally, the number of the preset topic categories is at least two; the number of the language models, and the number of the at least language-based weighted finite state devices are respectively consistent with the number of preset topic categories; Adding the user preset information corresponding to the preset theme category to the pre-generated at least language-based weighted finite state converter by means of replacing the label, including: determining a preset theme category to which the to-be-identified voice signal belongs; a pre-generated, at least language-based weighted finite state converter corresponding to the preset theme category; by replacing the corresponding label with the user preset information corresponding to the preset theme category, Add the client preset information to the selected weighted finite state converter.

Optionally, the determining the preset topic category to which the to-be-identified voice signal belongs is implemented by determining, according to the type of the client that collects the voice signal, or the application, the preset topic category.

Optionally, the preset theme category includes: making a call or sending a short message, playing a music, or setting a command; the corresponding user preset information includes: a contact name in the address book, a music name in the music library, Or, an instruction in the instruction set.

Optionally, the merging operation comprises: merging using a prediction-based method.

Optionally, the vocabulary used in pre-training the language model is consistent with the words included in the pronunciation dictionary.

Optionally, the calculating the feature vector corresponds to the probability of searching for the basic unit of the space, comprising: calculating a probability that the feature vector corresponds to each triphone state by using the pre-trained DNN model; and corresponding to the triphone state according to the feature vector. Probability, using a pre-trained HMM model to calculate the probability that the feature vector corresponds to each triphone.

Optionally, the execution speed of the step of calculating the probability that the feature vector corresponds to each triphone state by using the pre-trained DNN model is improved by: using data parallel processing capability provided by the hardware platform.

Optionally, the extracting the feature vector sequence of the to-be-identified speech signal includes: performing frame processing on the speech signal to be recognized according to a preset frame length to obtain a plurality of audio frames; and extracting feature vectors of each audio frame to obtain the A sequence of feature vectors.

Optionally, the extracting the feature vector of each audio frame comprises: extracting an MFCC feature, a PLP feature, or an LPC feature.

Optionally, after the obtaining the word sequence corresponding to the feature vector sequence, performing the following operations: verifying the accuracy of the word sequence by performing text matching with the user preset information, and according to the verification result Generate corresponding speech recognition results.

Optionally, the performing the text matching with the user preset information to verify the accuracy of the word sequence, and obtaining the corresponding voice recognition result according to the verification result, including: selecting, according to the word sequence, corresponding to the Determining a word to be verified of the user preset information; searching for the to-be-verified word in the user-preset information; if found, determining to pass the accuracy verification, and using the word sequence as a speech recognition result; Otherwise, the word sequence is corrected by a pinyin-based fuzzy matching method, and the corrected word sequence is used as a speech recognition result.

Optionally, the correcting the word sequence by using a pinyin-based fuzzy matching manner includes: converting the to-be-verified word into a to-be-verified pinyin sequence; converting each word in the user-side preset information into a ratio a pair of Pinyin sequences; calculating a similarity between the Pinyin sequence to be verified and each of the Pinyin sequences in sequence, and selecting a word sorted according to the similarity from high to low from the user preset information; The corrected word sequence is obtained by replacing the word to be verified in the word sequence with the selected word.

Optionally, the similarity includes: a similarity calculated based on an edit distance.

Optionally, the method is implemented on a user equipment, where the user equipment includes: a smart mobile terminal, a smart speaker, or a robot.

Correspondingly, the present application further provides a voice recognition device, comprising: a search space generating unit, configured to generate, by using a preset voice knowledge source, a search space for decoding a voice signal including user preset information; a vector extracting unit, configured to extract a feature vector sequence of the speech signal to be recognized; a probability calculating unit, configured to calculate a probability that the feature vector corresponds to the basic unit of the search space; and a decoding search unit, configured to input the probability, in the A decoding operation is performed in the search space to obtain a sequence of words corresponding to the sequence of feature vectors.

Optionally, the search space generating unit is configured to: add, by using a replacement label, a user preset information corresponding to the preset theme category to the pre-generated at least language-based weighted finite state converter, and Obtaining a single weighted finite state converter based on a triphone state tied field table, a pronunciation dictionary, and a language model; the language model is pre-generated by a language model training unit, and the language model training unit is used The preset named entity in the text of the training language model is replaced with a label corresponding to the preset theme category, and the language model is trained using the text.

Optionally, the search space generating unit includes: a first user information adding subunit, configured to add, according to the manner of replacing the label, a pre-generated language model-based weighted finite state converter corresponding to the preset topic category User preset information; a weighted finite state converter merging subunit for using the weighted finite state converter to which the user preset information is added, and a pre-generated triphone-based state binding field table The weighted finite state converter of the pronunciation dictionary is combined to obtain the single weighted finite state converter.

Optionally, the decoding space generating unit includes: a second user information adding subunit, configured to add a preset theme category to the pre-generated at least language model-based weighted finite state converter by replacing the label Corresponding user preset information; a unified weighted finite state converter obtaining subunit, configured to obtain a triphone based state binding field table, a pronunciation dictionary, and after the second user information adding subunit completes the adding operation, And a single weighted finite state converter of the language model; wherein the second user information adding subunit comprises: a topic determining subunit, configured to determine a preset topic category to which the to-be-identified speech signal belongs; weighted finite state converter selection a subunit, configured to select the at least language-based weighted finite state converter corresponding to the preset topic category, and a label replacement subunit, configured to correspond to the preset topic category The manner in which the user preset information replaces the corresponding label to the selected weighted finite state converter Add client preset information.

Optionally, the topic determining subunit is specifically configured to determine, according to a type of the client that collects the voice signal, or an application, the belonging preset topic category.

Optionally, the weighted finite state converter combining subunit is specifically configured to perform a combining operation by using a prediction based method, and obtain the single weighted finite state converter.

Optionally, the probability calculation unit includes: a triphone state probability calculation sub-unit, configured to calculate a probability that the feature vector corresponds to each triphone state by using a pre-trained DNN model; and a triphone probability calculation sub-unit for The vector corresponds to the probability of each of the triphone states, and the pre-trained HMM model is used to calculate the probability that the feature vector corresponds to each triphone.

Optionally, the feature vector extraction unit includes: a framing sub-unit, configured to perform framing processing on the speech signal to be recognized according to a preset frame length, to obtain a plurality of audio frames; and a feature extraction sub-unit, configured to extract each audio The feature vector of the frame results in the sequence of feature vectors.

Optionally, the device includes: an accuracy verification unit, configured to verify, after the decoding search unit obtains a sequence of words corresponding to the sequence of feature vectors, verify the word sequence by performing text matching with the preset information of the UE Accuracy, and generate corresponding speech recognition results based on the verification results.

Optionally, the accuracy verification unit includes: a to-be-verified word selection sub-unit, configured to select a to-be-verified word corresponding to the user-preset information from the word sequence; and the search sub-unit is used in the Searching for the to-be-verified word in the user-preset information; the recognition result confirmation sub-unit is configured to, after the finding sub-unit finds the to-be-verified word, determine to pass the accuracy verification, and the word sequence is As a speech recognition result, the recognition result correction subunit is configured to, after the search subunit does not find the to-be-verified word, correct the word sequence through a pinyin-based fuzzy matching manner, and use the corrected word sequence as a speech Identify the results.

Optionally, the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, configured to convert the to-be-verified word into a to-be-verified pinyin sequence; and a comparison pinyin sequence conversion subunit, Each word in the preset information of the client is converted into a comparison pinyin sequence, and a similarity calculation selection sub-unit is configured to sequentially calculate the similarity between the to-be-verified pinyin sequence and each of the comparison pinyin sequences, and from the Selecting, in the user preset information, a word that is sorted according to the similarity from high to low; the word to be verified is replaced by a subunit for replacing the word to be verified in the word sequence with the selected word, and the correction is obtained. After the word sequence.

In addition, the present application further provides another voice recognition method, including: acquiring a sequence of words corresponding to the voice signal to be recognized through decoding; verifying the accuracy of the word sequence by performing text matching with the preset information of the user, and according to the verification result Generate corresponding speech recognition results.

Optionally, the performing the text matching with the user preset information to verify the accuracy of the word sequence, and generating a corresponding voice recognition result according to the verification result, including: selecting, from the word sequence, corresponding to the user Ending the information of the information to be verified; searching for the word to be verified in the preset information of the user; if found, determining to pass the accuracy verification, and using the word sequence as a speech recognition result; The word matching sequence is modified based on the pinyin-based fuzzy matching method, and the corrected word sequence is used as the speech recognition result.

Optionally, the correcting the word sequence by using a pinyin-based fuzzy matching manner includes: converting the to-be-verified word into a to-be-verified pinyin sequence; converting each word in the user-side preset information into a ratio a pair of Pinyin sequences; calculating a similarity between the Pinyin sequence to be verified and each of the Pinyin sequences in sequence, and selecting a word sorted according to the similarity from high to low from the user preset information; The corrected word sequence is obtained by replacing the word to be verified in the word sequence with the selected word.

Correspondingly, the present application further provides another speech recognition apparatus, including: a word sequence acquisition unit, configured to obtain a word sequence corresponding to the to-be-identified speech signal through decoding, and a word sequence verification unit, configured to perform preset information with the user terminal. The text matching verifies the accuracy of the word sequence, and generates a corresponding speech recognition result according to the verification result.

Optionally, the word sequence verification unit includes: a to-be-verified word selection sub-unit, configured to select a to-be-verified word corresponding to the user-preset information from the word sequence; and the search sub-unit is used in the Searching for the to-be-verified word in the user-preset information; the recognition result confirmation sub-unit is configured to, after the finding sub-unit finds the to-be-verified word, determine to pass the accuracy verification, and the word sequence is As a speech recognition result, the recognition result correction subunit is configured to, after the search subunit does not find the to-be-verified word, correct the word sequence through a pinyin-based fuzzy matching manner, and use the corrected word sequence as a speech Identify the results.

Optionally, the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, configured to convert the to-be-verified word into a to-be-verified pinyin sequence; and a comparison pinyin sequence conversion subunit, Each word in the preset information of the client is converted into a comparison pinyin sequence, and a similarity calculation selection sub-unit is configured to sequentially calculate the similarity between the to-be-verified pinyin sequence and each of the comparison pinyin sequences, and from the Selecting, in the user preset information, a word that is sorted according to the similarity from high to low; the word to be verified is replaced by a subunit for replacing the word to be verified in the word sequence with the selected word, and the correction is obtained. After the word sequence.

Compared with the prior art, the present application has the following advantages: the voice recognition method provided by the present application is based on using a preset voice knowledge source to generate a search space for decoding a voice signal including user preset information. Calculating a probability that the feature vector extracted from the speech signal to be recognized corresponds to the basic unit of the search space, and performing a decoding operation in the search space according to the probability, thereby obtaining a word sequence corresponding to the speech signal to be recognized. The above method provided by the present application includes the user preset information when generating the search space for decoding, so that the information related to the client can be relatively accurately identified when the voice signal collected by the user end is recognized. Thereby, the accuracy of speech recognition can be improved, and the user experience is improved.

901‧‧‧Search space generation unit

902‧‧‧Feature vector extraction unit

903‧‧‧ probability calculation unit

904‧‧‧Decoding search unit

1101‧‧‧word sequence acquisition unit

1102‧‧‧word sequence verification unit

1 is a flowchart of an embodiment of a voice recognition method according to the present application; FIG. 2 is a flowchart of a process for generating a search space for decoding a voice signal including user preset information according to an embodiment of the present application; 3 is a schematic diagram of a G structure WFST before performing a replacement operation according to an embodiment of the present application; FIG. 4 is a schematic diagram of a G structure WFST after performing a replacement operation according to an embodiment of the present application; FIG. 5 is an extraction provided by an embodiment of the present application. A flowchart of the processing of the feature vector sequence of the speech signal to be recognized; FIG. 6 is a flowchart of the process of calculating the probability that the feature vector corresponds to each triphone provided by the embodiment of the present application; FIG. 7 is a text matching verification provided by the embodiment of the present application. A flowchart of the process for generating the corresponding speech recognition result according to the accuracy of the word sequence; FIG. 8 is an overall frame diagram of the speech recognition provided by the embodiment of the present application; FIG. 9 is an embodiment of the speech recognition device of the present application. FIG. 10 is a flowchart of another embodiment of a speech recognition method of the present application; FIG. 11 is another speech recognition apparatus of the present application. Schematic diagram of the embodiment.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the application. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar promotion without departing from the scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed below.

In the present application, a voice recognition method and corresponding device, and another voice recognition method and corresponding device are respectively provided, which are described in detail in the following embodiments. For the sake of easy understanding, prior to describing the embodiments, the technical solutions, related technical terms, and manners of writing the embodiments of the present application are briefly described.

The voice recognition method provided by the present application is generally applied to an application in which voice is used as a human-computer interaction medium. Such an application recognizes the collected voice signal as text, and then performs a corresponding operation according to the text, where the voice signal usually involves the user. Local preset information (for example, the contact name in the address book). The existing speech recognition technology uses a common search space for decoding and recognizing the above-mentioned speech signal to be recognized, and the general search space does not consider the difference between such applications in different user terminals, and thus generally cannot effectively recognize the speech signal. Content related to the local information of the client leads to low recognition accuracy. In response to this problem, the technical solution of the present application integrates the user preset information in the process of constructing a search space for decoding a voice signal, which is equivalent to customizing the specific voice recognition requirement of the user end, thereby being able to effectively identify Local information related to the client, to achieve the purpose of improving the accuracy of speech recognition.

In a speech recognition system, the process of obtaining a sequence of words that best matches the speech signal to be recognized is called decoding. The search space used to decode the speech signal described in the present application refers to all possible speech recognition covered by the speech knowledge source (eg, acoustic model, pronunciation dictionary, language model, etc.) involved in the speech recognition system. The resulting space. Correspondingly, the decoding process is a process of searching and matching in the search space for the speech signal to be recognized to obtain the best matching word sequence.

The form of the search space may be diversified, and various knowledge sources may be used in different independent search spaces. The decoding process is a layer-by-layer calculation process; a weighted finite state converter (Weighted Finite) may also be used. The State Transducer (WFST) search space organically integrates various knowledge sources into a unified WFST network (also known as the WFST search space). In view of the fact that the latter facilitates the introduction of different knowledge sources and can improve the search efficiency, it is a preferred method for voice recognition in the technical solution of the present application. Therefore, the embodiment based on the WFST network is mainly described in the embodiment provided by the present application.

The core of the WFST search space is to use a weighted finite state converter to analogize the grammatical structure of the language and the associated acoustic properties. The specific operation method is: the knowledge sources at different levels are respectively expressed in the form of WFST, and then the WFST features and the merge algorithm are used to integrate the above-mentioned WFSTs at different levels into a single WFST network, which is used for voice. Identify the search space.

The basic unit of the WFST network (the basic input unit that drives WFST for state transition) can be selected according to specific needs. Considering the influence of the phoneme context on phoneme pronunciation, in order to obtain higher recognition accuracy, a context-dependent Contextual Dependent triphone (Triphone or Triphone) is used as the WFST network in the embodiment provided by the present application. The basic unit, the corresponding knowledge source for constructing the WFST search space includes: a triphone state tied column table, a pronunciation dictionary, and a language model.

The triphone state binding field table usually includes a binding relationship between each of the three phonemes based on the pronunciation characteristics. Generally, when the acoustic model is trained in the triphone element, the number of possible combinations of the triphones is large. In order to reduce the requirements of the training materials, it is usually possible to cluster different triphones under the maximum likelihood criterion based on the pronunciation characteristics and using the decision tree clustering method, and bind the triphones with the same pronunciation characteristics to the binding function by using the bundling technique. Together, in order to perform parameter sharing, the triphone state state linkage table is obtained; the pronunciation dictionary usually includes a correspondence between phonemes and words, and is a bridge between the acoustic layer (physical layer) and the semantic layer. Correlating the content of the acoustic layer with the content of the semantic layer; the language model provides knowledge of the language structure and is used to calculate the probability of the word sequence appearing in natural language. In the specific implementation, n yuan is usually used. (n-gram) A grammar language model that can be modeled by the possibility that statistical words follow each other.

When using the WFST network constructed based on the above knowledge source for speech recognition, in order to drive the WFST to perform the required search, the feature vector sequence of the speech signal to be recognized may be extracted first, and then the pre-trained model is used to calculate the corresponding feature vector corresponding to each The probability of the triphone, and performing a decoding operation in the WFST search space according to the probability of each of the three phonemes, to obtain a sequence of words corresponding to the speech signal to be recognized.

It should be noted that, in the embodiment provided by the present application, a context-dependent triphone is used as a basic unit of the WFST network. In other embodiments, other voice units may also be used as a basic unit of the WFST network, for example, a single Phoneme, or triphone status. Different basic units are used. When constructing the search space and when there is a certain difference according to the feature vector computer rate, for example, the triphone state is used as the basic unit, then the HMM-based acoustic model can be merged when constructing the WFST network. When speech recognition is performed, the probability that the feature vector corresponds to each triphone state can be calculated. The foregoing is a modification of the specific implementation manner. As long as the user preset information is included in the process of constructing the search space, the technical solution of the present application can also be implemented without departing from the technical core of the present application. Within the scope of protection of the application.

Hereinafter, embodiments of the present application will be described in detail. Please refer to FIG. 1, which is a flowchart of an embodiment of a speech recognition method of the present application. The method includes the steps 101 to 104. In a specific implementation, in order to improve the execution efficiency, the related preparation work (this stage may also be referred to as a preparation stage) may be completed before the step 101, and a class-based language model is generated. A structured WFST and an acoustic model for speech recognition are provided to prepare for the execution of step 101. The preparation phase will be described in detail below.

In the preparation phase, the language model can be trained in such a manner that the preset named entity in the text for training the language model is replaced with a label corresponding to the preset topic category, and the language model is trained using the text. The named entity generally refers to an entity having a specific category in the text, such as a person name, a song name, an institution name, a place name, and the like.

The following is an example of making a call application: the default subject category is a call, the corresponding label is “$CONTACT”, and the preset named entity is a person name. Then, when the language model is pre-trained, the name of the person in the training text can be replaced. For the corresponding label, for example, replace "Xiaoming" in "I want to call Xiaoming" with "$CONTACT", and then get the training text as "I want to call $CONTACT". A text-based language model after the above entity replacement is used to obtain a class-based language model. On the basis of training to obtain the above language model, it is also possible to generate a WFST based on a language model in advance, hereinafter referred to as a WFST of a G structure.

Preferably, in order to reduce the size of the WFST of the language model and the corresponding G structure, when the language model is pre-trained, the text for the preset topic category (also referred to as class-based training text) may be selected for training. For example, if the preset theme category is a call, the text for the preset theme category may include: I want to call Xiao Ming, call Xiao Ming, and the like.

Considering the diversity of client devices and applications that use voice as a human-computer interaction medium, two or more topic categories can be preset, and class-based language models are pre-trained and built for each topic category. G structure WFST based on the language model.

In the preparation stage, a WFST based on a pronunciation dictionary may be pre-built, a WFST referred to as an L structure, and a WFST based on a triphone-state state-binding field table, referred to as a WFST of a C structure, and in a preset manner for each of the above WFSTs Perform targeted and selective merge operations. For example, you can merge the WFST of the C structure and the L structure into the WFST of the CL structure, or merge the WFST of the L structure and the G structure into the WFST of the LG structure, or The WFST of the C structure, the L structure, and the G structure is merged into the WFST of the CLG structure. In this embodiment, the WFST of the CL structure and the WFST of the G structure are generated in the preparation phase (for the description of the merge operation, refer to the related text in step 101).

In addition, an acoustic model for speech recognition can also be pre-trained in the preparation phase. In this embodiment, each triphone is characterized by an HMM (Hidden Markov Model), and the implicit state of the HMM is a state in the triphone (each triphone usually contains three states) The GMM (Gaussian mixture model) model is used to determine the transmitter rate of each eigenvector in each implicit state of the HMM. The eigenvectors extracted from a large amount of speech data are used as training samples, and the Baum-Welch algorithm is used to learn. The parameters of the GMM model and the HMM model can obtain a GMM model corresponding to each state and an HMM model corresponding to each triphone. In a subsequent step 103, the pre-trained GMM and HMM models can then be used to calculate the probability that the feature vector corresponds to each triphone.

In order to improve the accuracy of speech recognition, the present embodiment replaces the GMM model with a DNN (Deep Neural Networks) model for speech recognition, and correspondingly, it can be pre-trained for input based on the feature vector in the preparation stage. A DNN model corresponding to the probability of each triphone state is output. In a specific implementation, on the basis of training the GMM and the HMM model, a label corresponding to each triphone state may be added to the training sample by forcibly aligning the training samples, and the labeled training sample is trained to obtain the DNN model.

It should be noted that, in the specific implementation, since the calculation amount in the preparation phase is relatively large, the requirements on the memory and the calculation speed are relatively high, and therefore the operation in the preparation phase is usually completed on the server side. In order to complete the voice recognition function without the network access environment, the method provided by the present application is usually implemented on the user equipment, so each WFST generated in the preparation phase and each model used for calculating the acoustic probability may be Pre-installed into the client device, for example: packaged with the application and installed to the client.

So far, the preparation phase involved in the embodiment has been described in detail, and the specific steps 101 to 104 of the present embodiment will be described in detail below.

Step 101: Generate, by using a preset voice knowledge source, a search space for decoding a voice signal that includes preset information of the UE.

This step builds the WFST search space to prepare for subsequent speech recognition. In the specific implementation, this step is usually performed in the startup phase (also referred to as the initialization phase) of the client application using voice as the human-computer interaction medium, and the pre-generated at least language-based weighting is limited by means of replacing the label. The state preset information corresponding to the preset theme category is added to the state converter, and a single weighted finite state converter based on the triphone state tied field table, the pronunciation dictionary, and the language model is obtained.

The processing of this step may include the following steps 101-1 to 101-4, which will be further described below in conjunction with FIG. 2.

Step 101-1: Determine a preset topic category to which the to-be-identified voice signal belongs.

In a specific implementation, the preset preset topic category may be determined according to a client type that collects the voice signal, or an application. The preset theme categories include: making a call, sending a short message, playing a music piece, setting a command, or other subject categories related to the application scene. The user preset information corresponding to making a call or sending a short message includes: a contact name in the address book; the user preset information corresponding to the played music includes: a music name in the music library; and a user corresponding to the setting instruction The preset information includes: instructions in the command set; the topic categories related to other application scenarios may also correspond to the user preset information involved in the application scenario, and are not described here.

For example, for a smart phone, the preset theme category to which the to-be-identified voice signal belongs may be determined according to the type of the client: making a call or sending a short message, and the corresponding user preset information is the contact name in the address book; for a smart speaker, It can be determined that the theme category is: playing the music, the corresponding user preset information is the music name in the music library; for the robot, the theme category can be determined as: setting instruction, and the corresponding user preset information is an instruction in the instruction set.

Considering that the user equipment can simultaneously have multiple applications using voice as a human-machine interaction medium, different applications involve different user-preset information, for example, a smart phone can also install a music player based on voice interaction. In the case, the preset theme category to which the to-be-identified voice signal belongs may be determined according to the currently launched application.

Step 101-2: Select a G structure WFST that is generated in advance and corresponds to the preset topic category.

For the case where there are multiple preset topic categories, a plurality of G structures WFST are usually generated in the preparation phase, and each G structure WFST corresponds to a different preset topic category. This step selects the G structure WFST corresponding to the preset topic category determined in step 101-1 from the plurality of G structures WFST generated in advance.

Step 101-3: Add user preset information to the selected G structure WFST by replacing the corresponding label with the user preset information corresponding to the preset theme category.

When training the class-based language model for each preset topic category in the preparation phase, the preset named entity in the training text is replaced with a label corresponding to the corresponding topic category, for example, the topic category is to make a call or send a text message, and the training text is The name of the person in the box is replaced with the "$CONTACT" tag, and the theme category is to play the music, and the name of the music in the training text is replaced with the "$SONG" tag. Therefore, the generated WFST of the G structure usually includes the corresponding theme category. Label information. In this step, the user preset information corresponding to the preset theme category determined in step 101-1 is used, and the corresponding label in the G structure WFST selected in step 101-2 is replaced, thereby adding a user terminal to the selected G structure WFST. The purpose of the preset information.

For example, if the subject category is to make a call or send a short message, the "$CONTACT" tag in the G structure WFST can be replaced with the contact name in the local address book of the client, such as "Zhang San", "Li Si", etc.; If the category is to play a song, you can replace the "$SONG" tag in the G structure WFST with the song name in the local library of the client, such as "The Volunteer March". A specific alternative can be implemented by replacing the state transition link corresponding to the tag with a plurality of sets of parallel state transition links. Please refer to FIG. 3 and FIG. 4 for an example of replacing with the contact person in the client address book, wherein FIG. 3 is a schematic diagram of the G structure WFST before replacement, and FIG. 4 is a view of "Zhang San" in the address book. Schematic diagram of the G structure WFST obtained after the replacement of "Li Si".

In step 101-4, the WFST of the G structure with the user preset information added is merged with the WFST of the pre-generated CL structure to obtain a single WFST network.

In this embodiment, the knowledge source used for speech recognition involves content from the language layer (language model) to the physical layer (triphone state linkage field table). The task of this step is to merge different levels of WFST (also Called together, combined, to get a single WFST network.

For both WFSTs, the basic condition for merging is that one of the WFST output symbols is a subset of another WFST input symbol set. Under the premise of satisfying the above requirements, if two WFSTs, for example, A and B, respectively, are integrated into a new WFST:C, then each state of C consists of the state of A and the state of B, each of C Success paths are composed of A's success path P _a and B's success path P _b , the input is i[P]=i[P _a ], and the output is o[P]=o[P _b ], The weighted value is obtained by performing corresponding operations on the weighted values of P _a and P _b , and the resulting C contains the finite state converter characteristics common to A and B and the search space. In the specific implementation, the merge operation of the two WFSTs can be completed by using the merge algorithm provided by the OpenFst library.

Specifically, in this embodiment, it can be understood that the WFST of the L structure can be regarded as the correspondence between the mono phoneme and the word, and the WFST of the C structure establishes the correspondence between the triphone and the mono phoneme, and the output and the L structure. The input of the WFST corresponds to each other and can be merged. In the preparation stage of this embodiment, the WFST of the CL structure is obtained through the merge. In this step, the WFST of the CL structure and the preset information of the UE are added in step 101-3. The G structure WFST is merged, and a WFST network with the input triphone probability and output as a word sequence is obtained, thereby integrating the WFSTs at different levels corresponding to different knowledge sources into a single WFST network, which constitutes a use. A search space for speech recognition.

Preferably, in order to speed up the merging speed of the CL structure WFST and the G structure WFST, and reduce the time-consuming initialization, the embodiment does not adopt the conventional WFST merging method when performing the merging operation, but adopts a prediction-based merging method. (lookahead merge method). The lookahead merging method, that is, during the merging process of the two WFSTs, by predicting the future path, determining whether the currently performed merge operation will result in a non-coaccessible state, and if so, blocking the current state The operation does not perform subsequent merge operations. Through the prediction, the unnecessary merge operation can be terminated early, which not only saves the merge time, but also reduces the size of the finally generated WFST and reduces the occupation of the storage space. In the specific implementation, the screening function with the lookahead function provided by the OpenFst library can be used to implement the above prediction filtering function.

Preferably, in order to speed up the merging speed of the CL structure WFST and the G structure WFST, the vocabulary used in the pre-training of the language model in this embodiment is consistent with the words contained in the pronunciation dictionary. In general, the number of words in the vocabulary is usually larger than the number of words in the pronunciation dictionary, and the number of words in the vocabulary is directly related to the size of the WFST of the G structure, if the WFST of the G structure is relatively large, and CL The WFST of the structure is more time-consuming to be merged. Therefore, when training the language model in the preparation stage, the scale of the vocabulary is reduced, and the words in the vocabulary are consistent with the words in the pronunciation dictionary, thereby achieving the reduced CL structure. The effect of the merge time of WFST and G structure WFST.

At this point, through the steps 101-1 to 101-4, the initialization process of the technical solution is completed, and the WFST search space including the preset information of the user end is generated.

It should be noted that, in this embodiment, the WFST of the CL structure is merged in advance in the preparation phase, and the WFST of the G structure is generated. In this step 101, the user preset information is added to the G structure WFST, and the CL structure and The G structure merges to get a single WFST. In other implementations, other merge policies may also be adopted. For example, the WFST merge of the LG structure is pre-completed in the preparation phase. In this step 101, the user preset information is added to the WFST, and then generated in the preparation phase. The C-structure WFST is merged; or, the WFST merge of the CLG structure is directly completed in the preparation phase, and it is also possible to add the user-preset information to the WFST in this step 101. Considering that the WFST generated in the preparation phase occupies the storage space of the client, in the application scenario where there are multiple class-based language models (the corresponding WFST with multiple G structures), if each G structure WFST and other in the preparation phase The WFST merges and occupies a large storage space. Therefore, the merge mode adopted in this embodiment is a preferred implementation manner, which can reduce the occupation of the client storage space by the WFST generated in the preparation phase.

Step 102: Extract a sequence of feature vectors of the speech signal to be recognized.

The speech signal to be recognized is usually a time domain signal. In this step, a sequence of feature vectors capable of characterizing the speech signal is obtained through two processes of framing and extracting feature vectors, which will be further described below with reference to FIG. 5.

Step 102-1: Perform frame processing on the speech signal to be recognized according to a preset frame length to obtain a plurality of audio frames.

In a specific implementation, the frame length may be preset according to requirements, for example, may be set to 10 ms, or 15 ms, and then the speech signal to be recognized is segmented frame by frame according to the frame length, thereby dividing the speech signal into a plurality of audio frames. Depending on the segmentation strategy used, adjacent audio frames may or may not overlap or overlap.

Step 102-2: Extract a feature vector of each audio frame to obtain the feature vector sequence.

After the speech signal to be recognized is sliced into a plurality of audio frames, the feature vector capable of characterizing the speech signal can be extracted frame by frame. Since the description ability of the speech signal in the time domain is relatively weak, the Fourier transform can usually be performed for each audio frame, and then the frequency domain feature is extracted as the feature vector of the audio frame. For example, the MFCC (Mel Frequency Cepstrum Coefficient-Mel can be extracted). Frequency cepstral coefficient) feature, PLP (Perceptual Linear Predictive) feature, or LPC (Linear Predictive Coding) feature.

The following takes the MFCC feature of an audio frame as an example to further describe the extraction process of the feature vector. First, the time domain signal of the audio frame is transmitted through FFT (Fast Fourier Transformation) to obtain corresponding spectrum information, and the spectrum information is transmitted through the Mel filter group to obtain a Mel spectrum, and cepstrum analysis is performed on the Mel spectrum. The core generally uses DCT (Discrete Cosine Transform) to perform inverse transform, and then takes a preset N coefficients (for example, N=12 or 38), and then obtains the feature vector of the audio frame: MFCC feature. Each audio frame is processed in the above manner, and a series of feature vectors characterizing the speech signal, that is, the feature vector sequence described in the present application, can be obtained.

Step 103: Calculate the probability that the feature vector corresponds to the basic unit of the search space.

In this embodiment, the basic unit of the WFST search space is a triphone, so this step calculates the probability that the feature vector corresponds to each triphone. In order to improve the accuracy of speech recognition, the present embodiment uses the HMM model and the DNN model with powerful feature extraction capabilities to calculate the probability. In other embodiments, other methods may also be used, for example, using traditional GMM and HMM models. The probability that the probability can also achieve the technical solution of the present application is also within the protection scope of the present application.

In a specific implementation, the probability that the feature vector corresponds to each triphone can be further calculated on the basis that the calculated feature vector corresponds to each triphone state. The processing of this step is further described below with reference to FIG. 6.

Step 103-1: Calculate the probability that the feature vector corresponds to each triphone state by using the pre-trained DNN model.

In the preparation stage of the embodiment, the DNN model has been pre-trained. In this step, the feature vector extracted in step 102 is used as the input of the DNN model, and the probability that the feature vector corresponds to each triphone state can be obtained. For example, if the number of triphones is 1000 and each triphone contains 3 states, then there are a total of 3000 triphone states. In this step, the DNN model outputs: the feature vector corresponds to the probability of each of the 3000 triphone states.

Preferably, since the amount of calculation involved in adopting the DNN model is usually large, the present embodiment improves the speed of calculation using the DNN model by utilizing the data parallel processing capability provided by the hardware platform. For example, currently more embedded devices and mobile devices are used in ARM architecture platforms. On most current ARM platforms, there are SIMD (single instruction multiple data) NEON instruction sets. The set can process multiple data in one instruction, and has certain data parallel processing capability. In this embodiment, the vectorization programming can form a programming generic of single instruction flow and multiple data flow, so that the hardware platform can be fully utilized. The data processing capability provided is provided to accelerate the calculation of DNN.

When implementing the technical solution of the present application on the client device, in order to match the hardware capability of the client, the size of the DNN model is usually reduced, which often leads to a decrease in the accuracy of the DNN model, and the ability to recognize different voice content. Will go down. In this embodiment, since the hardware acceleration mechanism is utilized, the scale of the DNN model can be reduced or minimized, so that the accuracy of the DNN model can be maximized and the recognition accuracy can be improved.

Step 103-2: Calculate the probability that the feature vector corresponds to each triphone by using the pre-trained HMM model according to the probability that the feature vector corresponds to the triphone state.

In the preparation stage, the HMM model for each triphone is already trained. This step is based on the probability that several consecutive feature vectors correspond to the triphone states, and the HMM model is used to calculate the transfer probability corresponding to each triphone. The feature vector corresponds to the probability of each triphone.

The calculation process is actually a process of calculating the corresponding transfer probability according to the propagation process of successive feature vectors on each HMM. The following is a further description of the calculation process for calculating the probability of a certain three phonemes (including three states). Where pe(i,j) represents the transmitter rate of the ith frame feature vector in the jth state, and pt(h,k) represents the transfer probability from the h state to the k state: 1) the eigenvector of the first frame Corresponding to state 1 of the corresponding HMM, having a transmitter rate pe(1,1); 2) for a feature vector of the second frame, if transitioning from state 1 to state 1 of the HMM, the corresponding probability is pe(1,1) ) *pt(1,1)*pe(2,1), if transitioning from state 1 to state 2, the corresponding probability is pe(1,1)*pt(1,2)*pe(2,2), Judging whether to transfer to state 1 or state 2 according to the above probability; 3) using the similar calculation method for the feature vector of the third frame and the feature vector of the subsequent frame until the transition from state 3 to the propagation of the HMM Ending, thereby obtaining the probability that the feature vector of successive multi-frames is directed to the HMM, that is, the probability of corresponding to the triphone of the HMM representation.

For the continuously input feature vector, the transfer probability spread on each HMM is calculated in the above manner, thereby obtaining the probability corresponding to each triphone.

Step 104: Perform a decoding operation in the search space by using the probability, and obtain a sequence of words corresponding to the sequence of feature vectors.

According to the probability that the feature vector outputted in step 103 corresponds to each triphone, a decoding operation is performed in the WFST network to obtain a sequence of words corresponding to the sequence of feature vectors. This process is typically a search process that performs a graph search and finds the path with the highest score. The commonly used search method is the Viterbi algorithm, which has the advantage of using a dynamic programming method to save computational complexity and time synchronization decoding.

Considering that in the actual decoding process, due to the huge search space, the calculation of the Viterbi algorithm is still very large. In order to reduce the amount of calculation and increase the calculation speed, the possible subsequent paths of all paths are not expanded in the decoding process. Rather, only those paths that are near the best path are developed, that is, the appropriate pruning strategy can be used to improve the search efficiency in the process of searching with the Viterbi algorithm. For example, the Viterbi column search algorithm can be used or It is a bar graph pruning strategy.

So far, the word sequence corresponding to the feature vector sequence is obtained by decoding, that is, the recognition result corresponding to the speech signal to be recognized is acquired. Since the user preset information is added when the search space for voice recognition is constructed in step 101, the voice recognition process can generally accurately identify the voice content related to the local information of the user terminal.

In order to further ensure the accuracy of the word sequence obtained through the above decoding process, the present embodiment further provides a preferred embodiment: the information is preset through the user terminal. Text matching is performed to verify the accuracy of the word sequence, and corresponding speech recognition results are generated according to the verification result.

In a specific implementation, the above preferred embodiment may include the steps 104-1 to 104-4 listed below, which will be further described below with reference to FIG.

Step 104-1: Select, from the sequence of words, a word to be verified corresponding to the preset information of the UE.

For example, for the calling application, the preset information of the user terminal is the name of the contact person in the address book, and the result of the voice recognition is the word sequence “calling Xiao Ming”, then the method of matching with the template or the syntax parsing process may be It is determined that "Xiao Ming" in the sequence of words is a word to be verified corresponding to the preset information of the client.

Step 104-2: Search for the to-be-verified word in the user preset information. If found, determine the transmission accuracy verification, perform step 104-3, otherwise perform step 104-4.

In this step, by performing an exact match on the text level, it is determined whether the to-be-verified word belongs to the corresponding user-preset information, thereby verifying the accuracy of the word sequence.

Still using the example in step 104-1, this step searches the user address book for the presence of the "Xiaoming" contact, that is, whether the information related to the name of the contact in the address book contains the string "Xiaoming". If yes, it is determined that the accuracy verification is passed, and step 104-3 is continued; otherwise, the process proceeds to step 104-4.

Step 104-3, the word sequence is used as a speech recognition result.

Executing this step, the word to be verified included in the sequence of words obtained by decoding is consistent with the preset information of the client, and the sequence of words can be output as a speech recognition result, thereby triggering the use of the speech recognition result. The application performs the appropriate action.

Step 104-4, correcting the word sequence by using a pinyin-based fuzzy matching method, and using the corrected word sequence as a speech recognition result.

Executing this step, generally speaking, the word to be verified included in the sequence of words obtained by decoding is inconsistent with the preset information of the client. If the sequence of words is output as a speech recognition result, the related application Normally, the correct operation cannot be performed, so in this case, the word sequence can be corrected as necessary through the fuzzy matching of the pinyin level.

In a specific implementation, the foregoing correcting function may be implemented by: searching a pronunciation dictionary, converting the to-be-verified word into a to-be-verified pinyin sequence, and converting each word in the user-preset information into a comparison. a pinyin sequence, and then calculating a similarity between the to-be-verified pinyin sequence and each of the aligned pinyin sequences, and selecting a word sorted according to the similarity from high to low from the user-preset information, and finally using The selected word replaces the word to be verified in the sequence of words to obtain the corrected sequence of words.

In the specific implementation, the similarity between the two pinyin sequences can be calculated in different ways. In this embodiment, the similarity is calculated based on the edit distance, for example, the edit distance between the two pinyin sequences is compared with the phase 1 The reciprocal of the summation is used as the similarity. The Edit Distance refers to the minimum number of editing operations required between one string and another, and the editing operation includes replacing one character with another character, inserting one Character, delete a character, in general, the smaller the edit distance, the greater the similarity between the two strings.

Still using the example in the above step 104-1, the word sequence is "calling Xiaoming", and the word to be verified is "Xiaoming". If the "Xiaoming" is not found in the contact person of the user address book, the pronunciation can be searched. The dictionary converts Xiao Ming into the pinyin sequence "xiaoming" to be verified, and converts each contact name in the address book into the corresponding pinyin sequence, that is, compares the pinyin sequence, and then calculates "xiaoming" and each comparison in turn. The edit distance between the pinyin sequences, and then select the contact name corresponding to the aligned pinyin sequence with the smallest edit distance (the highest similarity) (for example, "Xiao Min" corresponding to "xiaomin"), replacing the word sequence The word to be verified is completed, thereby correcting the sequence of words, and the corrected sequence of words can be used as the final speech recognition result.

In the specific implementation, the similarity between the Pinyin sequence to be verified and each of the Pinyin sequences can be calculated first and sorted according to the similarity from high to low, and several (eg, three) aligned Pinyin are selected. The words corresponding to the sequence are then presented to the user by means of screen output, etc., by which the user selects the correct word and replaces the word to be verified in the word sequence according to the word selected by the user.

So far, the specific implementation manner of the speech recognition method provided by the present application is described in detail through the above steps 101-104. For ease of understanding, please refer to FIG. 8 , which is an overall frame diagram of the speech recognition process provided by this embodiment. The dotted line frame corresponds to the preparation phase described in this embodiment, and the solid line frame corresponds to a specific speech recognition process.

It should be noted that the step 101 described in this embodiment may be executed once every time the client application that uses voice as the interaction medium starts, that is, each time the startup includes the user preset information, The search space in which the voice signal is decoded may also generate the search space and store the subsequent update mode only when the client application is first started, thereby reducing the time for generating the search space each time the application starts. Overhead (you can directly use the previously generated search space) to improve the efficiency of speech recognition and improve the user experience.

In addition, the method provided by the present application is generally implemented on a client device, including: a smart mobile terminal, a smart speaker, a robot, or other device that can run the method, and the embodiment is described at the user end. Specific embodiments of the methods provided herein are implemented. However, in other embodiments, the method provided by the present application may also be implemented in an application scenario based on a client and a server mode, in which case each WFST generated in the preparation phase and a model for acoustic probability calculation need not be used. Pre-installed into the client device, each time the client application starts, the corresponding user preset information can be uploaded to the server, and the subsequent acquired voice signal to be recognized is also uploaded to the server, and the server side is The method provided by the present application is implemented, and the decoded sequence of words is transmitted back to the user end, and the technical solution of the present application can also be implemented, and corresponding beneficial effects are obtained.

In summary, the voice recognition method provided by the present application is relatively accurate when the voice signal collected by the user end is recognized because the user terminal preset information is included in the search space for decoding the voice signal. The local information related to the user end is identified, so that the accuracy of the speech recognition can be improved, and the user experience is improved.

In particular, the method provided by the present application is used for voice recognition on the user equipment. Since the local information of the user terminal is added, the problem of the accuracy of the recognition due to the probability calculation model and the reduction of the search space size can be compensated to some extent. Therefore, the requirement for voice recognition without network access environment can be satisfied, and a certain recognition accuracy can also be achieved. Further, after the word sequence is decoded, the accuracy of the speech recognition can be further improved by using the matching verification scheme based on the text level and the pinyin level given in the embodiment. The actual test results show that the character error rate (CER) of the conventional speech recognition method is about 20%, and using the method provided by this application, the character error rate is below 3%. The above data fully demonstrates the method. The beneficial effects are significant.

In the above embodiment, a voice recognition method is provided. Correspondingly, the present application further provides a voice recognition device. Please refer to FIG. 9, which is a schematic diagram of an embodiment of a speech recognition apparatus of the present application. Since the device embodiment is substantially similar to the method embodiment, the description is relatively simple, and the relevant portions can be referred to the description of the method embodiment. The device embodiments described below are merely illustrative.

The voice recognition device of the embodiment includes: a search space generating unit 901, configured to generate, by using a preset voice knowledge source, a search space for decoding a voice signal including user preset information; feature vector extraction a unit 902, configured to extract a sequence of feature vectors of the to-be-recognized speech signal, a probability calculation unit 903, configured to calculate a probability that the feature vector corresponds to the basic unit of the search space, and a decoding search unit 904, configured to input the probability at the A decoding operation is performed in the search space to obtain a sequence of words corresponding to the sequence of feature vectors.

Optionally, the search space generating unit is configured to: add, by using a replacement label, a user preset information corresponding to the preset theme category to the pre-generated at least language-based weighted finite state converter, and Obtaining a single weighted finite state converter based on a triphone state tied field table, a pronunciation dictionary, and a language model; the language model is pre-generated by a language model training unit, and the language model training unit is used The preset named entity in the text of the training language model is replaced with a label corresponding to the preset theme category, and the language model is trained using the text.

Optionally, the search space generating unit includes: a first user information adding subunit, configured to add, according to the manner of replacing the label, a pre-generated language model-based weighted finite state converter corresponding to the preset topic category User preset information; a weighted finite state converter merging subunit for using the weighted finite state converter to which the user preset information is added, and a pre-generated triphone-based state binding field table The weighted finite state converter of the pronunciation dictionary is combined to obtain the single weighted finite state converter.

Optionally, the decoding space generating unit includes: a second user information adding subunit, configured to add a preset theme category to the pre-generated at least language model-based weighted finite state converter by replacing the label Corresponding user preset information; a unified weighted finite state converter obtaining subunit, configured to obtain a triphone based state binding field table, a pronunciation dictionary, and after the second user information adding subunit completes the adding operation, And a single weighted finite state converter of the language model; wherein the second user information adding subunit comprises: a topic determining subunit, configured to determine a preset topic category to which the to-be-identified speech signal belongs; weighted finite state converter selection a subunit, configured to select the at least language-based weighted finite state converter corresponding to the preset topic category, and a label replacement subunit, configured to correspond to the preset topic category The manner in which the user preset information replaces the corresponding label to the selected weighted finite state converter Plus default client information.

Optionally, the topic determining subunit is specifically configured to determine, according to a type of the client that collects the voice signal, or an application, the belonging preset topic category.

Optionally, the weighted finite state converter combining subunit is specifically configured to perform a combining operation by using a prediction based method, and obtain the single weighted finite state converter.

Optionally, the probability calculation unit includes: a triphone state probability calculation sub-unit, configured to calculate a probability that the feature vector corresponds to each triphone state by using a pre-trained DNN model; and a triphone probability calculation sub-unit for The vector corresponds to the probability of each of the triphone states, and the pre-trained HMM model is used to calculate the probability that the feature vector corresponds to each triphone.

Optionally, the feature vector extraction unit includes: a framing sub-unit, configured to perform framing processing on the speech signal to be recognized according to a preset frame length, to obtain a plurality of audio frames; and a feature extraction sub-unit, configured to extract each audio The feature vector of the frame results in the sequence of feature vectors.

Optionally, the device includes: an accuracy verification unit, configured to verify, after the decoding search unit obtains a sequence of words corresponding to the sequence of feature vectors, verify the word sequence by performing text matching with the preset information of the UE Accuracy, and generate corresponding speech recognition results based on the verification results.

Optionally, the accuracy verification unit includes: a to-be-verified word selection sub-unit, configured to select a to-be-verified word corresponding to the user-preset information from the word sequence; and the search sub-unit is used in the Searching for the to-be-verified word in the user-preset information; the recognition result confirmation sub-unit is configured to, after the finding sub-unit finds the to-be-verified word, determine to pass the accuracy verification, and the word sequence is As a speech recognition result, the recognition result correction subunit is configured to, after the search subunit does not find the to-be-verified word, correct the word sequence through a pinyin-based fuzzy matching manner, and use the corrected word sequence as a speech Identify the results.

Optionally, the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, configured to convert the to-be-verified word into a to-be-verified pinyin sequence; and a comparison pinyin sequence conversion subunit, Each word in the preset information of the client is converted into a comparison pinyin sequence, and a similarity calculation selection sub-unit is configured to sequentially calculate the similarity between the to-be-verified pinyin sequence and each of the comparison pinyin sequences, and from the Selecting, in the user preset information, a word that is sorted according to the similarity from high to low; the word to be verified is replaced by a subunit for replacing the word to be verified in the word sequence with the selected word, and the correction is obtained. After the word sequence.

In addition, the present application further provides another voice recognition method. Please refer to FIG. 10 , which is a flowchart of another embodiment of the voice recognition method provided by the present application. Again, the following highlights the differences. Another speech recognition method provided by the present application includes:

Step 1001: Obtain a sequence of words corresponding to the to-be-identified speech signal by decoding.

For speech recognition, the process of decoding is the process of searching in the search space for speech recognition to obtain the best word sequence corresponding to the speech signal to be recognized. The search space may be a WFST network based on various knowledge sources, or may be other forms of search space; the search space may include user preset information, or may not include user preset information, and the embodiment is This is not specifically limited.

Step 1002: Verify the accuracy of the word sequence by performing text matching with the preset information of the client, and generate a corresponding voice recognition result according to the verification result.

The step of the present invention includes the following steps: selecting a word to be verified corresponding to the preset information of the user from the sequence of words; searching for the word to be verified in the preset information of the user terminal; The accuracy verification is performed, and the word sequence is used as a speech recognition result; otherwise, the word sequence is corrected by a pinyin-based fuzzy matching method, and the modified word sequence is used as a speech recognition result.

The correcting the word sequence by the pinyin-based fuzzy matching method includes: converting the to-be-verified word into a to-be-verified pinyin sequence; and converting each word in the user-side preset information into a comparison pinyin sequence; And sequentially calculating a similarity between the to-be-verified pinyin sequence and each of the aligned pinyin sequences, and selecting, from the user-preset information, a word that ranks according to the similarity from high to low; using the selected word The word to be verified in the sequence of words is replaced to obtain the corrected sequence of words.

The converted pinyin sequence can be implemented by searching a pronunciation dictionary, and the similarity can be calculated according to an editing distance between two pinyin sequences.

The method provided by the present application is generally applied to an application that uses voice as an interactive medium. The voice to be recognized collected by such an application may involve user-side information, and the method provided by the present application transmits the decoded word sequence. Text matching with the user preset information can verify the accuracy of the word sequence, thereby providing a basis for necessary correction of the word sequence. Further, by using the fuzzy matching based on the pinyin layer, the word sequence can be corrected, thereby improving the accuracy of the speech recognition.

In the above embodiment, another voice recognition method is provided, and correspondingly, the present application also provides another voice recognition device. Please refer to FIG. 11 , which is a schematic diagram of an embodiment of another speech recognition apparatus of the present application. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment. The device embodiments described below are merely illustrative.

The speech recognition apparatus of the embodiment includes: a word sequence obtaining unit 1101, configured to obtain a word sequence corresponding to the to-be-recognized speech signal by decoding; the word sequence verification unit 1102 is configured to perform text matching by using the preset information of the user end. The accuracy of the word sequence is verified, and corresponding speech recognition results are generated according to the verification result.

Optionally, the word sequence verification unit includes: a to-be-verified word selection sub-unit, configured to select a to-be-verified word corresponding to the user-preset information from the word sequence; and the search sub-unit is used in the Searching for the to-be-verified word in the user-preset information; the recognition result confirmation sub-unit is configured to, after the finding sub-unit finds the to-be-verified word, determine to pass the accuracy verification, and the word sequence is As a speech recognition result, the recognition result correction subunit is configured to, after the search subunit does not find the to-be-verified word, correct the word sequence through a pinyin-based fuzzy matching manner, and use the corrected word sequence as a speech Identify the results.

Optionally, the recognition result correction subunit includes: a pinyin sequence conversion subunit to be verified, configured to convert the to-be-verified word into a to-be-verified pinyin sequence; and a comparison pinyin sequence conversion subunit, Each word in the preset information of the client is converted into a comparison pinyin sequence, and a similarity calculation selection sub-unit is configured to sequentially calculate the similarity between the to-be-verified pinyin sequence and each of the comparison pinyin sequences, and from the Selecting, in the user preset information, a word that is sorted according to the similarity from high to low; the word to be verified is replaced by a subunit for replacing the word to be verified in the word sequence with the selected word, and the correction is obtained. After the word sequence.

The present application is disclosed in the above preferred embodiments, but it is not intended to limit the present application, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application. The scope of protection should be based on the scope defined by the claims of the present application.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, a network interface, and memory.

The memory may include non-permanent memory, random access memory (RAM) and/or non-volatile memory in computer readable media, such as read only memory (ROM) or flash memory (flash) RAM). Memory is an example of a computer readable medium.

1. Computer readable media including both permanent and non-permanent, removable and non-removable media can be stored by any method or technique. Information can be computer readable instructions, data structures, modules of programs, or other materials. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), and other types of random access memory (RAM). Read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM only, digitally versatile A compact disc (DVD) or other optical storage, magnetic cassette, magnetic tape storage or other magnetic storage device or any other non-transportable medium can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media, such as modulated data signals and carrier waves.

2. Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Thus, the present application can take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining the software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable code therein. .

Claims (37)

 A voice recognition method, comprising: generating a search space for decoding a voice signal by using a preset voice knowledge source; and extracting a feature vector sequence of the voice signal to be recognized; The feature vector corresponds to the probability of searching for the basic unit of the space; at the probability of inputting, performing a decoding operation in the search space to obtain a sequence of words corresponding to the sequence of feature vectors.    The speech recognition method according to claim 1, wherein the search space comprises: a weighted finite state converter.    The speech recognition method according to claim 2, wherein the search space basic unit comprises: a context-dependent triphone; the preset knowledge source includes: a pronunciation dictionary, a language model, and a triphone state linkage column. Bit table.    The voice recognition method according to the third aspect of the patent application, wherein the predetermined voice knowledge source generates a search space for decoding a voice signal including user preset information, including: a method, adding, to a pre-generated at least language-based weighted finite state converter, user preset information corresponding to a preset topic category, and obtaining a single based on a triphone state state column table, a pronunciation dictionary, and a language model a weighted finite state converter; wherein the language model is pre-trained by replacing a preset named entity in the text for training the language model with a label corresponding to the preset topic category, and training the text using the text Language model.    The speech recognition method according to claim 4, wherein the method of replacing the label adds a user preset corresponding to the preset theme category to the pre-generated at least language-based weighted finite state converter Information, and a single weighted finite state converter based on a triphone state tied field table, pronunciation dictionary, and language model, including: adding a pre-generated language model-based weighted finite state converter by replacing the label User preset information corresponding to the preset theme category; the weighted finite state converter to which the user preset information is added, and the weighted finite state transition based on the pre-generated triphone-based state-binding field table and pronunciation dictionary The combiner is combined to obtain the single weighted finite state converter.    The speech recognition method of claim 4, wherein the text for training the language model is a text that points to the preset topic category.    The speech recognition method according to claim 4, wherein the number of the preset subject categories is at least two; the number of the language models, and the number of the at least language-based weighted finite state devices are respectively The number of the subject categories is consistent; the user preset information corresponding to the preset topic category is added to the pre-generated at least language-based weighted finite state converter by means of the replacement label, including: determining the to-be-identified voice signal a preset theme category; a pre-generated, at least language-based weighted finite state converter corresponding to the preset theme category; and replacing the corresponding user preset information corresponding to the preset theme category The way the tag is added to add the user preset information to the selected weighted finite state converter.    According to the voice recognition method of claim 7, wherein the determining the preset subject category to which the to-be-identified voice signal belongs is implemented by: determining the belonging according to the type of the client that collects the voice signal, or the application program. The default theme category.    According to the speech recognition method of claim 8, wherein the preset theme category includes: making a call or sending a short message, playing a music, or setting a command; and corresponding user preset information includes: The name of the contact, the name of the song in the library, or the instruction in the command set.    The speech recognition method of claim 5, wherein the merging operation comprises: merging using a prediction-based method.    The speech recognition method according to claim 4, wherein the vocabulary used to pre-train the language model is consistent with the words included in the pronunciation dictionary.    The speech recognition method according to claim 3, wherein the calculating feature vector corresponds to a probability of searching for a basic unit of the space, comprising: calculating a probability that the feature vector corresponds to each triphone state by using a pre-trained DNN model; The feature vector corresponds to the probability of each of the triphone states, and the pre-trained HMM model is used to calculate the probability that the feature vector corresponds to each triphone.    The speech recognition method according to claim 12, wherein the execution speed of the step of calculating the probability that the feature vector corresponds to each triphone state using the pre-trained DNN model is improved by: providing by using a hardware platform Data processing capabilities.    The speech recognition method according to any one of claims 1 to 13, wherein the extracting the feature vector sequence of the speech signal to be recognized comprises: performing frame processing on the speech signal to be recognized according to a preset frame length, Obtaining a plurality of audio frames; extracting feature vectors of the audio frames to obtain the feature vector sequence.    The speech recognition method according to claim 14, wherein the extracting the feature vector of each audio frame comprises: extracting an MFCC feature, a PLP feature, or an LPC feature.    The speech recognition method according to any one of claims 1 to 13, wherein after the word sequence corresponding to the feature vector sequence is obtained, performing the following operation: performing text by using the preset information of the user terminal Matching verifies the accuracy of the word sequence and generates corresponding speech recognition results based on the verification results.    According to the speech recognition method of claim 16, wherein the text is matched with the preset information of the user end to verify the accuracy of the word sequence, and the corresponding speech recognition result is obtained according to the verification result, including: Selecting a word to be verified corresponding to the preset information of the client in the sequence of words; searching for the to-be-verified word in the preset information of the user; if found, determining to verify by the accuracy, and using the sequence of words as a voice The result is identified; otherwise, the word sequence is corrected by the pinyin-based fuzzy matching method, and the corrected word sequence is used as the speech recognition result.    The speech recognition method according to claim 17, wherein the correcting the word sequence by using a pinyin-based fuzzy matching method comprises: converting the to-be-verified word into a to-be-verified pinyin sequence; Each word in the conversion is converted into a comparison pinyin sequence; the similarity between the to-be-verified pinyin sequence and each of the comparison pinyin sequences is calculated in turn, and the user-preset information is selected to be sorted according to the similarity from high to low. The word before; the word to be verified in the sequence of words is replaced with the selected word to obtain the corrected sequence of words.    The speech recognition method according to claim 18, wherein the similarity comprises: a similarity calculated based on an edit distance.    The voice recognition method according to any one of claims 1 to 13, wherein the method is implemented on a client device; the client device includes: a smart mobile terminal, a smart speaker, or a robot.    A voice recognition device, comprising: a search space generating unit, configured to generate, by using a preset voice knowledge source, a search space for decoding a voice signal including user preset information; and a feature vector extracting unit a feature vector sequence for extracting a speech signal to be recognized; a probability calculation unit configured to calculate a probability that the feature vector corresponds to the basic unit of the search space; and a decoding search unit configured to perform decoding in the search space at the probability of input Operation, obtaining a sequence of words corresponding to the sequence of feature vectors.    The speech recognition apparatus according to claim 21, wherein the search space generating unit is specifically configured to add and preset to a pre-generated at least language-based weighted finite state converter by replacing the label. a user-preset information corresponding to the topic category, and a single weighted finite state converter based on the triphone state-binding field table, the pronunciation dictionary, and the language model; the language model is pre-generated by the language model training unit, The language model training unit is configured to replace the preset named entity in the text for training the language model with a label corresponding to the preset theme category, and use the text to train the language model.    The speech recognition apparatus according to claim 22, wherein the search space generating unit comprises: a first user information adding subunit for weighting the pre-generated language model based on the manner of replacing the label Adding user preset information corresponding to the preset theme category to the state converter; and weighting the finite state converter merging subunit for using the weighted finite state converter to which the user preset information is added, and the pre-generated The single weighted finite state converter is obtained by combining the weighted finite state converters based on the triphone state tied field table and the pronunciation dictionary.    The speech recognition apparatus according to claim 22, wherein the decoding space generating unit comprises: a second user information adding subunit, configured to transmit the pre-generated at least language model based weighting by means of replacing the label The finite state converter adds a user preset information corresponding to the preset theme category; the unified weighted finite state converter acquisition subunit is configured to obtain the triphone based on the third user information adding subunit after the adding operation is completed. a state-binding field table, a pronunciation dictionary, and a single weighted finite state converter of the language model; wherein the second user information adding sub-unit includes: a topic determining sub-unit, configured to determine a preset to which the to-be-identified voice signal belongs a subject category; a weighted finite state converter selection subunit, configured to select the pre-generated, at least language-based weighted finite state converter corresponding to the preset topic category; a label replacement sub-unit for transmitting The manner in which the preset information of the preset theme category replaces the corresponding label, The selected weighted finite state transducer UE added preset information.    The speech recognition apparatus according to claim 24, wherein the theme determining sub-unit is specifically configured to determine the belonging preset topic category according to the type of the client that collects the voice signal or the application.    The speech recognition apparatus according to claim 23, wherein the weighted finite state converter merging subunit is specifically configured to perform a merging operation using a prediction based method and obtain the single weighted finite state converter.    The speech recognition device according to claim 21, wherein the probability calculation unit comprises: a triphone state probability calculation sub-unit, configured to calculate a probability that the feature vector corresponds to each triphone state by using the pre-trained DNN model; The triphone probability calculation sub-unit is configured to calculate the probability that the feature vector corresponds to each triphone by using the pre-trained HMM model according to the probability that the feature vector corresponds to the triphone state.    The speech recognition apparatus according to any one of claims 21 to 27, wherein the feature vector extraction unit comprises: a framing subunit for performing framing processing on the speech signal to be recognized according to a preset frame length Obtaining a plurality of audio frames; and extracting a sub-unit for extracting feature vectors of each audio frame to obtain the feature vector sequence.    The speech recognition apparatus according to any one of claims 21 to 27, further comprising: an accuracy verification unit, configured to transmit the word sequence corresponding to the feature vector sequence after the decoding search unit obtains The client preset information performs text matching to verify the accuracy of the word sequence, and generates a corresponding speech recognition result according to the verification result.    The speech recognition device according to claim 29, wherein the accuracy verification unit comprises: a to-be-verified word selection sub-unit, configured to select a to-be-verified word corresponding to the user-preset information from the word sequence. a search subunit for finding the to-be-verified word in the user preset information; a recognition result confirmation subunit, configured to, after the search subunit finds the to-be-verified word, determine to verify by the accuracy, and The word sequence is used as a speech recognition result; the recognition result correction sub-unit is configured to correct the word sequence through the pinyin-based fuzzy matching method after the search sub-unit does not find the to-be-verified word, and use the corrected word sequence as a speech Identify the results.    The speech recognition device according to claim 30, wherein the recognition result correction subunit comprises: a pinyin sequence conversion subunit to be verified, for converting the word to be verified into a pinyin sequence to be verified; a sequence conversion sub-unit, configured to convert each word in the preset information of the user end into a comparison pinyin sequence; the similarity calculation selection sub-unit is configured to sequentially calculate the between the pinyin sequence to be verified and each of the comparison pinyin sequences Similarity, and selecting, from the user preset information, a word that ranks according to the similarity from high to low; the word to be verified replaces the subunit, and replaces the word to be verified in the word sequence with the selected word. , get the corrected word sequence.    A voice recognition method, comprising: obtaining a sequence of words corresponding to a voice signal to be recognized through decoding; verifying the accuracy of the word sequence by performing text matching with the preset information of the user terminal, and generating a corresponding voice according to the verification result Identify the results.    According to the speech recognition method of claim 32, the text is matched with the preset information of the user end to verify the accuracy of the word sequence, and the corresponding speech recognition result is generated according to the verification result, including: Selecting a word to be verified corresponding to the preset information of the user in the sequence of words; searching for the to-be-verified word in the preset information of the user; if found, determining to verify by the accuracy, and using the word sequence as a speech recognition The result is otherwise corrected by the pinyin-based fuzzy matching method, and the modified word sequence is used as the speech recognition result.    The speech recognition method according to claim 33, wherein the correcting the word sequence by using a pinyin-based fuzzy matching method comprises: converting the to-be-verified word into a to-be-verified pinyin sequence; Each word in the conversion is converted into a comparison pinyin sequence; the similarity between the to-be-verified pinyin sequence and each of the comparison pinyin sequences is calculated in turn, and the user-preset information is selected to be sorted according to the similarity from high to low. The word before; the word to be verified in the sequence of words is replaced with the selected word to obtain the corrected sequence of words.    A speech recognition apparatus, comprising: a word sequence obtaining unit, configured to obtain a word sequence corresponding to the to-be-recognized speech signal through decoding; the word sequence verification unit is configured to verify the text by performing text matching with the user preset information. The accuracy of the word sequence, and the corresponding speech recognition result is generated according to the verification result.    The speech recognition apparatus according to claim 35, wherein the word sequence verification unit comprises: a to-be-verified word selection sub-unit, for selecting a to-be-verified word corresponding to the user-preset information from the word sequence. a search subunit for finding the to-be-verified word in the user preset information; a recognition result confirmation subunit, configured to, after the search subunit finds the to-be-verified word, determine to verify by the accuracy, and The word sequence is used as a speech recognition result; the recognition result correction sub-unit is configured to correct the word sequence through the pinyin-based fuzzy matching method after the search sub-unit does not find the to-be-verified word, and use the corrected word sequence as a speech Identify the results.    The speech recognition apparatus according to claim 36, wherein the recognition result correction subunit comprises: a to-be-verified pinyin sequence conversion sub-unit, configured to convert the to-be-verified word into a to-be-verified pinyin sequence; a sequence conversion sub-unit, configured to convert each word in the preset information of the user end into a comparison pinyin sequence; the similarity calculation selection sub-unit is configured to sequentially calculate the between the pinyin sequence to be verified and each of the comparison pinyin sequences Similarity, and selecting, from the user preset information, a word that ranks according to the similarity from high to low; the word to be verified replaces the subunit, and replaces the word to be verified in the word sequence with the selected word. , get the corrected word sequence.