WO2021043015A1 - 语音识别方法及装置、神经网络训练方法及装置 - Google Patents

语音识别方法及装置、神经网络训练方法及装置 Download PDF

Info

Publication number
WO2021043015A1
WO2021043015A1 PCT/CN2020/110742 CN2020110742W WO2021043015A1 WO 2021043015 A1 WO2021043015 A1 WO 2021043015A1 CN 2020110742 W CN2020110742 W CN 2020110742W WO 2021043015 A1 WO2021043015 A1 WO 2021043015A1
Authority
WO
WIPO (PCT)
Prior art keywords
network
sub
state information
target
spectrum
Prior art date
Application number
PCT/CN2020/110742
Other languages
English (en)
French (fr)
Inventor
王珺
林永业
苏丹
俞栋
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP20860648.3A priority Critical patent/EP3926623A4/en
Priority to JP2021564425A priority patent/JP7337953B2/ja
Publication of WO2021043015A1 publication Critical patent/WO2021043015A1/zh
Priority to US17/476,345 priority patent/US20220004870A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/16—Speech classification or search using artificial neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/06—Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
    • G10L15/063—Training
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
    • G10L2015/025—Phonemes, fenemes or fenones being the recognition units
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
    • G10L25/18—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being spectral information of each sub-band

Definitions

  • This application relates to the field of artificial intelligence technology, and specifically to a neural network training method for realizing speech recognition, a neural network training device for realizing speech recognition, a speech recognition method, a speech recognition device, electronic equipment, and a computer-readable storage medium.
  • the purpose of the embodiments of this application is to provide a neural network training method for realizing speech recognition, a neural network training device for realizing speech recognition, a speech recognition method, a speech recognition device, electronic equipment, and a computer-readable storage medium, which can further improve the complexity Speech recognition performance under interference sound conditions.
  • a neural network training method for realizing speech recognition which is executed by an electronic device, wherein the neural network includes a first sub-network, a second sub-network, and a third sub-network, and the method includes :
  • sample data includes a mixed speech spectrum and its labeled phonemes
  • the parameters of the first sub-network, the second sub-network, and the third sub-network are updated.
  • a neural network-based speech recognition method is provided, which is executed by an electronic device, wherein the neural network includes a first sub-network, a second sub-network, and a third sub-network, and the method includes:
  • phoneme recognition is performed based on the intermediate transition characterization.
  • a neural network training device for realizing speech recognition.
  • the neural network includes a first sub-network, a second sub-network and a third sub-network, and the device includes:
  • a data acquisition module configured to acquire sample data, where the sample data includes a mixed speech spectrum and annotated phonemes;
  • the target speech extraction module is used to extract the target speech frequency spectrum from the mixed speech frequency spectrum through the first sub-network;
  • An adaptive conversion module configured to perform adaptive conversion on the target speech spectrum through the second sub-network to obtain an intermediate transition characterization
  • a speech recognition module configured to perform phoneme recognition based on the intermediate transition characterization through a third sub-network
  • the parameter update module is configured to update the parameters of the first sub-network, the second sub-network, and the third sub-network according to the result of the phoneme recognition and the labeled phoneme.
  • a speech recognition device based on a neural network wherein the neural network includes a first sub-network, a second sub-network, and a third sub-network, and the device includes:
  • the data acquisition module is used to acquire the mixed speech spectrum to be recognized
  • the target speech extraction module is used to extract the target speech frequency spectrum from the mixed speech frequency spectrum through the first sub-network;
  • An adaptive conversion module configured to perform adaptive conversion on the target speech spectrum through the second sub-network to obtain an intermediate transition characterization
  • the speech recognition module is configured to perform phoneme recognition based on the intermediate transition characterization through the third sub-network.
  • an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute by executing the executable instructions
  • the above neural network training method or speech recognition method is provided.
  • a computer-readable storage medium on which a computer program is stored, and the computer program implements the above neural network training method or speech recognition method when executed by a processor.
  • FIG. 1 shows a schematic diagram of an exemplary system architecture of a neural network training method and device to which the embodiments of the present application can be applied;
  • FIG. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application
  • FIG. 3 schematically shows a flowchart of a neural network training method according to an embodiment of the present application
  • Fig. 4 schematically shows a flowchart of a process of extracting a target speech spectrum in an embodiment of the present application
  • Fig. 5 schematically shows a signal flow diagram of an LSTM unit in an embodiment of the present application
  • Fig. 6 schematically shows a flow chart of generating implicit state information of the current conversion process in an embodiment of the present application
  • Fig. 7 schematically shows a flowchart of a process of phoneme recognition in an embodiment of the present application
  • Fig. 8 schematically shows a flowchart of a voice recognition method according to an embodiment of the present application
  • Fig. 9 schematically shows an architecture diagram of an automatic speech recognition system according to an embodiment of the present application.
  • FIG. 10A schematically shows a reference diagram of the recognition effect of the automatic speech recognition system in an embodiment of the present application
  • FIG. 10B schematically shows a reference diagram of the recognition effect of the automatic speech recognition system in an embodiment of the present application
  • Fig. 11 schematically shows a block diagram of a neural network training device according to an embodiment of the present application
  • Fig. 12 schematically shows a block diagram of a speech recognition device according to an embodiment of the present application.
  • FIG. 1 shows a schematic diagram of a system architecture of an exemplary application environment of a neural network training method and device for realizing voice recognition, and a voice recognition method and device that can be applied to embodiments of the present application.
  • the system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104 and a server 105.
  • the network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105.
  • the network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, and so on.
  • the terminal devices 101, 102, 103 may include, but are not limited to, smart speakers, smart TVs, smart TV boxes, desktop computers, portable computers, smart phones, tablet computers, and so on. It should be understood that the numbers of terminal devices, networks, and servers in FIG. 1 are merely illustrative. According to implementation needs, there can be any number of terminal devices, networks, and servers.
  • the server 105 may be a server cluster composed of multiple servers.
  • the neural network training method or the voice recognition method provided in the embodiment of the present application may be executed by the server 105, and accordingly, the neural network training method or the voice recognition device may be set in the server 105.
  • the neural network training method or speech recognition method provided in the embodiments of the present application can also be executed by the terminal equipment 101, 102, 103, and correspondingly, the neural network training method or speech recognition device can also be set in the terminal equipment 101, 102, 103 .
  • the neural network training method or speech recognition method provided in the embodiments of the present application can also be executed by the terminal equipment 101, 102, 103 and the server 105. Accordingly, the neural network training method or speech recognition device can be set on the terminal equipment 101, 102 , 103 and server 105, there is no special limitation on this in this exemplary embodiment.
  • the terminal devices 101, 102, and 103 may encode and send the mixed voice data to be recognized to the server 105 after acquiring the mixed voice data;
  • the data is decoded and its spectral characteristics are extracted to obtain the mixed speech spectrum; then the target speech spectrum is extracted from the mixed speech spectrum through the first sub-network, and the target speech spectrum is adaptively converted through the second sub-network to obtain the intermediate transition characterization.
  • the phoneme recognition is performed based on the intermediate transition characterization through the third sub-network; after the recognition is completed, the server 105 may return the recognition result to the terminal devices 101, 102, 103.
  • Fig. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application. It should be noted that the computer system 200 of the electronic device shown in FIG. 2 is only an example, and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
  • the computer system 200 includes a central processing unit (CPU) 201, which can be based on a program stored in a read-only memory (ROM) 202 or loaded from a storage portion 208 into a random access memory (RAM) 203 Programs, and perform various appropriate actions and processing, such as the methods described in Figure 3, Figure 4, Figure 6, Figure 7, and Figure 8 described below.
  • RAM 203 various programs and data required for system operation are also stored.
  • the CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204.
  • An input/output (I/O) interface 205 is also connected to the bus 204.
  • the following components are connected to the I/O interface 205: an input part 206 including a keyboard, a mouse, etc.; an output part 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and speakers, etc.; a storage part 208 including a hard disk, etc. ; And a communication section 209 including a network interface card such as a LAN card, a modem, and the like. The communication section 209 performs communication processing via a network such as the Internet.
  • the drive 210 is also connected to the I/O interface 205 as needed.
  • a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that the computer program read from it is installed into the storage section 208 as needed.
  • the process described below with reference to the flowchart can be implemented as a computer software program.
  • the embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart.
  • the computer program can be downloaded and installed from the network through the communication part 209, and/or the computer program can be installed from the removable medium 211.
  • the computer program is executed by the central processing unit (CPU) 201, various functions defined in the method and device of the present application are executed.
  • the computer system 200 may further include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
  • AI Artificial Intelligence
  • AI Artificial Intelligence
  • digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.
  • artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.
  • Artificial intelligence technology is a comprehensive discipline, covering a wide range of fields, including both hardware-level technology and software-level technology.
  • Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, and mechatronics.
  • Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning.
  • speech Technology automatic speech recognition technology
  • TTS speech synthesis technology
  • voiceprint recognition technology Enabling computers to be able to listen, see, speak, and feel is the future development direction of human-computer interaction, among which voice has become one of the most promising human-computer interaction methods in the future.
  • a speech separation stage For the recognition of mixed speech, it usually includes a speech separation stage and a phoneme recognition stage.
  • a cascaded framework including a speech separation model and a phoneme recognition model is provided, thereby allowing independent modular research of the two stages.
  • the speech separation model and phoneme recognition model are trained separately in the training phase.
  • the speech separation model will inevitably introduce signal errors and signal distortions in the processing process, and these signal errors and signal distortions are not considered in the training process of the phoneme recognition model, which will lead to the speech recognition of this cascaded framework. The performance drops sharply.
  • one of the solutions provided by the inventor is to jointly train the speech separation model and the phoneme recognition model, which can significantly reduce the recognition error rate in noise robust speech recognition and multi-speaker speech recognition tasks.
  • an independent framework is provided, so that the speech separation stage is directly operated in the Mel filter domain, so as to be consistent with the phoneme recognition stage in the feature domain.
  • this technical solution may result in the inability to obtain better speech separation results; at the same time, the speech separation algorithm is constantly progressing, and this independent framework is difficult to quickly and flexibly integrate the first Three-way algorithm.
  • a joint framework is provided in which a deep neural network is used to learn the Mel filter frame-by-frame affine transformation function; however, this method is difficult to effectively model complex dynamic problems, and then It is difficult to handle speech recognition tasks under complex interference sound conditions.
  • this exemplary embodiment provides a neural network training method for realizing speech recognition.
  • the neural network training method can be applied to the aforementioned server 105; and can also be applied to one or more of the aforementioned terminal devices 101, 102, 103.
  • the neural network training method for realizing speech recognition may include the following steps:
  • Step S310 Obtain sample data, where the sample data includes the mixed speech spectrum and its labeled phonemes;
  • Step S320 Extract the target voice spectrum from the mixed voice spectrum through the first sub-network
  • Step S330 Perform adaptive conversion on the target speech spectrum through the second sub-network to obtain an intermediate transition characterization
  • Step S340 Perform phoneme recognition based on the intermediate transition characterization through the third sub-network
  • Step S350 According to the result of the phoneme recognition and the labeled phoneme, the parameters of the first sub-network, the second sub-network, and the third sub-network are updated.
  • the target speech spectrum extracted by the first sub-network can be adaptively converted through the second sub-network to obtain an intermediate transition characterization that can be input to the third sub-network for phoneme recognition, thereby completing
  • the speech separation stage and the phoneme recognition stage are bridged to realize an end-to-end speech recognition system; on this basis, the first sub-network, the second sub-network and the third sub-network are jointly trained to reduce the speech separation stage
  • the introduced signal error and signal distortion affect the performance of the phoneme recognition stage; therefore, the method provided in this example embodiment can improve the speech recognition performance under complex interference sound conditions and improve the user experience; at the same time, in this example embodiment
  • the first sub-network and the third sub-network can easily integrate third-party algorithms with higher flexibility.
  • step S310 sample data is obtained, and the sample data includes a mixed speech spectrum and annotated phonemes.
  • each set of sample data may include a mixed speech and annotated phonemes for the mixed speech.
  • the mixed speech can be a speech signal interfered by non-stationary noise, such as background music or multi-speaker interference, etc., so that the speech of different sound sources will be mixed, and the received speech will be mixed speech;
  • the label of mixed speech Phonemes indicate which phonemes are specifically included in the mixed speech;
  • the phoneme labeling method may be manual labeling, or the historical recognition result may be used as labeling phonemes; this exemplary embodiment does not specifically limit this.
  • each set of sample data may also include the reference voice corresponding to the mixed voice.
  • the reference voice may be, for example, a monophonic voice signal received when the speaker speaks in a quiet environment or in a stable noise interference environment; of course, it may also be obtained from the mixed voice in advance by other methods such as clustering. Extract the reference voice.
  • the mixed voice and the reference voice can be framed according to a certain frame length and frame shift to obtain the voice data of the mixed voice in each frame and the voice data of the reference voice in each frame. Then, the mixed voice data and the frequency spectrum characteristics of the reference voice can be extracted.
  • the spectral characteristics of the mixed voice data and the spectral characteristics of the reference voice data may be extracted based on Short Time Fourier Transform (STFT) or other methods.
  • STFT Short Time Fourier Transform
  • the mixed voice data of the nth frame may be expressed as x(n), and the mixed voice data x(n) may be regarded as target voice data.
  • the linear superposition of the interfering voice data s I (n), namely The reference voice data can be expressed as s s (n).
  • the mixed speech spectrum corresponding to the mixed speech data can be expressed as a T ⁇ F dimensional vector x
  • the reference speech spectrum corresponding to the reference speech data can be expressed as a T ⁇ F dimensional vector s s ; where T is the total number of frames , F is the number of frequency bands per frame.
  • step S320 the target voice spectrum is extracted from the mixed voice spectrum through the first sub-network.
  • the method for extracting the target speech frequency spectrum based on the Ideal Ratio Mask will be used as an example for description; however, this exemplary embodiment is not limited to this, and in the present application In other exemplary embodiments, other methods may also be used to extract the target speech spectrum.
  • the target speech spectrum can be extracted through the following steps S410 to S440. among them:
  • step S410 the mixed speech spectrum is embedded in a multi-dimensional vector space to obtain an embedded vector corresponding to each time-frequency window of the mixed speech spectrum.
  • the mixed speech frequency spectrum can be embedded in a K-dimensional vector space through a deep neural network model.
  • the above-mentioned deep neural network may be composed of a multi-layer two-way long and short-term memory network (BiLSTM, Bi-Long Short-Term Memory), such as a two-way long and short-term memory network including four layers of peephole connections.
  • the time memory network can have 600 hidden nodes and so on.
  • the above-mentioned deep neural network can also be replaced with various other effective network models; for example, convolutional neural network (CNN, Convolutional Neural Network) combined with other network structures, or models such as time-delay network and gated volume Other models such as product neural network; the model type and topological structure of deep neural network are not limited in this application.
  • CNN convolutional neural network
  • CNN Convolutional Neural Network
  • ⁇ extract represents a two-way long and short-term memory network
  • the network parameters of each time-frequency window are V f, t corresponding to the embedding vector; among them, t ⁇ [1, T], f ⁇ [1, F].
  • step S420 an ideal ratio mask is used to weight and normalize each embedded vector of the mixed speech to obtain an attractor corresponding to the target speech spectrum.
  • the ideal ratio mask m s can be calculated by
  • a supervised label w may also be set, which includes Through the supervised label w, the spectrum of each frame of the speech spectrum can be compared with a spectrum threshold.
  • the supervised label of the frame of the spectrum is set to 0; otherwise, it is set to 1.
  • the supervised label w can be as follows:
  • the attractor a s corresponding to the target speech spectrum can be as follows:
  • ⁇ represents the element multiplication of the matrix.
  • step S430 the target masking matrix corresponding to the target speech spectrum is obtained by calculating the similarity between each embedded vector of the mixed speech and the attractor.
  • the distance between each embedding vector of the mixed speech and the attractor can be calculated, and the value of each distance can be mapped to the range of [0,1], thereby representing each embedding vector and the attractor
  • the similarity between each embedding vector V f,t of the mixed speech and the attractor a s can be calculated by the following formula to obtain the target masking matrix corresponding to the target speech spectrum
  • Sigmoid is an sigmoid function, which can map variables to the range of [0,1], so as to facilitate the subsequent extraction of the target speech spectrum.
  • the similarity between each embedding vector of the mixed speech and the attractor can also be calculated based on, for example, a tanh function or other methods, and the corresponding to the target speech spectrum can be obtained.
  • the target masking matrix also belongs to the protection scope of this application.
  • step S440 based on the target masking matrix, the target speech spectrum is extracted from the mixed speech spectrum.
  • the target masking matrix can be used By weighting the aforementioned mixed speech spectrum x, the target speech spectrum in the aforementioned mixed speech spectrum can be extracted by time-frequency window; for the mixed speech spectrum x f,t of a certain time-frequency window, the larger the target masking matrix is, the corresponding The more spectral information of the time-frequency window will be extracted.
  • the target speech spectrum can be obtained by extracting the following formula
  • step S330 the target speech spectrum is adaptively converted through the second sub-network to obtain an intermediate transition characterization.
  • the target speech spectrum of each time-frequency window can be adaptively converted in sequence according to the time-frequency window sequence of the target speech spectrum; wherein, the conversion process for one of the time-frequency windows can be Including: generating the hidden state information of the current conversion process according to the target speech spectrum of the time-frequency window for the current conversion process and the hidden state information of the previous conversion process; and obtaining the current conversion process based on each of the hidden state information
  • the intermediate transition characterization of the targeted time-frequency window takes a network based on LSTM (Long Short-Term Memory) as an example to describe the above conversion process in detail.
  • the LSTM network is a processing unit (hereinafter referred to as the LSTM unit).
  • the LSTM unit usually includes a forget gate, an input gate, and an output gate; in this example embodiment, one LSTM unit can perform one of the above conversion processes.
  • FIG. 6 is a process of an LSTM unit generating implicit state information of the current conversion process, which may include the following steps S610 to S650. among them:
  • step S610 according to the target speech spectrum of the current time-frequency window and the implicit state information of the previous conversion process, the candidate state information, the input weight of the candidate state information, the forgetting weight of the target state information of the previous conversion process, and the implicit state information of the previous conversion process are calculated.
  • the output weight of the target state information in the current conversion process in particular:
  • the forget gate is used to determine how much information is discarded from the target state information of the previous conversion process, so the forgetting weight is used to indicate the weight of the target state information of the previous conversion process that is not forgotten (that is, can be retained); the forgetting weight can be essentially Is the weight matrix.
  • the target speech spectrum of the current time-frequency window and the implicit state information of the previous conversion process can be encoded through the activation function used to characterize the forgetting gate, and mapped to a value between 0 and 1, to obtain the previous The forgetting weight of the target state information in the conversion process. Among them, 0 means completely discarded, and 1 means completely reserved.
  • the forgetting weight f t of the target state information of the previous conversion process can be calculated according to the following formula:
  • h t-1 implies the state represented by a conversion process of information
  • S t represents the current time-frequency target speech spectral window
  • [sigma] represents the activation function Sigmod function
  • W f and b f represents a forgetting parameter Sigmod door function
  • [h t-1 , S t ] represents the combination of h t-1 and S t .
  • the input gate is used to determine how much information is important and needs to be preserved in the target speech spectrum currently input.
  • the target speech spectrum of the current time-frequency window and the implicit state information of the previous conversion process can be encoded by the activation function used to characterize the input gate to obtain the candidate state information and the input weight of the candidate state information ;
  • the input weight of the candidate state information is used to determine how much new information in the candidate state information can be added to the target state information.
  • the candidate status information can be calculated according to the following formula
  • tanh indicates that the activation function is a positive curve bitangent function
  • W c and b c indicate the parameters of the tanh function in the input gate.
  • ⁇ represents the activation function Sigmod function
  • W i represents the activation function
  • b i represents the parameter input gates Sigmod function
  • the output gate is used to determine what information should be included in the hidden state information output to the next LSTM unit.
  • the target speech spectrum of the current time-frequency window and the implicit state information of the previous conversion process can be encoded through the activation function used to characterize the output gate to obtain the output weight of the target state information of the current conversion process.
  • the candidate state information o t can be calculated according to the following formula:
  • ⁇ represents the activation function Sigmod function
  • W o and b o represent the parameters of the Sigmod function in the output gate.
  • step S620 the target state information of the previous conversion process is retained according to the forgetting weight to obtain first intermediate state information.
  • the obtained first intermediate state information can be Among them, C t-1 represents the target state information of the previous conversion process.
  • step S630 the candidate state information is retained according to the input weight of the candidate state information to obtain second intermediate state information.
  • the obtained second intermediate state information can be
  • step S640 the target state information of the current conversion process is obtained according to the first intermediate state information and the second intermediate state information. For example, the target state information of the current conversion process
  • step S650 the current conversion process target state information is retained according to the output weight of the current conversion process target state information, and the implicit state information of the current conversion process is obtained.
  • the implicit state information of the current conversion process is obtained.
  • the implicit state information h t is obtained by adaptively converting the target speech spectrum sequence of each time-frequency window, that is, the adaptive conversion is performed through the forward LSTM.
  • adaptive conversion can also be performed through a bidirectional long-short-term memory (BiLSTM) network; furthermore, in other exemplary embodiments, a peephole-connected multilayer bidirectional long- and short-term memory network can also be used for adaptation sexual conversion, thereby further improving the accuracy of adaptive conversion.
  • the target speech spectrum of each time-frequency window can be adaptively converted in the reverse order to obtain the implicit state information.
  • the hidden state information h t and the hidden state information Splicing obtained BiLSTM network output i.e., hidden state information H t, H t so that information may be better characterized by the bidirectional timing dependent features hidden state.
  • each implicit state information may be performed on each implicit state information to obtain the current conversion process
  • the intermediate transition characterization of the time-frequency window may be performed on each implicit state information to obtain the current conversion process
  • non-negative mapping may also be implemented in other ways such as ReLU (Rectified Linear Unit, linear rectification function), which is not particularly limited in this exemplary embodiment.
  • ReLU Rectified Linear Unit, linear rectification function
  • ⁇ adapt represents the two-way long and short-term memory network Network parameters.
  • the features of the frequency window are spliced to obtain the intermediate transition characterization of the current time-frequency window; after the above processing, the intermediate transition characterization is obtained It should be noted that in other exemplary embodiments of the present application, part of the processing process can also be selected from the foregoing processing process to be executed, and other methods can also be selected for processing, which also belong to the protection scope of the present application.
  • step S340 phoneme recognition is performed based on the intermediate transition characterization through the third sub-network.
  • the intermediate transition characterization f output by the above-mentioned second sub-network may be input to the third sub-network to obtain the posterior probability y t of the phonemes included in the intermediate transition characterization.
  • the third sub-network may be a Convolutional Long-Short-Term-Memory Deep-Neural-Network (CLDNN) based on an optimal center loss (Center-Loss-optimal) , which can be expressed as CL_CLDNN network below.
  • 1. It can be used to calculate the posterior probability of the phoneme; ⁇ recog ⁇ , W, b ⁇ represents the parameters of the CL_CLDNN network.
  • the third sub-network can perform phoneme recognition based on the intermediate transition characterization through the following steps S710 to S730. among them:
  • a multi-dimensional filter is applied to the intermediate transition representation through at least one convolutional layer to generate a convolutional layer output, so that spectral disparity can be reduced.
  • two convolutional layers may be included, and each convolutional layer may include 256 feature maps; the first convolutional layer may use a 9 ⁇ 9 time-frequency domain filter, and the second layer A 4 ⁇ 3 time domain-frequency domain filter can be used.
  • a linear layer may be followed by the last convolutional layer for dimensionality reduction.
  • the output of the convolutional layer is used in at least one recursive layer to generate the output of the recursive layer, so as to realize the timing feature modeling.
  • the recursive layer may include multiple layers of long and short-term memory networks; for example, two layers of long- and short-term memory networks can be connected after the above linear layer, and each long- and short-term memory network can use 832 processing units, 512 Dimensional mapping layer to reduce dimensionality.
  • the recursive layer may also include a GRU (Gated Recurrent Unit) network or other RNN (Recurrent Neural Networks, recurrent neural network) structure. This exemplary embodiment China does not make any special restrictions on this.
  • the output of the recursive layer is provided to at least one fully connected layer, and a nonlinear function is applied to the output of the fully connected layer to obtain the posterior probability of the phoneme included in the intermediate transition characterization.
  • the fully connected layer may be, for example, a two-layer DNN (Deep Neural Network, deep neural network) structure; each layer of the DNN structure may include 1024 neurons, and the feature space can be mapped to easier The output layer of the classification.
  • the output layer can be classified using a non-linear function such as the Softmax function or the tanh function, so as to obtain the posterior probability of the phonemes included in the intermediate transition characterization.
  • step S350 the parameters of the first sub-network, the second sub-network, and the third sub-network are updated according to the result of the phoneme recognition and the labeled phoneme.
  • the joint loss function of the first sub-network, the second sub-network, and the third sub-network may be determined first.
  • the aforementioned Center-Loss-optimal and cross-entropy loss may be used as the joint loss function; of course, in other exemplary embodiments of the present application, other losses may also be used as the joint loss function.
  • This exemplary embodiment is not limited to this.
  • the result of the phoneme recognition and the labeled phonemes may be input into the joint loss function, and the joint loss function value can be calculated.
  • the parameters of the first sub-network, the second sub-network, and the third sub-network may be updated according to the joint loss function value.
  • the joint loss function can be minimized as the training target, and methods such as Stochastic Gradient Descent (SGD) and Back Propagation (BP) can be used for the first sub-network, the second sub-network, and the second sub-network.
  • SGD Stochastic Gradient Descent
  • BP Back Propagation
  • the parameters of the three sub-networks are updated until convergence, such as the number of training iterations reaches the maximum number or the joint loss function value no longer decreases, etc.
  • This example embodiment also provides a method for speech recognition based on a neural network, which may be obtained by training through the training method in the foregoing exemplary embodiment.
  • the voice recognition method can be applied to one or more of the aforementioned terminal devices 101, 102, 103; and can also be applied to the aforementioned server 105.
  • the voice recognition method may include the following steps S810 to S840. among them:
  • step S810 the mixed speech spectrum to be recognized is acquired.
  • the mixed voice may be a voice signal that is interfered by non-stationary noise, such as background music or multi-speaker interference, etc., so that voices from different sound sources are mixed, and the received voice is mixed. voice.
  • the mixed voice can be divided into frames according to a certain frame length and frame shift to obtain the voice data of the mixed voice in each frame.
  • the spectral characteristics of the mixed voice data can be extracted.
  • the spectral features of the mixed voice data may be extracted based on Short Time Fourier Transform (STFT) or other methods.
  • STFT Short Time Fourier Transform
  • the mixed voice data of the nth frame may be expressed as x(n), and the mixed voice data x(n) may be regarded as target voice data.
  • the linear superposition of interfering voice data s I (n) namely After performing short-time Fourier transform on the mixed speech data x(n), take the logarithm of the result of the short-time Fourier transform to obtain the spectral characteristics of the mixed speech data; for example, the mixed speech data corresponding to the mixed speech data
  • the frequency spectrum is expressed as a T ⁇ F dimensional vector x; where T is the total number of frames, and F is the number of frequency bands per frame.
  • step S820 the target voice spectrum is extracted from the mixed voice spectrum through the first sub-network.
  • the method for extracting the target speech frequency spectrum based on the Ideal Ratio Mask will be used as an example for description; however, this exemplary embodiment is not limited to this, and in the present application In other exemplary embodiments, other methods may also be used to extract the target speech spectrum. For example:
  • the mixed speech spectrum into a multi-dimensional vector space to obtain the embedding vector corresponding to each time-frequency window of the mixed speech spectrum.
  • the two-way long and short-term memory network as an example, which can convert the mixed speech spectrum from the vector space Map to a higher-dimensional vector space
  • the obtained embedded matrix V of the mixed speech spectrum is as follows:
  • ⁇ extract represents a two-way long and short-term memory network
  • the network parameters of each time-frequency window are V f, t corresponding to the embedding vector; among them, t ⁇ [1, T], f ⁇ [1, F].
  • step S320 obtain the global attractor obtained in step S320 in the above training process
  • a target masking matrix corresponding to the target speech spectrum is obtained.
  • the embedding vectors V f, t and the global attractor of the mixed speech can be calculated by the following formula The similarity between, obtain the target masking matrix corresponding to the target speech spectrum
  • the target speech spectrum is extracted from the mixed speech spectrum.
  • the target speech spectrum can be obtained by extracting the following formula
  • step S830 the target speech spectrum is adaptively converted through the second sub-network to obtain an intermediate transition characterization.
  • the target speech spectrum of each time-frequency window may be sequentially adaptively converted; and the conversion process for a time-frequency window may include: The target speech spectrum of the time-frequency window for the current conversion process and the hidden state information of the previous conversion process are generated to generate the hidden state information of the current conversion process; and based on each of the hidden state information, the time for the current conversion process is obtained The intermediate transition characterization of the frequency window.
  • each LSTM unit of a bidirectional long short-term memory (BiLSTM) network can perform the above-mentioned various conversion processes.
  • the output of the BiLSTM network can also be squared, thereby realizing non-negative mapping.
  • the non-negative mapping result can be as follows:
  • ⁇ adapt represents the two-way long and short-term memory network Network parameters.
  • the non-negative mapping process you can also Perform a series of differentiable operations, such as element-wise logarithm, calculation of first-order difference, calculation of second-order difference, etc.; in addition, it can also perform global mean variance normalization and add the characteristics of the front and back time-frequency windows. For example, for the current time-frequency window, the characteristics of the current time-frequency window, the characteristics of W time-frequency windows before the current time-frequency window, and the characteristics of W time-frequency windows after the current time-frequency window can be combined, that is, a total of 2W+1 time-frequency windows.
  • the features of the frequency window are spliced to obtain the intermediate transition characterization of the current time-frequency window; after the above processing, the intermediate transition characterization is obtained
  • step S840 phoneme recognition is performed based on the intermediate transition characterization through the third sub-network.
  • the intermediate transition characterization f output by the above-mentioned second sub-network may be input to the third sub-network to obtain the posterior probability y t of the phonemes included in the intermediate transition characterization.
  • the third sub-network may be a CL_CLDNN network.
  • 1 , Can be used to calculate the posterior probability of the phoneme; ⁇ recog ⁇ , W, b ⁇ represents the parameters of the CL_CLDNN network.
  • the automatic speech recognition system may include a first sub-network 910, a second sub-network 920, and a third sub-network 930. among them:
  • the first sub-network 910 may extract the target voice spectrum from the mixed voice spectrum.
  • the first sub-network may include a multi-layer (such as four-layer) BiLSTM network connected by a peephole, and each layer of the BiLSTM network may include 600 hidden nodes; at the same time, after the last layer of the BiLSTM network, a full network can be connected.
  • the connection layer is used to map the 600-dimensional hidden state information into a 24000-dimensional embedding vector.
  • the mixed speech spectrum may be, for example, a 512-dimensional short-time Fourier transform spectrum feature with a sampling rate of 16000 Hz, a sub-frame frame length of 25 ms, and a frame shift of 10 ms.
  • the mixed speech spectrum After inputting the mixed speech spectrum into the first sub-network 910, it can be mapped into an embedding vector through the BiLSTM network; then, the similarity between the embedding vector and the attractor can be calculated to obtain the target masking matrix; furthermore, it can be masked based on the target Matrix, extract the target speech frequency spectrum S from the mixed speech frequency spectrum.
  • the reference speech frequency spectrum can also be input to the first sub-network 910, so that the ideal ratio mask can be calculated according to the reference speech frequency spectrum, and then the embedded vectors of the mixed speech can be weighted and regularized according to the ideal ratio mask to obtain the Attractor.
  • the second sub-network 920 may be used to perform adaptive conversion on the target speech spectrum to obtain an intermediate transition characterization.
  • the second sub-network 920 may include a multi-layer (for example, two-layer) BiLSTM network connected by a peephole, and each layer of the BiLSTM network may include 600 hidden nodes.
  • the implicit state information H can be subjected to non-negative mapping, element-wise logarithm, calculation of the first-order difference, calculation of the second-order difference, normalization of the global mean variance, and the addition of the characteristics of the time-frequency window before and after the preset processing, etc.
  • the intermediate transition characterization f may be, for example, a 40-dimensional fbank feature vector.
  • the third sub-network 930 may be used to perform phoneme recognition based on the intermediate transition characterization.
  • the third sub-network 920 may include a CL_CLDNN network. After inputting the intermediate transition characterization f into the third sub-network, the posterior probability y t of the phonemes included in the intermediate transition characterization can be obtained; taking Mandarin Chinese as an example, the posterior probabilities of about 12,000 categories of phonemes can be output.
  • the batch size of sample data can be set to 24, the initial learning rate ⁇ is set to 10 -4 , the decay coefficient of the learning rate is set to 0.8, and the convergence judgment condition is set to the comprehensive loss function value continuous 3 There is no improvement in the second iteration (epoch).
  • the dimension K of the embedding vector is set to 40, the number of Mel filter bands D is set to 40, and the number of time-frequency windows when adding the features of the front and back time-frequency windows is set to 5.
  • Set the weight ⁇ of the center loss to 0.01.
  • the output of the convolutional layer in the CL_CLDNN network and the output of the LSTM network can both be batch normalized to achieve faster convergence and better generalization.
  • FIG. 10A and FIG. 10B are reference diagrams of the voice recognition effect of the above-mentioned automatic voice recognition system.
  • Fig. 10A is a speech recognition task interfered by background music
  • Fig. 10B is a speech recognition task interfered by other speakers.
  • the vertical axis uses Relative Word Error Rate Reduction (WERR) to characterize the recognition effect
  • the horizontal axis represents different decibel dB (decibel) signal-to-noise ratio interference test conditions, a total of five types Signal to noise ratio: 0dB, 5dB, 10dB, 15dB and 20dB.
  • WERR Relative Word Error Rate Reduction
  • the line P1 and the line P4 represent the WERR obtained by the comparison between the automatic speech recognition system and the baseline system in this example embodiment
  • the line P2 and the line P5 represent the existing advanced automatic speech recognition system (such as DNN learning Mei The robust speech recognition joint training architecture of the filter frame-by-frame affine transformation function) and the baseline system WERR
  • line P3 represents the automatic speech recognition system in the exemplary embodiment combined with target speaker tracking processing, and compared with the baseline system WERR.
  • the above-mentioned existing advanced automatic speech recognition system is equivalent to the automatic speech recognition system in this example embodiment in terms of parameter complexity; but it can be seen from FIG. 10A and FIG. 10B that the WERR of the automatic speech recognition system in this example embodiment is Both recognition tasks are significantly better than the existing advanced automatic speech recognition system; this shows that the automatic speech recognition system in this example embodiment can effectively model the problem of temporal complexity, which can further improve the complex interference Speech recognition performance under acoustic conditions.
  • the automatic speech recognition system in this exemplary embodiment also has a high degree of flexibility, such as allowing flexible integration of various types of speech separation modules and phoneme recognition modules into the first sub-network and The third sub-network, and the automatic speech recognition system in this exemplary embodiment will not be at the cost of any single module performance impairment.
  • a neural network training device for realizing speech recognition.
  • the neural network training device can be applied to a server; it can also be applied to a terminal device.
  • the neural network includes first to third sub-networks.
  • the neural network training device 1100 may include a data acquisition module 1110, a target speech extraction module 1120, an adaptive conversion module 1130, a speech recognition module 1140, and parameters Update module 1150. among them:
  • the data acquisition module 1110 may be used to acquire sample data, where the sample data includes a mixed speech spectrum and its labeled phonemes;
  • the target voice extraction module 1120 may be used to extract the target voice spectrum from the mixed voice spectrum through the first sub-network;
  • the adaptive conversion module 1130 may be used to perform adaptive conversion on the target speech spectrum through the second sub-network to obtain an intermediate transition characterization
  • the speech recognition module 1140 may be used to perform phoneme recognition based on the intermediate transition characterization through the third sub-network;
  • the parameter update module 1150 may be configured to update the parameters of the first sub-network, the second sub-network, and the third sub-network according to the result of the phoneme recognition and the labeled phoneme.
  • the target speech extraction module 1120 extracts the target speech frequency spectrum from the mixed speech frequency spectrum by the following steps: embedding the mixed speech frequency spectrum into a multi-dimensional vector space to obtain the mixed speech The embedded vector corresponding to each time-frequency window of the frequency spectrum; weighted and normalized each embedded vector of the mixed speech spectrum using an ideal ratio mask to obtain the attractor corresponding to the target speech frequency spectrum; by calculating the value of the mixed speech frequency spectrum The similarity between each embedding vector and the attractor is used to obtain a target masking matrix corresponding to the target speech spectrum; based on the target masking matrix, the target speech spectrum is extracted from the mixed speech spectrum.
  • the device further includes:
  • the global attractor calculation module may be used to obtain the attractors corresponding to each of the sample data and calculate the average value of each attractor to obtain the global attractor.
  • the adaptive conversion module 1130 adaptively converts the target speech spectrum through the following steps: according to the time-frequency window sequence of the target speech spectrum, The target speech spectrum of the window undergoes adaptive conversion in turn; wherein, the conversion process for a time-frequency window includes:
  • the target speech spectrum of the time-frequency window for the current conversion process and the hidden state information of the previous conversion process According to the target speech spectrum of the time-frequency window for the current conversion process and the hidden state information of the previous conversion process, generate the hidden state information of the current conversion process; and based on each of the hidden state information, obtain the target speech of the current conversion process The intermediate transition characterization of the time-frequency window.
  • the adaptive conversion module 1130 generates the implicit state information of the current conversion process through the following steps: according to the target speech spectrum of the current time-frequency window and the implicit state information of the previous conversion process State information, calculating the candidate state information, the input weight of the candidate state information, the forgetting weight of the target state information of the previous conversion process, and the output weight of the target state information of the current conversion process; and calculating the previous conversion process according to the forgetting weight
  • the target state information is retained to obtain the first intermediate state information
  • the candidate state information is retained according to the input weight of the candidate state information to obtain the second intermediate state information; according to the first intermediate state information and the second intermediate state information
  • the state information obtains the target state information of the current conversion process; the target state information of the current conversion process is retained according to the output weight of the target state information of the current conversion process, and the implicit state information of the current conversion process is obtained.
  • the adaptive conversion module 1130 obtains the intermediate transition characterization of the time-frequency window for the current conversion process based on each of the implicit state information through the following steps:
  • the implicit state information is processed by one or more of the following to obtain the intermediate transition characterization of the time-frequency window targeted by the current conversion process:
  • Non-negative mapping element-wise logarithm, calculation of the first-order difference, calculation of the second-order difference, normalization of the global mean variance, and addition of the characteristics of the before and after time-frequency windows.
  • the speech recognition module 1140 performs phoneme recognition based on the intermediate transition representation through the following steps: applying a multi-dimensional filter to the intermediate transition representation through at least one convolutional layer, Generate a convolutional layer output; use the convolutional layer output in at least one recursive layer to generate a recursive layer output; provide the recursive layer output to at least one fully connected layer, and apply non-convolutional layer output to the output of the fully connected layer Linear function to obtain the posterior probability of the phonemes included in the intermediate transition characterization.
  • the recursive layer includes a long and short-term memory network.
  • the parameter update module 1150 updates the parameters of the first sub-network, the second sub-network, and the third sub-network through the following steps: determining the first sub-network The joint loss function of the network, the second sub-network, and the third sub-network; calculate the joint loss function value according to the result of the phoneme recognition, the labeled phoneme, and the joint loss function; and calculate the joint loss function value according to the joint loss function value
  • the parameters of the first sub-network, the second sub-network, and the third sub-network are updated.
  • the first sub-network includes a peephole-connected multi-layer long and short-term memory network; the second sub-network includes a peephole-connected multi-layer long and short-term memory network.
  • a voice recognition device based on a neural network is also provided.
  • the voice recognition device can be applied to the server; it can also be applied to the terminal equipment.
  • the neural network includes first to third sub-networks.
  • the neural network training device 1200 may include a data acquisition module 1210, a target speech extraction module 1220, an adaptive conversion module 1230, and a speech recognition module 1240. among them:
  • the data acquisition module 1210 may be used to acquire the mixed speech spectrum to be recognized
  • the target voice extraction module 1220 may be used to extract the target voice spectrum from the mixed voice spectrum through the first sub-network;
  • the adaptive conversion module 1230 may be used to perform adaptive conversion on the target speech spectrum through the second sub-network to obtain an intermediate transition characterization
  • the speech recognition module 1240 may be used to perform phoneme recognition based on the intermediate transition characterization through the third sub-network.
  • the target speech spectrum extracted by the first sub-network can be adaptively converted through the second sub-network to obtain an intermediate transition characterization that can be input into the third sub-network for phoneme recognition, so that the pairing can be completed
  • the speech separation stage and the phoneme recognition stage are bridged to realize an end-to-end speech recognition system; on this basis, the first sub-network, the second sub-network and the third sub-network are jointly trained to reduce the introduction of the speech separation stage
  • the signal error and signal distortion affect the performance of the phoneme recognition stage; therefore, the method provided by the example embodiment of this application can improve the speech recognition performance under complex interference sound conditions and improve the user experience; at the same time, the example embodiment of this application
  • the first sub-network and the third sub-network can easily integrate third-party algorithms and have higher flexibility.
  • modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory.
  • the features and functions of two or more modules or units described above may be embodied in one module or unit.
  • the features and functions of a module or unit described above can be further divided into multiple modules or units to be embodied.
  • this application also provides a computer-readable medium.
  • the computer-readable medium may be included in the electronic device described in the above-mentioned embodiments; or it may exist alone without being assembled into the electronic device. in.
  • the above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device realizes the method described in the above-mentioned embodiment.
  • the electronic device described can implement various steps and the like in the above-mentioned exemplary embodiments.
  • the computer-readable medium shown in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable removable Programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
  • the computer-readable signal medium may include a data signal propagated in the baseband or as a part of the carrier, and the computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
  • the computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
  • each block in the flowchart or block diagram may represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more for realizing the specified logic function.
  • Executable instructions may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by It is realized by a combination of dedicated hardware and computer instructions.

Abstract

本申请提供一种语音识别方法及装置、神经网络训练方法及装置；涉及人工智能技术领域。该神经网络训练方法由电子设备执行，包括：获取样本数据，所述样本数据包括混合语音频谱及其标注音素；通过第一子网络从混合语音频谱中提取目标语音频谱；通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；通过第三子网络，基于所述中间过渡表征进行音素识别；根据所述音素识别的结果以及所述标注音素，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。

Description

语音识别方法及装置、神经网络训练方法及装置
本申请要求于2019年9月5日提交中国专利局、申请号为201910838469.5、发明名称为“语音识别方法及装置、神经网络训练方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能技术领域，具体而言，涉及一种实现语音识别的神经网络训练方法、实现语音识别的神经网络训练装置、语音识别方法、语音识别装置、电子设备以及计算机可读存储介质。
发明背景
随着科学技术的发展和硬件计算能力的大幅提升，目前越来越多的基于深度学习技术实现语音识别。
但声学场景中语音识别的实现往往受限于声学场景的变化性。举例而言，单声道语音信号受到非平稳噪声干扰的情形，如受背景音乐或多说话人干扰等，在实际应用场景中普遍存在。
尽管深度学习技术的引入为语音识别技术带来了巨大的性能改进，但传统的语音识别技术在这些复杂的环境中性能仍存在待优化之处。
需要说明的是，在上述背景技术部分公开的信息仅用于加强对本申请的背景的理解，因此可以包括不构成对本领域普通技术人员已知的现有技术的信息。
发明内容
本申请实施例的目的在于提供一种实现语音识别的神经网络训练方法、实现语音识别的神经网络训练装置、语音识别方法、语音识别装置、电子设备以及计算机可读存储介质，进而可以提升在复杂干扰声条件下的语音识别性能。
根据本申请的一个方面，提供一种实现语音识别的神经网络训练方法，由电子设备执行，其中，所述神经网络包括第一子网络，第二子网络和第三子网络，所述方法包括：
获取样本数据，所述样本数据包括混合语音频谱及其标注音素；
通过第一子网络从混合语音频谱中提取目标语音频谱；
通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
通过第三子网络，基于所述中间过渡表征进行音素识别；
根据所述音素识别的结果以及所述标注音素，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。
根据本申请的一个方面，提供一种基于神经网络的语音识别方法，由电子设备执行，其中，所述神经网络包括第一子网络，第二子网络和第三子网络，所述方法包括：
获取待识别的混合语音频谱；
通过第一子网络从混合语音频谱中提取目标语音频谱；
通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
通过第三子网络，基于所述中间过渡表征进行音素识别。
根据本申请的一个方面，提供一种实现语音识别的神经网络训练装置，所述神经网络包括第一子网络，第二子网络和第三子网络，所述装置包括：
数据获取模块，用于获取样本数据，所述样本数据包括混合语音频谱及其标注音素；
目标语音提取模块，用于通过第一子网络从混合语音频谱中提取目标语音频谱；
适应性转换模块，用于通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
语音识别模块，用于通过第三子网络，基于所述中间过渡表征进行音素识别；
参数更新模块，用于根据所述音素识别的结果以及所述标注音素，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。
根据本申请的一个方面，提供一种基于神经网络的语音识别装置，其中，所述神经网络包括第一子网络，第二子网络和第三子网络，所述装置包括：
数据获取模块，用于获取待识别的混合语音频谱；
目标语音提取模块，用于通过第一子网络从混合语音频谱中提取目标语音频谱；
适应性转换模块，用于通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
语音识别模块，用于通过第三子网络，基于所述中间过渡表征进行音素识别。
根据本申请的一个方面，提供一种电子设备，包括：处理器；以及存储器，用于存储所述处理器的可执行指令；其中，所述处理器配置为经由执行所述可执行指令来执行上述神经网络训练方法或语音识别方法。
根据本申请的一个方面，提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述神经网络训练方法或语音识别方 法。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本申请。
附图简要说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理。显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1示出了可以应用本申请实施例的一种神经网络训练方法及装置的示例性系统架构的示意图；
图2示出了适于用来实现本申请实施例的电子设备的计算机系统的结构示意图；
图3示意性示出了根据本申请的一个实施例的神经网络训练方法的流程图；
图4示意性示出了本申请的一个实施例中提取目标语音频谱的过程的流程图；
图5示意性示出了本申请的一个实施例中LSTM单元的信号流转图；
图6示意性示出了本申请的一个实施例中生成当前转换过程的隐含状态信息的流程图；
图7示意性示出了本申请的一个实施例中进行音素识别的过程的流程图；
图8示意性示出了根据本申请的一个实施例的语音识别方法的流程图；
图9示意性示出了根据本申请的一个实施例的自动语音识别系统的架构图；
图10A示意性示出了本申请的一个实施例中自动语音识别系统的识别效果参考图；
图10B示意性示出了本申请的一个实施例中自动语音识别系统的识别效果参考图；
图11示意性示出了根据本申请的一个实施例的神经网络训练装置的框图；
图12示意性示出了根据本申请的一个实施例的语音识别装置的框图。
实施方式
现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本申请将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技 术人员。所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。在下面的描述中，提供许多具体细节从而给出对本申请的实施方式的充分理解。然而，本领域技术人员将意识到，可以实践本申请的技术方案而省略所述特定细节中的一个或更多，或者可以采用其它的方法、组元、装置、步骤等。在其它情况下，不详细示出或描述公知技术方案以避免喧宾夺主而使得本申请的各方面变得模糊。
此外，附图仅为本申请的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。
图1示出了可以应用本申请实施例的一种实现语音识别的神经网络训练方法及装置、语音识别方法及装置的示例性应用环境的系统架构的示意图。
如图1所示，系统架构100可以包括终端设备101、102、103中的一个或多个，网络104和服务器105。网络104用以在终端设备101、102、103和服务器105之间提供通信链路的介质。网络104可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。终端设备101、102、103可以包括但不限于智能音箱、智能电视、智能电视盒子、台式计算机、便携式计算机、智能手机和平板电脑等等。应该理解，图1中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要，可以具有任意数目的终端设备、网络和服务器。比如服务器105可以是多个服务器组成的服务器集群等。
本申请实施例所提供的神经网络训练方法或语音识别方法可以由服务器105执行，相应的，神经网络训练方法或语音识别装置可以设置于服务器105中。本申请实施例所提供的神经网络训练方法或语音识别方法也可以由终端设备101、102、103执行，相应的，神经网络训练方法或语音识别装置也可以设置于终端设备101、102、103中。本申请实施例所提供的神经网络训练方法或语音识别方法还可以由终端设备101、102、103与服务器105共同执行，相应地，神经网络训练方法或语音识别装置可以设置于终端设备101、102、103与服务器105中，本示例性实施例中对此不做特殊限定。
举例而言，在一种示例性实施例中，可以是终端设备101、102、103在获取待识别的混合语音数据之后，对其进行编码并发送至服务器105；服务器105对接收到的混合语音数据进行解码，并提取其频谱特征，得到混合语音频谱；接着 通过第一子网络从混合语音频谱中提取目标语音频谱，通过第二子网络对目标语音频谱进行适应性转换以得到中间过渡表征，通过第三子网络基于中间过渡表征进行音素识别；在完成识别之后，服务器105可以将识别结果返回至终端设备101、102、103。
图2示出了适于用来实现本申请实施例的电子设备的计算机系统的结构示意图。需要说明的是，图2示出的电子设备的计算机系统200仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。
如图2所示，计算机系统200包括中央处理单元(CPU)201，其可以根据存储在只读存储器(ROM)202中的程序，或者从存储部分208加载到随机访问存储器(RAM)203中的程序，而执行各种适当的动作和处理，比如，下面描述的图3，图4，图6，图7和图8中描述的方法。在RAM 203中，还存储有系统操作所需的各种程序和数据。CPU 201、ROM 202以及RAM 203通过总线204彼此相连。输入/输出(I/O)接口205也连接至总线204。
以下部件连接至I/O接口205：包括键盘、鼠标等的输入部分206；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分207；包括硬盘等的存储部分208；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分209。通信部分209经由诸如因特网的网络执行通信处理。驱动器210也根据需要连接至I/O接口205。可拆卸介质211，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器210上，以便于从其上读出的计算机程序根据需要被安装入存储部分208。
特别地，根据本申请的实施例，下文参考流程图描述的过程可以被实现为计算机软件程序。例如，本申请的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，可以通过通信部分209从网络上下载和安装该计算机程序，和/或从可拆卸介质211安装该计算机程序。在该计算机程序被中央处理单元(CPU)201执行时，执行本申请的方法和装置中限定的各种功能。在一些实施例中，计算机系统200还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。
人工智能(Artificial Intelligence,AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个综合技术，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式做出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法， 使机器具有感知、推理与决策的功能。
人工智能技术是一门综合学科，涉及领域广泛，既有硬件层面的技术也有软件层面的技术。人工智能基础技术一般包括如传感器、专用人工智能芯片、云计算、分布式存储、大数据处理技术、操作/交互系统、机电一体化等技术。人工智能软件技术主要包括计算机视觉技术、语音处理技术、自然语言处理技术以及机器学习/深度学习等几大方向。
其中，语音处理技术(Speech Technology)的关键技术有自动语音识别技术(ASR)和语音合成技术(TTS)以及声纹识别技术。让计算机能听、能看、能说、能感觉，是未来人机交互的发展方向，其中语音成为未来最被看好的人机交互方式之一。
本申请中技术方案会涉及到上述语音处理技术。以下对本申请实施例的技术方案进行详细阐述：
对于混合语音的识别，通常包括语音分离阶段和音素识别阶段。一种相关技术中，提供了包括语音分离模型和音素识别模型的级联框架，从而允许对两个阶段分别独立的进行模块化研究。在这种模块化方法中，会在训练阶段对语音分离模型和音素识别模型分别进行训练。然而，语音分离模型在处理过程不可避免的会引入信号误差和信号失真，而这些信号误差和信号失真在音素识别模型的训练过程中是不曾考虑的，因此会导致这种级联框架的语音识别的性能急剧下降。
基于上述问题，发明人提供的一种解决思路中，是对语音分离模型和音素识别模型进行联合训练，从而可以在噪声鲁棒语音识别和多说话人语音识别任务中显著的减小识别错误率。举例而言：
在发明人提出的一种技术方案中，是提供一种独立框架，使语音分离阶段直接在梅尔滤波域中操作，从而与音素识别阶段在特征域上保持一致。但由于语音分离阶段一般不在梅尔滤波域中实现，因此该技术方案可能导致无法得到较优的语音分离结果；同时，语音分离算法在不断进步发展，而这种独立框架难以快速灵活地整合第三方算法。在发明人提出的另一种技术方案中，是提供一种联合框架，其中采用深度神经网络学习梅尔滤波逐帧仿射变换函数；但该方法难以有效的对复杂动态问题进行建模，进而难以处理复杂干扰声条件下的语音识别任务。
基于上述一个或多个问题，本示例实施方式提供了一种实现语音识别的神经网络训练方法。该神经网络训练方法可以应用于上述服务器105；也可以应用于上述终端设备101、102、103中的一个或多个。参考图3所示，该实现语音识别的神经网络训练方法可以包括以下步骤：
步骤S310.获取样本数据，所述样本数据包括混合语音频谱及其标注音素；
步骤S320.通过第一子网络从混合语音频谱中提取目标语音频谱；
步骤S330.通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
步骤S340.通过第三子网络，基于所述中间过渡表征进行音素识别；
步骤S350.根据所述音素识别的结果以及所述标注音素，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。
在本示例实施方式所提供的方法中，可以通过第二子网络对第一子网络提取的目标语音频谱进行适应性转换，得到可以输入第三子网络进行音素识别的中间过渡表征，从而可以完成对语音分离阶段和音素识别阶段的桥接，实现一种端到端的语音识别系统；在此基础上，对第一子网络、第二子网络和第三子网络进行联合训练，降低了语音分离阶段引入的信号误差和信号失真对音素识别阶段的性能影响；因此，在本示例实施方式所提供的方法可以提升在复杂干扰声条件下的语音识别性能，提升用户体验；同时，本示例实施方式中的第一子网络和第三子网络可以方便的整合第三方算法，具有更高灵活性。
下面，在另一实施例中，对上述步骤进行更加详细的说明。
在步骤S310中，获取样本数据，所述样本数据包括混合语音频谱及其标注音素。
本示例实施方式中，可以首先获取多组样本数据，每组样本数据可以包括混合语音以及对于混合语音的标注音素。混合语音可以为受到非平稳噪声干扰的语音信号，例如受背景音乐或多说话人干扰等，从而会出现不同音源的语音混叠的情况，进而导致接收到的语音为混合语音；混合语音的标注音素即表示该混合语音具体包括哪些音素；音素标注方式可以是人工标注，也可以是采用历史识别结果作为标注音素；本示例性实施例中对此不做特殊限定。此外，每组样本数据还可以包括该混合语音对应的参考语音。其中，所述参考语音例如可以是当说话人在安静环境下或者平稳噪声干扰环境下说话时，接收到的单声道语音信号；当然，也可以通过如聚类等其他方法预先从混合语音中提取得到参考语音。
在获取混合语音和参考语音之后，可以按照一定的帧长以及帧移对混合语音和参考语音进行分帧处理，得到混合语音在各帧的语音数据以及参考语音在各帧的语音数据。接着，可以提取混合语音数据以及参考语音的频谱特征。举例而言，本示例实施方式中，可以基于短时傅立叶变换(Short Time Fourier Transform，STFT)或者其他方式提取混合语音数据的频谱特征以及参考语音数据的频谱特征。
例如，本示例实施方式中可以将第n帧的混合语音数据表示为x(n)，混合语 音数据x(n)可以认为是目标语音数据
Figure PCTCN2020110742-appb-000001
和干扰语音数据s I(n)的线性叠加，即
Figure PCTCN2020110742-appb-000002
参考语音数据可以表示为s s(n)。对于混合语音数据x(n)和参考语音数据s s(n)进行短时傅里叶变换后，对短时傅里叶变换的结果取对数，即可得到混合语音数据和参考语音数据的频谱特征；例如，可以将混合语音数据对应的混合语音频谱表示为T×F维向量x，将参考语音数据对应的参考语音频谱表示为T×F维向量s s；其中，T是总帧数，F是每帧的频带个数。
在步骤S320中，通过第一子网络从混合语音频谱中提取目标语音频谱。
本示例实施方式中，将以采用基于理想比率掩模(Ideal Ratio Mask，IRM)的方法提取目标语音频谱为例进行说明；但本示例性实施例中并不以此为限，在本申请的其他示例性实施例中，也可以采用其他方法提取目标语音频谱。参考图4所示，本示例实施方式中，可以通过下述步骤S410至步骤S440提取目标语音频谱。其中：
在步骤S410中，将所述混合语音频谱嵌入到多维向量空间，得到所述混合语音频谱每个时频窗口对应的嵌入向量。
举例而言，本示例实施方式中，可以通过深度神经网络模型将所述混合语音频谱嵌入到K维向量空间。例如，上述深度神经网络可以由多层双向长短时记忆网络(BiLSTM，Bi-Long Short-Term Memory)构成，如包括4层窥孔连接(peephole connection)的双向长短时记忆网络，每层双向长短时记忆网络可有600个隐节点等。当然，上述深度神经网络也可以替换为各种其它有效的网络模型；例如，卷积神经网络(CNN，Convolutional Neural Network)和其它网络结构相结合得到的模型，或者如时延网络、闸控卷积神经网络等其它模型；本申请中不限定深度神经网络的模型类型和拓扑结构。
以双向长短时记忆网络为例，其可以将混合语音频谱从向量空间
Figure PCTCN2020110742-appb-000003
映射到更高维的向量空间
Figure PCTCN2020110742-appb-000004
具体而言，得到的混合语音频谱的嵌入矩阵V如下：
Figure PCTCN2020110742-appb-000005
其中Θ extract表示双向长短时记忆网络
Figure PCTCN2020110742-appb-000006
的网络参数；各时频窗口对应的嵌入向量为V f，t；其中，t∈[1，T]，f∈[1，F]。
在步骤S420中，利用理想比率掩模对所述混合语音的各嵌入向量进行加权规整，得到与所述目标语音频谱对应的吸引子。
举例而言，本示例实施方式中，可以通过|s s|/|x|计算理想比率掩模m s；然 后可以利用理想比率掩模m s对所述混合语音的各嵌入向量进行加权规整，得到与所述目标语音频谱对应的吸引子a s；其中，吸引子
Figure PCTCN2020110742-appb-000007
此外，为了去除低能量频谱窗口噪声以得到有效帧，本示例实施方式中，还可以设置有监督标注w，其中有监督标注
Figure PCTCN2020110742-appb-000008
通过有监督标注w，能够将语音频谱各帧的频谱分别与一频谱阈值比较，如果语音频谱某帧的频谱幅度小于频谱阈值，则该帧频谱的有监督标注取0；否则，取1。以频谱阈值为max(c)/100为例，有监督标注w可以如下所示：
Figure PCTCN2020110742-appb-000009
相应的，与所述目标语音频谱对应的吸引子a s可以如下所示：
Figure PCTCN2020110742-appb-000010
其中，⊙表示矩阵的元素乘法。
在步骤S430中，通过计算所述混合语音的各嵌入向量与吸引子之间的相似度，得到与所述目标语音频谱对应的目标掩蔽矩阵。
举例而言，本示例实施方式中，可以计算所述混合语音的各嵌入向量与吸引子之间距离，并将各距离取值映射到[0,1]范围，从而表征各嵌入向量与吸引子之间的相似度。例如，可以通过下式计算所述混合语音的各嵌入向量V f，t与吸引子a s之间的相似度，得到与所述目标语音频谱对应的目标掩蔽矩阵
Figure PCTCN2020110742-appb-000011
Figure PCTCN2020110742-appb-000012
Sigmoid为S型函数，可以将变量映射到[0,1]范围，从而便于后续目标语音频谱的提取。此外，在本申请的其他示例性实施例中，也可以基于如tanh函数或者其他方式计算所述混合语音的各嵌入向量与吸引子之间的相似度，并得到与所述目标语音频谱对应的目标掩蔽矩阵，这同样属于本申请的保护范围。
在步骤S440中，基于所述目标掩蔽矩阵，从所述混合语音频谱中提取所述目标语音频谱。
本示例实施方式中，可以利用目标掩蔽矩阵
Figure PCTCN2020110742-appb-000013
对上述混合语音频谱x进行加权，即可逐时频窗口提取上述混合语音频谱中的目标语音频谱；对于某时频窗口的混合语音频谱x f，t而言，目标掩蔽矩阵越大，相应该时频窗口的频谱信息也将 被提取得越多。举例而言，可以通过下式提取得到所述目标语音频谱
Figure PCTCN2020110742-appb-000014
Figure PCTCN2020110742-appb-000015
此外，本示例实施方式中，还可以获取在对各组所述样本数据进行训练时计算得到的上述吸引子，并计算各所述吸引子的均值，得到全局吸引子，从而便于在测试阶段，用以提取目标语音频谱。
在步骤S330中，通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征。
本示例实施方式中，第二子网络用于桥接上述第一子网络和下述第三子网络；其输入为第一子网络提取的目标语音频谱(下文将记为S，S＝{S 1，S 2，...，S T})，其输出的中间过渡表征的最终训练目标是最小化第三子网络的识别损失。基于此，本示例实施方式中，可以根据所述目标语音频谱的时频窗口顺序，对各时频窗口的目标语音频谱依次进行适应性转换；其中，针对一所述时频窗口的转换过程可以包括：根据当前转换过程针对的时频窗口的目标语音频谱和前一转换过程的隐含状态信息，生成当前转换过程的隐含状态信息；以及基于各所述隐含状态信息，得到当前转换过程针对的时频窗口的所述中间过渡表征。下面以基于LSTM(Long Short-Term Memory，长短时记忆)网络为例，对上述转换过程进行详细说明。
参考图5，LSTM网络为一个处理单元(以下简称LSTM单元)，LSTM单元通常包括遗忘门、输入门以及输出门；本示例实施方式中，可以通过一个LSTM单元执行一个上述转换过程。参考图6所示，图6为一个LSTM单元生成当前转换过程的隐含状态信息的过程，其可以包括下述步骤S610至步骤S650。其中：
在步骤S610中，根据当前时频窗口的目标语音频谱以及上一转换过程的隐含状态信息，计算候选状态信息、所述候选状态信息的输入权重、上一转换过程目标状态信息的遗忘权重和当前转换过程目标状态信息的输出权重。具体而言：
遗忘门用于决定从上一转换过程的目标状态信息中丢弃多少信息，所以遗忘权重用于表示上一转换过程的目标状态信息不被遗忘的(即能够保留的)权重；遗忘权重实质上可以为权重矩阵。示例性的，可以通过用于表征遗忘门的激活函数，对当前时频窗口的目标语音频谱以及上一转换过程的隐含状态信息进行编码，映射到0到1之间的数值，得到上一转换过程目标状态信息的遗忘权重。其中，0表示完全舍弃，1表示完全保留。例如，可以按照以下公式计算得到上一转换过程目标状态信息的遗忘权重f t：
f t＝σ(W f·[h t-1，S t]+b f)
其中，h t-1表示上一转换过程的隐含状态信息，S t表示当前时频窗口的目标 语音频谱，σ表示激活函数Sigmod函数，W f和b f表示遗忘门中Sigmod函数的参数，[h t-1，S t]表示将h t-1和S t进行组合。
输入门用于决定当前输入的目标语音频谱中，有多少信息是重要且需要保留下来的。示例性的，可以通过用于表征输入门的激活函数，对当前时频窗口的目标语音频谱以及上一转换过程的隐含状态信息进行编码，得到候选状态信息以及所述候选状态信息的输入权重；其中，所述候选状态信息的输入权重用于决定候选状态信息中有多少新的信息可以加入到目标状态信息中。
例如，可以按照以下公式计算得到候选状态信息
Figure PCTCN2020110742-appb-000016
Figure PCTCN2020110742-appb-000017
其中，tanh表示激活函数为正曲双切函数，W c和b c表示输入门中tanh函数的参数。
并可以按照以下公式计算得到候选状态信息的输入权重i t：
i t＝σ(W i·[h t-1，S t]+b i)
其中，σ表示激活函数Sigmod函数，W i和b i表示输入门中Sigmod函数的参数。
输出门用于确定输出至下一个LSTM单元的隐含状态信息中应该包含哪些信息。示例性的，可以通过用于表征输出门的激活函数，对当前时频窗口的目标语音频谱以及上一转换过程的隐含状态信息进行编码，得到当前转换过程目标状态信息的输出权重。例如，可以按照以下公式计算得到候选状态信息o t：
o t＝σ(W o·[h t-1，S t]+b o)
其中，σ表示激活函数Sigmod函数，W o和b o表示输出门中Sigmod函数的参数。
在步骤S620中，根据所述遗忘权重对所述上一转换过程目标状态信息进行保留，得到第一中间状态信息。举例而言，得到的第一中间状态信息可以为
Figure PCTCN2020110742-appb-000018
其中，C t-1表示上一转换过程目标状态信息。
在步骤S630中，根据所述候选状态信息的输入权重对所述候选状态信息进行保留，得到第二中间状态信息。举例而言，得到的第二中间状态信息可以为
Figure PCTCN2020110742-appb-000019
在步骤S640中，根据所述第一中间状态信息和第二中间状态信息，得到所述 当前转换过程目标状态信息。举例而言，当前转换过程目标状态信息
Figure PCTCN2020110742-appb-000020
在步骤S650中，根据所述当前转换过程目标状态信息的输出权重对所述当前转换过程目标状态信息进行保留，得到当前转换过程的隐含状态信息。举例而言，当前转换过程的隐含状态信息
Figure PCTCN2020110742-appb-000021
进一步的，在上述适应性转换中，是对各时频窗口的目标语音频谱顺序进行适应性转换得到隐含状态信息h t，即通过前向LSTM进行的适应性转换。本示例实施方式中，还可以通过双向长短时记忆(BiLSTM)网络进行适应性转换；更进一步的，在其他示例性实施例中，还可以使用窥孔连接的多层双向长短期记忆网络进行适应性转换，从而进一步的提升适应性转换的准确性。举例而言，可以在上述适应性转换过程的基础上，对各时频窗口的目标语音频谱逆序进行适应性转换得到隐含状态信息
Figure PCTCN2020110742-appb-000022
然后将隐含状态信息h t和隐含状态信息
Figure PCTCN2020110742-appb-000023
进行拼接得到BiLSTM网络的输出，即隐含状态信息H t，从而可以通过隐含状态信息H t更好的表征双向时序依赖特征。
为了使隐含状态信息H t能够更好的适配后续第三子网络，本示例实施方式中，还可以对所述各隐含状态信息进行以下一种或多种处理，得到当前转换过程针对的时频窗口的所述中间过渡表征。举例而言：
在fbank特征的标准计算过程中，会将输入的频谱进行平方运算，因此得到的fbank特征必然是非负的。为了匹配fbank特征的非负性，本示例实施方式中，可以对BiLSTM网络的输出进行平方处理，从而实现非负映射。此外，在本申请的其他示例性实施例中，也可以通过如ReLU(Rectified Linear Unit，线性整流函数)等其他方式实现非负映射，本示例性实施例中对此不做特殊限定。例如，非负映射结果可以如下式：
Figure PCTCN2020110742-appb-000024
其中D表示中间过渡表征的维度，Θ adapt表示双向长短时记忆网络
Figure PCTCN2020110742-appb-000025
的网络参数。
在进行非负映射处理之后，还可以对
Figure PCTCN2020110742-appb-000026
进行一系列的可微分运算，例如逐元素求对数、计算一阶差分、计算二阶差分等；此外，还可以执行全局均值方差归一化以及添加前后时频窗口的特征。例如，对于当前时频窗口，可以将当前时频窗口的特征、当前时频窗口之前W个时频窗口的特征以及当前时频窗口之后W 个时频窗口的特征，即共计2W+1个时频窗口的特征进行拼接，得到当前时频窗口的中间过渡表征；经过上述处理之后得到中间过渡表征
Figure PCTCN2020110742-appb-000027
需要说明的是，在本申请的其他示例性实施例中，也可以从上述处理过程中选择部分处理过程执行，还可以选择其他方式进行处理，这些同样属于本申请的保护范围。
在步骤S340中，通过第三子网络，基于所述中间过渡表征进行音素识别。
本示例实施方式中，可以将上述第二子网络输出的中间过渡表征f输入至第三子网络，获得所述中间过渡表征包括的音素的后验概率y t。举例而言，所述第三子网络可以是基于最优中心损失(Center-Loss-optimal)的卷积长短期记忆深度神经网络(Convolutional Long-Short-Term-Memory Deep-Neural-Network，CLDNN)，在下文可以将其表示为CL_CLDNN网络。中间过渡表征f输入至CL_CLDNN网络之后，可以进行如下式所示的运算：
Figure PCTCN2020110742-appb-000028
y t＝Softmax(Wu t+b)
其中，u t是CL_CLDNN网络倒数第二层(如下文所述的多个全连接层中的倒数第二层等)第t帧的输出；Softmax(z)＝e z/||e z|| 1，可以用于计算音素的后验概率；Θ recog＝{Γ，W，b}表示CL_CLDNN网络的参数。
下面，对CL_CLDNN网络的具体处理过程加以说明。参考图7所示，第三子网络可以通过下述步骤S710至步骤S730基于所述中间过渡表征进行音素识别。其中：
在步骤S710中，通过至少一个卷积层将多维滤波器应用于所述中间过渡表征，生成卷积层输出，从而可以降低谱差异性。例如，本示例实施方式中，可以包括两层卷积层，每层卷积层可以包括256个特征图；第一层卷积层可以采用9×9时域-频域滤波器，第二层可以采用4×3时域-频域滤波器。此外，由于最后一层卷积层的输出维度可能会很大，本示例实施方式中还可以在最后一层卷积层之后接一个线性层来进行降维。
在步骤S720中，在至少一个递归层中使用所述卷积层输出以生成递归层输出，从而实现时序特征建模。例如，本示例实施方式中，所述递归层可以包括多层长短期记忆网络；如可以在上述线性层之后接两层长短期记忆网络，每个长短期记忆网络可以采用832个处理单元、512维映射层来降维。在本申请的其他示 例性实施例中，所述递归层也可以包括如GRU(Gated Recurrent Unit，门控循环单元)网络或者其他RNN(Recurrent Neural Networks，循环神经网络)结构，本示例性实施例中对此不做特殊限定。
在步骤S730中，将所述递归层输出提供至至少一个全连接层，并对所述全连接层的输出应用非线性函数，以获得所述中间过渡表征包括的音素的后验概率。本示例实施方式中，所述全连接层例如可以为两层DNN(Deep Neural Network，深度神经网络)结构；每层DNN结构可以包括1024个神经元，通过DNN结构可以将特征空间映射到更容易分类的输出层。输出层可以采用如上述Softmax函数或者tanh函数等非线性函数进行分类，从而获得所述中间过渡表征包括的音素的后验概率。
在步骤S350中，根据所述音素识别的结果以及所述标注音素，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。
举例而言，本示例实施方式中，可以首先确定所述第一子网络、第二子网络以及第三子网络的联合损失函数。例如，本示例实施方式中可以基于上述中心损失(Center-Loss-optimal)和交叉熵损失作为联合损失函数；当然，在本申请的其他示例性实施例中，也可以采用其他损失作为联合损失函数，本示例性实施例中并不以此为限。
在确定联合损失函数之后，可以将所述音素识别的结果以及所述标注音素输入所述联合损失函数，计算得到联合损失函数值。在得到联合损失函数值之后，则可以根据所述联合损失函数值，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。例如，可以以最小化联合损失函数值为训练目标，采用如随机梯度下降(Stochastic Gradient Descent，SGD)以及反向传播(Back Propagation，BP)等方法对第一子网络、第二子网络以及第三子网络的参数进行更新，直至收敛，如训练迭代次数达到最大次数或者联合损失函数值不再减小等。
本示例实施方式还提供了一种基于神经网络的语音识别方法，该神经网络可以通过上述示例性实施例中的训练方法训练得到。该语音识别方法可以应用于上述终端设备101、102、103中的一个或多个；也可以应用于上述服务器105。参考图8所示，该语音识别方法可以包括下述步骤S810至步骤S840。其中：
在步骤S810中，获取待识别的混合语音频谱。
本示例实施方式中，混合语音可以为受到非平稳噪声干扰的语音信号，例如受背景音乐或多说话人干扰等，从而会出现不同音源的语音混叠的情况，进而导致接收到的语音为混合语音。在获取混合语音之后，可以按照一定的帧长以及帧移对混合语音进行分帧处理，得到混合语音在各帧的语音数据。接着，可以提取 混合语音数据的频谱特征。举例而言，本示例实施方式中，可以基于短时傅立叶变换(Short Time Fourier Transform，STFT)或者其他方式提取混合语音数据的频谱特征。
例如，本示例实施方式中可以将第n帧的混合语音数据表示为x(n)，混合语音数据x(n)可以认为是目标语音数据
Figure PCTCN2020110742-appb-000029
和干扰语音数据s I(n)的线性叠加，即
Figure PCTCN2020110742-appb-000030
对于混合语音数据x(n)进行短时傅里叶变换后，对短时傅里叶变换的结果取对数，即可得到混合语音数据频谱特征；例如，可以将混合语音数据对应的混合语音频谱表示为T×F维向量x；其中，T是总帧数，F是每帧的频带个数。
在步骤S820中，通过第一子网络从混合语音频谱中提取目标语音频谱。
本示例实施方式中，将以采用基于理想比率掩模(Ideal Ratio Mask，IRM)的方法提取目标语音频谱为例进行说明；但本示例性实施例中并不以此为限，在本申请的其他示例性实施例中，也可以采用其他方法提取目标语音频谱。举例而言：
首先，将所述混合语音频谱嵌入到多维向量空间，得到所述混合语音频谱每个时频窗口对应的嵌入向量。以双向长短时记忆网络为例，其可以将混合语音频谱从向量空间
Figure PCTCN2020110742-appb-000031
映射到更高维的向量空间
Figure PCTCN2020110742-appb-000032
具体而言，得到的混合语音频谱的嵌入矩阵V如下：
Figure PCTCN2020110742-appb-000033
其中Θ extract表示双向长短时记忆网络
Figure PCTCN2020110742-appb-000034
的网络参数；各时频窗口对应的嵌入向量为V f，t；其中，t∈[1，T]，f∈[1，F]。
接着，获取上述训练过程中步骤S320得到的全局吸引子
Figure PCTCN2020110742-appb-000035
并通过计算所述混合语音的各嵌入向量与全局吸引子之间的相似度，得到与所述目标语音频谱对应的目标掩蔽矩阵。例如，可以通过下式计算所述混合语音的各嵌入向量V f，t与全局吸引子
Figure PCTCN2020110742-appb-000036
之间的相似度，得到与所述目标语音频谱对应的目标掩蔽矩阵
Figure PCTCN2020110742-appb-000037
Figure PCTCN2020110742-appb-000038
然后，基于所述目标掩蔽矩阵，从所述混合语音频谱中提取所述目标语音频谱。例如，可以通过下式提取得到所述目标语音频谱
Figure PCTCN2020110742-appb-000039
Figure PCTCN2020110742-appb-000040
在步骤S830中，通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征。
本示例实施方式中，可以根据所述目标语音频谱的时频窗口顺序，对各时频窗口的目标语音频谱依次进行适应性转换；并且，针对一所述时频窗口的转换过程可以包括：根据当前转换过程针对的时频窗口的目标语音频谱和前一转换过程的隐含状态信息，生成当前转换过程的隐含状态信息；以及基于各所述隐含状态信息，得到当前转换过程针对的时频窗口的所述中间过渡表征。例如，本示例实施方式中，可以通过双向长短时记忆(BiLSTM)网络的各LSTM单元执行上述各个转换过程。
为了匹配fbank特征的非负性，本示例实施方式中，还可以对BiLSTM网络的输出进行平方处理，从而实现非负映射。例如，非负映射结果可以如下式：
Figure PCTCN2020110742-appb-000041
其中D表示中间过渡表征的维度，Θ adapt表示双向长短时记忆网络
Figure PCTCN2020110742-appb-000042
的网络参数。
在进行非负映射处理之后，还可以对
Figure PCTCN2020110742-appb-000043
进行一系列的可微分运算，例如逐元素求对数、计算一阶差分、计算二阶差分等；此外，还可以执行全局均值方差归一化以及添加前后时频窗口的特征。例如，对于当前时频窗口，可以将当前时频窗口的特征、当前时频窗口之前W个时频窗口的特征以及当前时频窗口之后W个时频窗口的特征，即共计2W+1个时频窗口的特征进行拼接，得到当前时频窗口的中间过渡表征；经过上述处理之后得到中间过渡表征
Figure PCTCN2020110742-appb-000044
在步骤S840中，通过第三子网络，基于所述中间过渡表征进行音素识别。
本示例实施方式中，可以将上述第二子网络输出的中间过渡表征f输入至第三子网络，获得所述中间过渡表征包括的音素的后验概率y t。举例而言，所述第三子网络可以是CL_CLDNN网络。中间过渡表征f输入至CL_CLDNN网络之后，可以进行如下式所示的运算：
Figure PCTCN2020110742-appb-000045
y t＝Softmax(Wu t+b)
其中，u t是CL_CLDNN网络倒数第二层(如下文所述的多个全连接层中的 倒数第二层等)第t帧输出；Softmax(z)＝e z/||e z|| 1，可以用于计算音素的后验概率；Θ recog＝{Γ，W，b}表示CL_CLDNN网络的参数。
下面，以描述自动语音识别系统的实现为例，结合上述方法实现进行阐述。参考图9所示，自动语音识别系统可以包括第一子网络910、第二子网络920和第三子网络930。其中：
第一子网络910可以从混合语音频谱中提取目标语音频谱。参考图9所示，第一子网络可以包括窥孔连接的多层(如四层)BiLSTM网络，每层BiLSTM网络可以包括600个隐节点；同时，在最后一层BiLSTM网络之后可以接一个全连接层，用于将600维的隐含状态信息映射为24000维嵌入向量。混合语音频谱例如可以是采样率为16000Hz，分帧帧长为25ms、帧移为10ms的512维的短时傅立叶变换频谱特征。将混合语音频谱输入第一子网络910之后，即可以通过BiLSTM网络将其映射为嵌入向量；然后，可以计算嵌入向量与吸引子之间的相似度，得到目标掩蔽矩阵；进而，可以基于目标掩蔽矩阵，从混合语音频谱中提取目标语音频谱S。在训练阶段，还可以向第一子网络910输入参考语音频谱，从而可以根据参考语音频谱计算理想比率掩模，进而可以根据理想比率掩模对混合语音的各嵌入向量进行加权规整，得到所述吸引子。
第二子网络920可以用于对所述目标语音频谱进行适应性转换以得到中间过渡表征。参考图9所示，第二子网络920可以包括窥孔连接的多层(如两层)BiLSTM网络，每层BiLSTM网络可以包括600个隐节点。将第一子网络输出的目标语音频谱S输入BiLSTM网络之后，可以得到BiLSTM网络输出的隐含状态信息H，H＝{H 1，H 1，...，H T}。接着，可以对隐含状态信息H进行非负映射、逐元素求对数、计算一阶差分、计算二阶差分、执行全局均值方差归一化以及添加前后时频窗口的特征等预设处理，得到中间过渡表征f。本示例实施方式中，中间过渡表征f例如可以为40维的fbank特征向量。
第三子网络930可以用于基于所述中间过渡表征进行音素识别。参考图9所示，第三子网络920可以包括CL_CLDNN网络。将中间过渡表征f输入至第三子网络之后，可以得到中间过渡表征包括的音素的后验概率y t；以中文普通话为例，可以输出大约在1.2万个类别的音素上的后验概率。
在具体训练时，可以将样本数据的批量大小设置为24，将最初的学习速率α设置为10 -4，将学习速率的衰退系数设置为0.8，将收敛判断条件设置为综合损失函数值连续3次迭代(epoch)均没有改善，将嵌入向量的维度K设置为40，将 梅尔滤波频带个数D设置为40，将添加前后时频窗口的特征时的时频窗口数量W设置为5，将中心损失的权重λ设置为0.01。此外，CL_CLDNN网络中的卷积层和LSTM网络的输出可以均进行批量归一化，以实现更快的收敛和更好的泛化。
参考图10A和图10B所示，是上述自动语音识别系统的语音识别效果参考图。图10A中为受背景音乐干扰的语音识别任务，图10B中为受其他说话人干扰的语音识别任务。在图10A和图10B中，纵轴通过相对单词错误率下降(Relative Word Error Rate Reduction,WERR)来表征识别效果，横轴表示不同分贝dB(分贝)的信噪比干扰测试条件，共五种信噪比：0dB、5dB、10dB、15dB和20dB。
在图10A和图10B中，线条P1和线条P4表示本示例实施方式中自动语音识别系统和基线系统对比得到的WERR；线条P2和线条P5表示现有先进自动语音识别系统(如采用DNN学习梅尔滤波逐帧仿射变换函数的鲁棒语音识别联合训练架构)和基线系统对比得到的WERR；线条P3表示本示例实施方式中自动语音识别系统结合目标说话人跟踪处理后，和基线系统对比得到的WERR。
其中，上述现有先进自动语音识别系统与本示例实施方式中自动语音识别系统在参数复杂度上相当；但从图10A和图10B可以看出，本示例实施方式中自动语音识别系统的WERR在两种识别任务下，均明显优于现有先进自动语音识别系统；这表明了本示例实施方式中自动语音识别系统能够对有时态复杂性的问题进行有效建模，从而进一步可以提升在复杂干扰声条件下的语音识别性能。
此外，除了识别性能得到显著提升之外，本示例实施方式中的自动语音识别系统还具有高度的灵活性，例如允许灵活的集成各种类型的语音分离模块和音素识别模块至第一子网络和第三子网络，且本示例实施方式中的自动语音识别系统不会以任何单个模块性能受损作为代价。
因此，将本示例实施方式中的自动语音识别系统应用到包括智能音箱、智能电视、在线语音识别系统、智能语音助手、同声传译以及虚拟人等多个项目和产品应用中，可以显著地改善自动语音识别的准确率，尤其是在复杂干扰环境中的识别性能；进而可以提升用户体验。
应当注意，尽管在附图中以特定顺序描述了本申请中方法的各个步骤，但是，这并非要求或者暗示必须按照该特定顺序来执行这些步骤，或是必须执行全部所示的步骤才能实现期望的结果。附加的或备选的，可以省略某些步骤，将多个步骤合并为一个步骤执行，以及/或者将一个步骤分解为多个步骤执行等。
进一步的，本示例实施方式中，还提供了一种实现语音识别的神经网络训练装置。该神经网络训练装置可以应用于服务器；也可以应用于终端设备。所述神经网络包括第一至第三子网络，参考图11所示，该神经网络训练装置1100可以 包括数据获取模块1110、目标语音提取模块1120、适应性转换模块1130、语音识别模块1140以及参数更新模块1150。其中：
数据获取模块1110可以用于获取样本数据，所述样本数据包括混合语音频谱及其标注音素；
目标语音提取模块1120可以用于通过第一子网络从混合语音频谱中提取目标语音频谱；
适应性转换模块1130可以用于通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
语音识别模块1140可以用于通过第三子网络，基于所述中间过渡表征进行音素识别；
参数更新模块1150可以用于根据所述音素识别的结果以及所述标注音素，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。
在本申请的一种示例性实施例中，所述目标语音提取模块1120通过下述步骤从混合语音频谱中提取目标语音频谱：将所述混合语音频谱嵌入到多维向量空间，得到所述混合语音频谱每个时频窗口对应的嵌入向量；利用理想比率掩模对所述混合语音频谱的各嵌入向量进行加权规整，得到与所述目标语音频谱对应的吸引子；通过计算所述混合语音频谱的各嵌入向量与吸引子之间的相似度，得到与所述目标语音频谱对应的目标掩蔽矩阵；基于所述目标掩蔽矩阵，从所述混合语音频谱中提取所述目标语音频谱。
在本申请的一种示例性实施例中，所述装置还包括：
全局吸引子计算模块可以用于获取各所述样本数据对应的所述吸引子，并计算各所述吸引子的均值，得到全局吸引子。
在本申请的一种示例性实施例中，所述适应性转换模块1130通过下述步骤对所述目标语音频谱进行适应性转换：根据所述目标语音频谱的时频窗口顺序，对各时频窗口的目标语音频谱依次进行适应性转换；其中，针对一所述时频窗口的转换过程包括：
根据当前转换过程针对的时频窗口的目标语音频谱和前一转换过程的隐含状态信息，生成当前转换过程的隐含状态信息；以及基于各所述隐含状态信息，得到当前转换过程针对的时频窗口的所述中间过渡表征。
在本申请的一种示例性实施例中，所述适应性转换模块1130通过下述步骤生成当前转换过程的隐含状态信息：根据当前时频窗口的目标语音频谱以及上一转换过程的隐含状态信息，计算候选状态信息、所述候选状态信息的输入权重、上一转换过程目标状态信息的遗忘权重和当前转换过程目标状态信息的输出权重； 根据所述遗忘权重对所述上一转换过程目标状态信息进行保留，得到第一中间状态信息；根据所述候选状态信息的输入权重对所述候选状态信息进行保留，得到第二中间状态信息；根据所述第一中间状态信息和第二中间状态信息，得到所述当前转换过程目标状态信息；根据所述当前转换过程目标状态信息的输出权重对所述当前转换过程目标状态信息进行保留，得到当前转换过程的隐含状态信息。
在本申请的一种示例性实施例中，所述适应性转换模块1130通过下述步骤基于各所述隐含状态信息得到当前转换过程针对的时频窗口的所述中间过渡表征：对各所述隐含状态信息进行以下一种或多种处理，得到当前转换过程针对的时频窗口的所述中间过渡表征：
非负映射、逐元素求对数、计算一阶差分、计算二阶差分、执行全局均值方差归一化以及添加前后时频窗口的特征。
在本申请的一种示例性实施例中，所述语音识别模块1140通过下述步骤基于所述中间过渡表征进行音素识别：通过至少一个卷积层将多维滤波器应用于所述中间过渡表征，生成卷积层输出；在至少一个递归层中使用所述卷积层输出以生成递归层输出；将所述递归层输出提供至至少一个全连接层，并对所述全连接层的输出应用非线性函数，以获得所述中间过渡表征包括的音素的后验概率。
在本申请的一种示例性实施例中，所述递归层包括长短期记忆网络。
在本申请的一种示例性实施例中，所述参数更新模块1150通过下述步骤对所述第一子网络、第二子网络以及第三子网络的参数进行更新：确定所述第一子网络、第二子网络以及第三子网络的联合损失函数；根据所述音素识别的结果、所述标注音素以及所述联合损失函数，计算联合损失函数值；根据所述联合损失函数值，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。
在本申请的一种示例性实施例中，所述第一子网络包括窥孔连接的多层长短期记忆网络；所述第二子网络包括窥孔连接的多层长短期记忆网络。
进一步的，本示例实施方式中，还提供了一种基于神经网络的语音识别装置。语音识别装置可以应用于服务器；也可以应用于终端设备。所述神经网络包括第一至第三子网络，参考图12所示，该神经网络训练装置1200可以包括数据获取模块1210、目标语音提取模块1220、适应性转换模块1230以及语音识别模块1240。其中：
数据获取模块1210可以用于获取待识别的混合语音频谱；
目标语音提取模块1220可以用于通过第一子网络从混合语音频谱中提取目标语音频谱；
适应性转换模块1230可以用于通过第二子网络对所述目标语音频谱进行适 应性转换以得到中间过渡表征；
语音识别模块1240可以用于通过第三子网络，基于所述中间过渡表征进行音素识别。
本申请示例实施方式所提供方法中，可以通过第二子网络对第一子网络提取的目标语音频谱进行适应性转换，得到可以输入第三子网络进行音素识别的中间过渡表征，从而可以完成对语音分离阶段和音素识别阶段的桥接，实现一种端到端的语音识别系统；在此基础上，对第一子网络、第二子网络和第三子网络进行联合训练，降低了语音分离阶段引入的信号误差和信号失真对音素识别阶段的性能影响；因此，本申请示例实施方式所提供方法可以提升在复杂干扰声条件下的语音识别性能，提升用户体验；同时，本申请示例实施方式中的第一子网络和第三子网络可以方便的整合第三方算法，具有更高灵活性。
上述装置中各模块或单元的具体细节已经在对应的方法示例性实施例中进行了详细的描述，因此此处不再赘述。
应当注意，尽管在上文详细描述中提及了用于动作执行的设备的若干模块或者单元，但是这种划分并非强制性的。实际上，根据本申请的实施方式，上文描述的两个或更多模块或者单元的特征和功能可以在一个模块或者单元中具体化。反之，上文描述的一个模块或者单元的特征和功能可以进一步划分为由多个模块或者单元来具体化。
作为另一方面，本申请还提供了一种计算机可读介质，该计算机可读介质可以是上述实施例中描述的电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备实现如上述实施例中所述的方法。例如，所述的电子设备可以实现上述示例性实施例中的各个步骤等。
需要说明的是，本申请所示的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是，但不限于，电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本申请中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本申请中，计算机可读的信号介质可以包括在基带中或者作为载 波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。
附图中的流程图和框图，图示了按照本申请各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
应当理解的是，本申请并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本申请的范围仅由所附的权利要求来限制。

Claims (20)

  1. 一种实现语音识别的神经网络训练方法，由电子设备执行，其中，所述神经网络包括第一子网络，第二子网络和第三子网络；所述方法包括：
    获取样本数据，所述样本数据包括混合语音频谱及其标注音素；
    通过所述第一子网络从混合语音频谱中提取目标语音频谱；
    通过所述第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
    通过所述第三子网络，基于所述中间过渡表征进行音素识别；
    根据所述音素识别的结果以及所述标注音素，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。
  2. 根据权利要求1所述的神经网络训练方法，通过第一子网络从混合语音频谱中提取目标语音频谱，包括：
    将所述混合语音频谱嵌入到多维向量空间，得到所述混合语音频谱每个时频窗口对应的嵌入向量；
    利用理想比率掩模对所述混合语音频谱的各嵌入向量进行加权规整，得到与所述目标语音频谱对应的吸引子；
    通过计算所述混合语音频谱的各嵌入向量与吸引子之间的相似度，得到与所述目标语音频谱对应的目标掩蔽矩阵；
    基于所述目标掩蔽矩阵，从所述混合语音频谱中提取所述目标语音频谱。
  3. 根据权利要求2所述的神经网络训练方法，所述方法还包括：
    获取各所述样本数据对应的所述吸引子，并计算各所述吸引子的均值，得到全局吸引子。
  4. 根据权利要求1所述的神经网络训练方法，所述通过第二子网络对所述目标语音频谱进行适应性转换，包括：
    根据所述目标语音频谱的时频窗口顺序，对各时频窗口的目标语音频谱依次进行适应性转换；其中，针对一所述时频窗口的转换过程包括：
    根据当前转换过程针对的时频窗口的目标语音频谱和前一转换过程的隐含状态信息，生成当前转换过程的隐含状态信息；以及
    基于各所述隐含状态信息，得到当前转换过程针对的时频窗口的所述中间过渡表征。
  5. 根据权利要求4所述的神经网络训练方法，生成当前转换过程的隐含状态信息，包括：
    根据当前时频窗口的目标语音频谱以及上一转换过程的隐含状态信息，计算 候选状态信息、所述候选状态信息的输入权重、上一转换过程目标状态信息的遗忘权重和当前转换过程目标状态信息的输出权重；
    根据所述遗忘权重对所述上一转换过程目标状态信息进行保留，得到第一中间状态信息；
    根据所述候选状态信息的输入权重对所述候选状态信息进行保留，得到第二中间状态信息；
    根据所述第一中间状态信息和第二中间状态信息，得到所述当前转换过程目标状态信息；
    根据所述当前转换过程目标状态信息的输出权重对所述当前转换过程目标状态信息进行保留，得到当前转换过程的隐含状态信息。
  6. 根据权利要求4所述的神经网络训练方法，基于各所述隐含状态信息，得到当前转换过程针对的时频窗口的所述中间过渡表征，包括：
    对各所述隐含状态信息进行以下一种或多种处理，得到当前转换过程针对的时频窗口的所述中间过渡表征：
    非负映射、逐元素求对数、计算一阶差分、计算二阶差分、执行全局均值方差归一化以及添加前后时频窗口的特征。
  7. 根据权利要求1所述的神经网络训练方法，通过第三子网络基于所述中间过渡表征进行音素识别，包括：
    通过至少一个卷积层将多维滤波器应用于所述中间过渡表征，生成卷积层输出；
    在至少一个递归层中使用所述卷积层输出以生成递归层输出；
    将所述递归层输出提供至至少一个全连接层，并对所述全连接层的输出应用非线性函数，以获得所述中间过渡表征包括的音素的后验概率。
  8. 根据权利要求7所述的神经网络训练方法，所述递归层包括长短期记忆网络。
  9. 根据权利要求1所述的神经网络训练方法，对所述第一子网络、第二子网络以及第三子网络的参数进行更新，包括：
    确定所述第一子网络、第二子网络以及第三子网络的联合损失函数；
    根据所述音素识别的结果、所述标注音素以及所述联合损失函数，计算联合损失函数值；
    根据所述联合损失函数值，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。
  10. 根据权利要求1～9任意一项所述的神经网络训练方法，所述第一子网络 包括窥孔连接的多层长短期记忆网络；所述第二子网络包括窥孔连接的多层长短期记忆网络。
  11. 一种基于神经网络的语音识别方法，由电子设备执行，其中，所述神经网络包括第一子网络，第二子网络和第三子网络；所述方法包括：
    获取待识别的混合语音频谱；
    通过所述第一子网络从混合语音频谱中提取目标语音频谱；
    通过所述第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
    通过所述第三子网络，基于所述中间过渡表征进行音素识别。
  12. 一种实现语音识别的神经网络训练装置，其中，所述神经网络包括第一子网络，第二子网络和第三子网络；所述装置包括：
    数据获取模块，用于获取样本数据，所述样本数据包括混合语音频谱及其标注音素；
    目标语音提取模块，用于通过第一子网络从混合语音频谱中提取目标语音频谱；
    适应性转换模块，用于通过第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
    语音识别模块，用于通过第三子网络，基于所述中间过渡表征进行音素识别；
    参数更新模块，用于根据所述音素识别的结果以及所述标注音素，对所述第一子网络、第二子网络以及第三子网络的参数进行更新。
  13. 根据权利要求12所述的装置，其中，所述目标语音提取模块进一步用于：
    将所述混合语音频谱嵌入到多维向量空间，得到所述混合语音频谱每个时频窗口对应的嵌入向量；利用理想比率掩模对所述混合语音频谱的各嵌入向量进行加权规整，得到与所述目标语音频谱对应的吸引子；通过计算所述混合语音频谱的各嵌入向量与吸引子之间的相似度，得到与所述目标语音频谱对应的目标掩蔽矩阵；基于所述目标掩蔽矩阵，从所述混合语音频谱中提取所述目标语音频谱。
  14. 根据权利要求13所述的装置，其中，所述装置还包括：
    全局吸引子计算模块，用于获取各所述样本数据对应的所述吸引子，并计算各所述吸引子的均值，得到全局吸引子。
  15. 根据权利要求12所述的装置，其中，所述适应性转换模块进一步用于：
    根据所述目标语音频谱的时频窗口顺序，对各时频窗口的目标语音频谱依次进行适应性转换；其中，针对一所述时频窗口的转换过程包括：
    根据当前转换过程针对的时频窗口的目标语音频谱和前一转换过程的隐含状 态信息，生成当前转换过程的隐含状态信息；以及基于各所述隐含状态信息，得到当前转换过程针对的时频窗口的所述中间过渡表征。
  16. 根据权利要求15所述的装置，其中，所述适应性转换模块进一步用于：
    根据当前时频窗口的目标语音频谱以及上一转换过程的隐含状态信息，计算候选状态信息、所述候选状态信息的输入权重、上一转换过程目标状态信息的遗忘权重和当前转换过程目标状态信息的输出权重；根据所述遗忘权重对所述上一转换过程目标状态信息进行保留，得到第一中间状态信息；根据所述候选状态信息的输入权重对所述候选状态信息进行保留，得到第二中间状态信息；根据所述第一中间状态信息和第二中间状态信息，得到所述当前转换过程目标状态信息；根据所述当前转换过程目标状态信息的输出权重对所述当前转换过程目标状态信息进行保留，得到当前转换过程的隐含状态信息。
  17. 根据权利要求15所述的装置，其中，所述适应性转换模块进一步用于：
    基于各所述隐含状态信息得到当前转换过程针对的时频窗口的所述中间过渡表征：对所述各所述隐含状态信息进行以下一种或多种处理，得到当前转换过程针对的时频窗口的所述中间过渡表征：
    非负映射、逐元素求对数、计算一阶差分、计算二阶差分、执行全局均值方差归一化以及添加前后时频窗口的特征。
  18. 一种基于神经网络的语音识别装置，其中，所述神经网络包括第一子网络，第二子网络和第三子网络；所述装置包括：
    数据获取模块，用于获取待识别的混合语音频谱；
    目标语音提取模块，用于通过所述第一子网络从混合语音频谱中提取目标语音频谱；
    适应性转换模块，用于通过所述第二子网络对所述目标语音频谱进行适应性转换以得到中间过渡表征；
    语音识别模块，用于通过所述第三子网络，基于所述中间过渡表征进行音素识别。
  19. 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1-11任一项所述的方法。
  20. 一种电子设备，包括：
    处理器；以及
    存储器，用于存储所述处理器的可执行指令；
    其中，所述处理器配置为经由执行所述可执行指令来执行权利要求1-11任一项所述的方法。
PCT/CN2020/110742 2019-09-05 2020-08-24 语音识别方法及装置、神经网络训练方法及装置 WO2021043015A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP20860648.3A EP3926623A4 (en) 2019-09-05 2020-08-24 VOICE RECOGNITION METHOD AND APPARATUS AND NEURON NETWORK LEARNING METHOD AND APPARATUS
JP2021564425A JP7337953B2 (ja) 2019-09-05 2020-08-24 音声認識方法及び装置、ニューラルネットワークの訓練方法及び装置、並びにコンピュータープログラム
US17/476,345 US20220004870A1 (en) 2019-09-05 2021-09-15 Speech recognition method and apparatus, and neural network training method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910838469.5 2019-09-05
CN201910838469.5A CN110600018B (zh) 2019-09-05 2019-09-05 语音识别方法及装置、神经网络训练方法及装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/476,345 Continuation US20220004870A1 (en) 2019-09-05 2021-09-15 Speech recognition method and apparatus, and neural network training method and apparatus

Publications (1)

Publication Number Publication Date
WO2021043015A1 true WO2021043015A1 (zh) 2021-03-11

Family

ID=68857742

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/110742 WO2021043015A1 (zh) 2019-09-05 2020-08-24 语音识别方法及装置、神经网络训练方法及装置

Country Status (5)

Country Link
US (1) US20220004870A1 (zh)
EP (1) EP3926623A4 (zh)
JP (1) JP7337953B2 (zh)
CN (1) CN110600018B (zh)
WO (1) WO2021043015A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113111654A (zh) * 2021-04-09 2021-07-13 杭州电子科技大学 一种基于分词工具共性信息和部分监督学习的分词方法
CN113114400A (zh) * 2021-04-14 2021-07-13 中南大学 基于时序注意力机制和lstm模型的信号频谱空洞感知方法
CN113377331A (zh) * 2021-07-05 2021-09-10 腾讯音乐娱乐科技（深圳）有限公司 一种音频数据处理方法、装置、设备及存储介质
WO2022265210A1 (ko) * 2021-06-18 2022-12-22 삼성전자주식회사 전자 장치 및 전자 장치의 개인화된 음성 처리 방법

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110600018B (zh) * 2019-09-05 2022-04-26 腾讯科技（深圳）有限公司 语音识别方法及装置、神经网络训练方法及装置
CN111243576B (zh) * 2020-01-16 2022-06-03 腾讯科技（深圳）有限公司 语音识别以及模型训练方法、装置、设备和存储介质
CN112750425B (zh) * 2020-01-22 2023-11-03 腾讯科技（深圳）有限公司 语音识别方法、装置、计算机设备及计算机可读存储介质
CN111341341B (zh) * 2020-02-11 2021-08-17 腾讯科技（深圳）有限公司 音频分离网络的训练方法、音频分离方法、装置及介质
CN111370031B (zh) * 2020-02-20 2023-05-05 厦门快商通科技股份有限公司 语音分离方法、系统、移动终端及存储介质
CN113362432B (zh) * 2020-03-04 2024-04-19 Tcl科技集团股份有限公司 一种面部动画生成方法及装置
CN111402867B (zh) * 2020-04-21 2021-01-22 北京字节跳动网络技术有限公司 混合采样率声学模型训练方法、装置及电子设备
CN111639157B (zh) * 2020-05-13 2023-10-20 广州国音智能科技有限公司 音频标记方法、装置、设备及可读存储介质
US11678120B2 (en) * 2020-05-14 2023-06-13 Nvidia Corporation Audio noise determination using one or more neural networks
CN111883181A (zh) * 2020-06-30 2020-11-03 海尔优家智能科技（北京）有限公司 音频检测方法、装置、存储介质及电子装置
CN111951783B (zh) * 2020-08-12 2023-08-18 北京工业大学 一种基于音素滤波的说话人识别方法
CN112331177A (zh) * 2020-11-05 2021-02-05 携程计算机技术（上海）有限公司 基于韵律的语音合成方法、模型训练方法及相关设备
US20220286696A1 (en) * 2021-03-02 2022-09-08 Samsung Electronics Co., Ltd. Image compression method and apparatus
CN113096679A (zh) * 2021-04-02 2021-07-09 北京字节跳动网络技术有限公司 音频数据处理方法和装置
CN113345464A (zh) * 2021-05-31 2021-09-03 平安科技（深圳）有限公司 语音提取方法、系统、设备及存储介质
CN114743554A (zh) * 2022-06-09 2022-07-12 武汉工商学院 基于物联网的智能家居交互方法及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11224098A (ja) * 1998-02-06 1999-08-17 Meidensha Corp 単語音声認識システムにおける環境適応装置
CN103810999A (zh) * 2014-02-27 2014-05-21 清华大学 基于分布式神经网络的语言模型训练方法及其系统
CN108172238A (zh) * 2018-01-06 2018-06-15 广州音书科技有限公司 一种语音识别系统中基于多个卷积神经网络的语音增强算法
CN110097894A (zh) * 2019-05-21 2019-08-06 焦点科技股份有限公司 一种端到端的语音情感识别的方法和系统
CN110189749A (zh) * 2019-06-06 2019-08-30 四川大学 语音关键词自动识别方法
CN110600018A (zh) * 2019-09-05 2019-12-20 腾讯科技（深圳）有限公司 语音识别方法及装置、神经网络训练方法及装置

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102097095A (zh) * 2010-12-28 2011-06-15 天津市亚安科技电子有限公司 一种语音端点检测方法及装置
CN103366759A (zh) * 2012-03-29 2013-10-23 北京中传天籁数字技术有限公司 语音数据的测评方法和装置
US9721562B2 (en) * 2013-12-17 2017-08-01 Google Inc. Generating representations of acoustic sequences
JP6536320B2 (ja) * 2015-09-28 2019-07-03 富士通株式会社 音声信号処理装置、音声信号処理方法及びプログラム
CN105869658B (zh) * 2016-04-01 2019-08-27 金陵科技学院 一种采用非线性特征的语音端点检测方法
CN106782602B (zh) * 2016-12-01 2020-03-17 南京邮电大学 基于深度神经网络的语音情感识别方法
CN109754789B (zh) * 2017-11-07 2021-06-08 北京国双科技有限公司 语音音素的识别方法及装置
CN108877775B (zh) * 2018-06-04 2023-03-31 平安科技（深圳）有限公司 语音数据处理方法、装置、计算机设备及存储介质
CN109545188B (zh) * 2018-12-07 2021-07-09 深圳市友杰智新科技有限公司 一种实时语音端点检测方法及装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11224098A (ja) * 1998-02-06 1999-08-17 Meidensha Corp 単語音声認識システムにおける環境適応装置
CN103810999A (zh) * 2014-02-27 2014-05-21 清华大学 基于分布式神经网络的语言模型训练方法及其系统
CN108172238A (zh) * 2018-01-06 2018-06-15 广州音书科技有限公司 一种语音识别系统中基于多个卷积神经网络的语音增强算法
CN110097894A (zh) * 2019-05-21 2019-08-06 焦点科技股份有限公司 一种端到端的语音情感识别的方法和系统
CN110189749A (zh) * 2019-06-06 2019-08-30 四川大学 语音关键词自动识别方法
CN110600018A (zh) * 2019-09-05 2019-12-20 腾讯科技（深圳）有限公司 语音识别方法及装置、神经网络训练方法及装置

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113111654A (zh) * 2021-04-09 2021-07-13 杭州电子科技大学 一种基于分词工具共性信息和部分监督学习的分词方法
CN113111654B (zh) * 2021-04-09 2022-03-08 杭州电子科技大学 一种基于分词工具共性信息和部分监督学习的分词方法
CN113114400A (zh) * 2021-04-14 2021-07-13 中南大学 基于时序注意力机制和lstm模型的信号频谱空洞感知方法
CN113114400B (zh) * 2021-04-14 2022-01-28 中南大学 基于时序注意力机制和lstm模型的信号频谱空洞感知方法
WO2022265210A1 (ko) * 2021-06-18 2022-12-22 삼성전자주식회사 전자 장치 및 전자 장치의 개인화된 음성 처리 방법
CN113377331A (zh) * 2021-07-05 2021-09-10 腾讯音乐娱乐科技（深圳）有限公司 一种音频数据处理方法、装置、设备及存储介质
CN113377331B (zh) * 2021-07-05 2023-04-07 腾讯音乐娱乐科技（深圳）有限公司 一种音频数据处理方法、装置、设备及存储介质

Also Published As

Publication number Publication date
JP2022531574A (ja) 2022-07-07
EP3926623A4 (en) 2022-10-19
CN110600018A (zh) 2019-12-20
JP7337953B2 (ja) 2023-09-04
EP3926623A1 (en) 2021-12-22
US20220004870A1 (en) 2022-01-06
CN110600018B (zh) 2022-04-26

Similar Documents

Publication Publication Date Title
WO2021043015A1 (zh) 语音识别方法及装置、神经网络训练方法及装置
WO2021143327A1 (zh) 语音识别方法、装置和计算机可读存储介质
CN110600017B (zh) 语音处理模型的训练方法、语音识别方法、系统及装置
Xu et al. Convolutional gated recurrent neural network incorporating spatial features for audio tagging
WO2021042870A1 (zh) 语音处理的方法、装置、电子设备及计算机可读存储介质
CN111243620B (zh) 语音分离模型训练方法、装置、存储介质和计算机设备
CN108962237A (zh) 混合语音识别方法、装置及计算机可读存储介质
CN109841226A (zh) 一种基于卷积递归神经网络的单通道实时降噪方法
CN111627458B (zh) 一种声源分离方法及设备
CN111243576A (zh) 语音识别以及模型训练方法、装置、设备和存储介质
JP2019528476A (ja) 音声認識方法及び装置
Bae et al. End-to-End Speech Command Recognition with Capsule Network.
CN111899757B (zh) 针对目标说话人提取的单通道语音分离方法及系统
CN110060657B (zh) 基于sn的多对多说话人转换方法
CN112767959B (zh) 语音增强方法、装置、设备及介质
CN114974280A (zh) 音频降噪模型的训练方法、音频降噪的方法及装置
CN115376495A (zh) 语音识别模型训练方法、语音识别方法及装置
JP7164098B2 (ja) 音声を認識する方法及び装置
CN113555032A (zh) 多说话人场景识别及网络训练方法、装置
Cornell et al. Implicit acoustic echo cancellation for keyword spotting and device-directed speech detection
CN116863920A (zh) 基于双流自监督网络的语音识别方法、装置、设备及介质
CN116434758A (zh) 声纹识别模型训练方法、装置、电子设备及存储介质
WO2022213825A1 (zh) 基于神经网络的端到端语音增强方法、装置
CN113707172B (zh) 稀疏正交网络的单通道语音分离方法、系统、计算机设备
CN112951270B (zh) 语音流利度检测的方法、装置和电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20860648

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020860648

Country of ref document: EP

Effective date: 20210915

ENP Entry into the national phase

Ref document number: 2021564425

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE