WO2023005729A1

WO2023005729A1 - Speech information processing method and apparatus, and electronic device

Info

Publication number: WO2023005729A1
Application number: PCT/CN2022/106426
Authority: WO
Inventors: 朱耀明; 董倩倩; 王明轩; 李磊
Original assignee: 北京有竹居网络技术有限公司
Priority date: 2021-07-28
Filing date: 2022-07-19
Publication date: 2023-02-02
Also published as: CN113571044A

Abstract

A speech information processing method. The method comprises: acquiring first acoustic feature information of at least one frame of speech information to be translated (101); under streaming speech recognition, determining whether the first acoustic feature information corresponds to complete semantics (102); and in response to a determination result being yes, performing a translation operation on the first acoustic feature information, so as to obtain a corresponding translation result (103). Therefore, the accuracy of a translation result can be improved, and the output delay of the translation result can be reduced. The present application further relates to a speech information processing apparatus, a speech information processing model, a speech information processing model training method, a speech information processing model training apparatus, an electronic device, and a computer-readable medium.

Description

Speech information processing method, device and electronic equipment

Cross References to Related Applications

This application claims the priority of the Chinese patent application with the application number 202110860672.X and the title of the invention "Voice Information Processing Method, Device and Electronic Equipment" filed on July 28, 2021, the entire content of which is incorporated by reference in In this application.

technical field

The present disclosure relates to the technical field of artificial intelligence, and in particular to a voice information processing method, device and electronic equipment.

Background technique

This section is intended to provide a background or context for implementations of the invention that are recited in the claims. The descriptions herein are not admitted to be prior art by inclusion in this section.

Speech Translation (ST) aims to translate source language speech into target language text, and is widely used in conference speeches, business meetings, cross-border customer service, overseas travel and other scenarios.

Traditional speech translation models usually first use a speech recognition model to convert speech into text in the source language, and then use a machine translation model to translate the recognized text in the source language into the target language.

Contents of the invention

This Disclosure section is provided to introduce a simplified form of concepts that are described in detail that follow in the Detailed Description section. This disclosure part is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

Embodiments of the present disclosure provide a voice information processing method, device and electronic equipment.

In a first aspect, an embodiment of the present disclosure provides a method for processing speech information, including: acquiring first acoustic feature information of at least one frame of speech information to be translated; and determining whether the first acoustic feature information is Corresponding to complete semantics; in response to the determination result being yes, performing a translation operation on the first acoustic feature information to obtain a corresponding translation result.

In a second aspect, an embodiment of the present disclosure provides a speech information processing model, including: an acoustic model, a semantic recognition model, and a translation model, wherein the acoustic model is used to: receive at least one frame in the streaming speech recognition mode The speech information to be translated, and extracting the first acoustic feature information of the at least one frame of the speech information to be translated; the semantic recognition model is used to: receive the at least one frame of the first acoustic feature in the streaming speech recognition mode information, and determine whether the at least one frame of first acoustic feature information corresponds to complete semantics; the translation model is used to determine a translation result of the first acoustic feature information in a streaming speech recognition mode.

In a third aspect, an embodiment of the present disclosure provides a training method for a speech information processing model, which is applied to the speech information processing model described in the second aspect, and the speech information processing model includes an acoustic model, a semantic recognition model, and a translation model , the method includes: obtaining a training sample set, the training sample set includes a plurality of training sample pairs, and the training sample pairs include original speech information in a first language and translation results corresponding to the original speech information in a second language; The original speech information in the sample pair is input to the acoustic model after initial training, and the translation result is used as the output of the translation model to train the speech information processing model to obtain a trained speech information processing model.

In a fourth aspect, an embodiment of the present disclosure provides a voice information processing device, including: an acquisition unit, configured to acquire at least one frame of first acoustic feature information of the voice information to be translated; a determination unit, configured to Next, determine whether the first acoustic feature information of the at least one frame satisfies a preset translation condition; the translation unit is configured to perform a translation operation on the first acoustic feature information in response to the determination result being yes, to obtain a corresponding translation result.

In a fifth aspect, an embodiment of the present disclosure provides a speech information processing model training device, including: an acquisition unit, configured to acquire at least one frame of first acoustic feature information of the speech information to be translated; a determination unit, configured to Under the speech recognition method, determine whether the at least one frame of first acoustic feature information satisfies a preset translation condition; the translation unit is configured to perform a translation operation on the first acoustic feature information in response to the determination result being yes, to obtain corresponding translation results.

In a sixth aspect, an embodiment of the present disclosure provides an electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the speech information processing method as described in the first aspect, or the training method of a speech information processing model as described in the third aspect.

In a seventh aspect, an embodiment of the present disclosure provides a computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, the speech information processing method as described in the first aspect is implemented, or as described in the third aspect The training method of the speech information processing model described above.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

FIG. 1 is a flowchart of an embodiment of a voice information processing method according to the present disclosure;

FIG. 2 is a flow chart of another embodiment of the voice information processing method according to the present disclosure;

Fig. 3 shows a schematic diagram of processing acoustic feature information by the continuous integration and distribution module in the embodiment shown in Fig. 2;

Fig. 4 shows a schematic structural diagram of a speech information processing model according to the present disclosure;

Fig. 5 shows a schematic flowchart of a training method of a speech information processing model according to the present disclosure;

FIG. 6 is a schematic structural diagram of an embodiment of a speech information processing device according to the present disclosure;

FIG. 7 is a schematic structural diagram of an embodiment of a training device for a speech information processing model according to the present disclosure;

FIG. 8 is an exemplary system architecture in which the voice information processing method and the voice information processing device according to an embodiment of the present disclosure can be applied;

Fig. 9 is a schematic diagram of a basic structure of an electronic device provided according to an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method implementations of the present disclosure may be executed in different orders, and/or executed in parallel. Additionally, method embodiments may include additional steps and/or omit performing illustrated steps. The scope of the present disclosure is not limited in this respect.

As used herein, the term "comprise" and its variations are open-ended, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one further embodiment"; the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the sequence of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more" multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are used for illustrative purposes only, and are not used to limit the scope of these messages or information.

Recently, end-to-end translation methods have been applied to both non-streaming speech translation and streaming translation. The inventors found that some solutions for applying the end-to-end translation method to streaming translation are to slice the source audio at a fixed time, and each language slice is regarded as a translation password, and applied to streaming speech translation. However, in real environments, the length of speech is often longer, resulting in end-to-end speech translation either introducing delays or causing translation errors. In order to solve the above-mentioned problems, the present disclosure proposes the following proposals. Please refer to FIG. 1 , which shows the flow of an embodiment of the speech information processing method according to the present disclosure. As shown in Figure 1, the speech information processing method comprises the following steps:

Step 101, acquire first acoustic feature information of at least one frame of speech information to be translated.

An end-to-end speech recognition model that maps audio directly to characters or words. The voice information to be translated may be voice information in the first language. The speech information to be translated may be the currently collected speech information of the speaker, or the pre-stored speech information of the speaker. The first language here may be any language, such as English, Chinese, French, etc. The translation result may correspond to the target language. The target language can be, for example, any other language other than the first language.

Speech information may include sequences of words. Various methods may be used to perform feature extraction on the above speech information to obtain the acoustic features of the speech information. The acoustic features of the speech information here can be extracted from the logarithmic-mel spectrogram of the speech.

In practice, acoustic features can be extracted frame by frame from the speech information to be translated. In some application scenarios, each audio frame may include multiple sampling points for discretizing a continuous audio signal, for example, an audio frame may include 1024 sampling points. Each audio frame may correspond to an acoustic feature sequence. The acoustic feature sequence of an audio frame may include an acoustic feature sequence composed of the acoustic features of each sampling point, and the acoustic features of each sampling point may include amplitude, phase, frequency, and correlation of various dimensions.

The first acoustic feature information of the at least one frame of speech information to be translated may include an acoustic feature sequence corresponding to the at least one frame of speech information to be translated.

Step 102, under streaming speech recognition, determine whether the first acoustic feature information corresponds to complete semantics.

The end-to-end speech translation involved in the present disclosure may include a streaming speech recognition mode and a non-streaming speech recognition mode.

Generally, the non-streaming speech translation mode refers to a translation mode that can listen to all speech audios to be translated at one time, and then generate translated text. The streaming voice translation mode refers to a translation mode that completes translation while receiving voice streams.

If the current speech translation mode is a streaming speech recognition mode, it may be determined whether at least one frame of first acoustic feature information satisfies a preset translation condition.

The preset translation condition here includes that at least one frame of first acoustic feature information corresponds to complete semantics.

If at least one frame of first acoustic feature information is complete semantics, go to step 103 . Otherwise, continue to obtain the acoustic feature sequence of at least one subsequent frame of speech information to be translated, and add the acoustic feature sequence of at least one subsequent frame of translated speech information to the acoustic feature sequence of at least one frame of speech information to be translated to obtain the updated first An acoustic feature information.

That is, when the feature sequence included in at least one frame of the first acoustic feature information corresponds to a complete semantics, the first acoustic feature information is then translated.

In some optional implementation manners, the above step 102 may include:

The above-mentioned first acoustic feature information is input into the pre-trained preset semantic recognition model, and the preset semantic recognition model is used to determine whether the first acoustic feature information corresponds to complete semantics.

The above-mentioned semantic recognition model may be various machine learning models, such as a convolutional neural network model. In some application scenarios, the above machine learning model may be, for example, a Continuous Integrate-and-Fire (CIF) model. The CIF model includes a second encoder. The CIF model can compress the first acoustic feature information obtained in step 101, and the CIF model can judge whether the compressed first acoustic feature information has complete semantics.

Step 103, in response to the determination result being yes, perform a translation operation on the first acoustic feature information to obtain a corresponding translation result.

Here, the translation operation is performed on the first acoustic feature information, and the expression of the above semantics in the target language may be determined according to the semantics corresponding to the first acoustic feature information, so as to obtain a translation result.

The above-mentioned translation result may be a translation result in speech form, or a translation result in text form.

The voice information processing method provided in this embodiment obtains the first acoustic feature information of at least one frame of the voice information to be translated; under streaming speech recognition, determines whether the first acoustic feature information corresponds to complete semantics; responds to the determination result If yes, perform a translation operation on the first acoustic feature information to obtain a corresponding translation result, and realize that under streaming speech recognition, when the first acoustic feature information of at least one frame of speech information to be translated corresponds to complete semantics, The translation realizes that in streaming translation, the voice information to be translated with complete semantics is automatically determined, and the translated voice information with complete semantics is translated. Streaming speech recognition in the related art is a scheme that intercepts speech information according to a fixed length or a fixed number of words, and extracts the feature sequence of the speech information for translation. Related technical solutions Since the speech information is intercepted by a fixed length or a fixed number of words for translation, the speech information to be translated in the source language intercepted by a fixed length or a fixed number of words may not have complete semantics, and the obtained target language translation result may not reflect the quality of the speech information to be translated. The original semantics makes the translation result poor. However, because the solution of this embodiment translates the translated voice information with complete semantics, a more accurate translation result can be obtained and the accuracy of the translation result is improved.

In addition, because this solution can translate the speech information to be translated with complete semantics after it is determined, there is no need to wait until the time period specified by the fixed length is over before translating. Therefore, the output delay of the translation result can be reduced.

In some optional implementations of this embodiment, the above voice information processing method further includes the following steps:

Step 104, under the non-streaming speech recognition, receive the multi-frame voice information to be translated until the input end instruction of the voice information is detected, obtain the second acoustic feature information of the multi-frame voice information to be translated, and analyze the second acoustic feature Perform translation operations on information to obtain corresponding translation results.

In these optional implementation manners, in the non-streaming speech recognition mode, the second acoustic feature information corresponding to all the speech information to be translated may be determined after receiving all the speech information to be translated. The second acoustic feature information may include multiple feature sequences. Multiple feature sequences can form a word vector matrix. Then the word vector matrix is analyzed and processed, and a translation operation is performed on the analyzed and processed word vector matrix to obtain translation results corresponding to all speech information to be translated. The translation result may be a translation result in speech form or a translation result in text form.

In these optional implementation manners, all speech information to be translated is translated in a non-streaming speech recognition mode.

That is to say, in the same set of translation solutions, streaming translation mode and non-streaming translation mode are provided, and in the translation mode selected by the user, the corresponding streaming translation or non-streaming translation is performed, realizing the use of the same set of translation solutions, Both streaming speech translation and non-streaming speech translation are taken into account.

Please refer to FIG. 2 , which shows a flow chart of another embodiment of the voice information processing method according to the present disclosure. As shown in Figure 2, the method includes the following steps:

Step 201: Input at least one frame of speech information to be processed into a pre-trained acoustic model to obtain the first acoustic feature information.

The above-mentioned acoustic model may be various machine learning models, such as a recurrent neural network model and the like. The aforementioned machine learning model may be a pre-trained machine learning model. This machine learning model can convert the input speech information into a sequence of features.

In some application scenarios, the above-mentioned acoustic model may be a masked acoustic model (Masked Acoustic Model, MAM). Shielded acoustic models can include encoders and prediction heads. When training the MAM, the sample audio data can be selected as input, and the sample encoding corresponding to the sample audio data is used as output. During training, MAM can select 15% of the input audio frames to cover them, and the model can predict the covered frames according to the context of the training text. The Prediction Head layer contains two layers of forward network. A preset loss function (such as L1Loss) can be used when training the MAM to minimize the gap between the vector of the 15% masked predicted value frame and the real frame vector.

Acoustic feature information may include, but not limited to, amplitude, phase, frequency, and correlation of each dimension. In addition to information features, the above acoustic feature information also includes feature information of human vocalizations. The above acoustic feature information corresponding to the same utterance by different people may be different.

Step 202: Input the first acoustic feature information into a pre-trained preset semantic recognition model, and use the preset semantic recognition model to determine whether the first acoustic feature information corresponds to complete semantics.

In the streaming translation mode, the first acoustic feature information output in step 201 can be input to the pre-trained preset semantic recognition model. The first acoustic feature information is compressed by a preset semantic recognition model, and it is judged whether the compressed first acoustic feature information corresponds to complete semantics.

The aforementioned preset semantic recognition model may include, for example, a Continuous Integrate-and-Fire (CIF) module. The CIF module can compress and align the first acoustic feature information output in step 201 . When performing compression processing on the first acoustic feature information, the CIF may divide multiple feature values in the first acoustic feature information into two parts, one part is used for the current compression process, and the other part is used for the next compression process.

Please refer to FIG. 3 , which shows a schematic diagram of processing acoustic feature information by the CIF module. The acoustic feature information includes an acoustic feature sequence and a weight corresponding to each feature vector in the acoustic feature sequence. The weight can represent the amount of information contained in the feature vector. As shown in Figure 3, the acoustic feature sequence can be [h ₁ , h ₂ , h ₃ , h ₄ , h ₅ , ...]; the corresponding weight sequence can be [α _1, α ₂ , α ₃ , α ₄ , α ₅ ...]. Multiple feature vectors can be divided into two parts, and the first part is used to calculate this compression process. Each compression process can integrate more than two feature vectors into a new feature vector. For example, the multiple feature vectors of the compression process are arranged according to the sequence of the vectors. As shown in FIG. 3 , the weight α ₂ of the feature vector h ₂ is split into α ₂₁ and α ₂₂ . α ₄ is split into α ₄₁ and α ₄₂ . During each compression, the sum of the weights (including weight components) corresponding to each feature vector is 1 during this compression. For example, when the sum of the weight α1 _of the feature vector _h1 and the weight component _α21 of the feature vector h2 is ₁ , it can be determined that the object of this compression is the component of the feature vector _h1 and the feature vector _h2 . As shown in Figure 3, the results of two compressions are as follows: c ₁ =α ₁ ×h ₁ +α ₂₁ ×h ₂ ; c ₂ =α ₂₂ ×h ₂ +α ₃ ×h ₃ +α ₄₁ ×h ₄ . The sum of α ₂₂ , α ₃ , and α ₄₁ is 1. When the sum of the weights of multiple feature vectors (including the weight components corresponding to the feature components) is 1, it can be considered that these feature vectors have complete semantics. These eigenvectors can be integrated.

Through the above process, the CIF module can determine whether the first acoustic feature information corresponds to complete semantics, and compress the first acoustic feature information corresponding to complete semantics.

Step 203, in response to the determination result being yes, input the acoustic feature information into the pre-trained translation model to obtain a translation result corresponding to the acoustic feature information.

The translation model mentioned above may be, for example, various neural network models, such as hidden Markov models. Preferably, the above-mentioned translation model may be a transformer model. The transformer model consists of an encoder and a decoder, receiving the word vector matrix output by the CIF module and completing the translation. The process of translating the word vector matrix output by the CIF module into the translation result of the target language by the above transformer model can be the same as the process of translating the word vector matrix by the existing transformer model, and will not be described here.

In addition, in some optional implementations, if under non-streaming speech recognition, the above step 201 may include receiving multiple frames of speech information to be translated until an input end instruction of the speech information is detected, and waiting for the received multi-frames to be translated. The translated speech information is input to the pre-trained acoustic model to obtain second acoustic feature information of multiple frames of speech information to be translated. Furthermore, the above speech information processing method further includes step 204, inputting the second acoustic feature information into the preset semantic recognition model, compressing and aligning the second acoustic feature information, and obtaining the compressed second acoustic feature information. Step 205, inputting the compressed second acoustic feature information into the pre-trained translation model to obtain a translation result corresponding to the second acoustic feature information.

The voice information processing method provided in this embodiment can improve the end-to-end voice translation by using the preset semantic recognition model to determine the first acoustic feature information with complete semantics, and using the translation model to obtain the translation result of the first acoustic feature information. speed and accuracy.

In addition, the speech information processing method provided in this embodiment completes streaming speech translation and non-streaming speech translation.

Please refer to FIG. 4 , which shows a speech information processing model according to the present disclosure. As shown in FIG. 4 , the speech information processing model includes: an acoustic model 401 , a semantic recognition model 402 and a translation model 403 .

The above speech information processing model can provide translation mode options. The translation mode selections include streaming speech recognition mode and non-streaming speech recognition mode. When using the above-mentioned speech information processing model to perform end-to-end speech information translation, the user can select a translation mode. According to the user's selection of the translation mode option, the speech information processing model works in a streaming speech recognition mode or a non-streaming speech recognition mode. After the translation mode option is selected, the speech information in the source language to be translated can be input into the speech information processing model, so that the speech information in the source language can be translated by the speech information processing model.

In the streaming speech recognition mode, the acoustic model 401 is configured to receive at least one frame of speech information to be translated, and extract first acoustic feature information of the at least one frame of speech information to be translated.

The semantic recognition model 402 is configured to receive the at least one frame of first acoustic feature information, and determine whether the at least one frame of first acoustic feature information corresponds to complete semantics.

The semantic recognition model may include a Continuous Integrate-and-Fire (CIF) module. The CIF module can perform semantic recognition on the first acoustic feature information, and compress and align the first acoustic feature information. The functions completed by the CIF module can refer to the process shown in FIG. 3 .

The translation model 403 is configured to determine that the first acoustic feature information corresponds to complete semantics, and determine a translation result corresponding to the first acoustic feature information. In the non-streaming mode, a translation result corresponding to the second acoustic feature information is determined.

In addition, if the voice information processing model works in the non-streaming voice recognition mode, the acoustic model 401 is used to: receive multiple frames of voice information to be translated, and extract multiple frames of voice information to be translated until the input end instruction of the voice information is detected The second acoustic characteristic information of the information. The semantic recognition model 402 is used for: compressing and aligning the second acoustic feature information; the translation model 403 is used for: determining the translation result of the second acoustic feature information.

For the functions performed by each model included in the speech information processing model in the speech information processing method, reference may be made to the description of the embodiment shown in FIG. 2 , which will not be repeated here.

Please refer to FIG. 5 , which shows the training method of the speech information processing model provided by the present disclosure. Speech information processing models include acoustic models, semantic recognition models and translation models. As shown in Figure 5, the method includes the following steps.

Step 501: Acquire a training sample set, the training sample set includes a plurality of training sample pairs, and the training sample pairs include original speech information in a first language and sample translation results corresponding to the original speech information in a second language.

Step 502: Input the original speech information in the training sample pair into the acoustic model after the initial training, use the translation result as the output of the translation model, train the speech information processing model, and obtain the trained speech information Handle the model.

In this embodiment, the aforementioned acoustic model may be a pre-trained model. The above-mentioned acoustic model may be a recurrent neural network model or the like. Preferably, the above-mentioned acoustic model may be a shielded acoustic model.

In the above training process, the speech information processing model may be trained using the second loss function and the third loss function. The second loss function here may be a quality loss function, and the third loss function may be a cross-entropy loss function. In some application scenarios, the above-mentioned triggering of the end of the training can be minimized by the sum of the above-mentioned second loss function and the third loss function. In some other application scenarios, the triggering of the end of the above training may be that the above training times reach the preset number of times required.

In some optional implementation manners, the above step 502 includes the following substeps:

First, obtain the sample code corresponding to the sample translation result.

Secondly, the original speech information is input into the acoustic model after initial training, and the semantic recognition model is trained by using the sample code of the sample translation result and the first loss function to obtain the trained semantic recognition model.

The first loss function here may be a quality loss function.

The above semantic recognition model may include a Continuous Integrate-and-Fire (CIF) module.

In these optional implementations, the semantic recognition model can be trained first. Then train the speech information processing model as a whole. The overall training times of the speech information processing model can be reduced.

The speech information processing model obtained through the above training can work in either the streaming speech recognition mode or the non-streaming speech recognition mode.

Further referring to FIG. 6 , as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of speech information processing devices, which correspond to the method embodiments shown in FIG. 1 , and can be specifically applied to in various electronic devices.

As shown in FIG. 6 , the speech information processing apparatus of this embodiment includes: an acquisition unit 601 , a determination unit 602 , and a translation unit 603 . Wherein, the acquiring unit 601 is configured to acquire at least one frame of the first acoustic feature information of the speech information to be translated; the determining unit 602 is configured to determine whether the at least one frame of the first acoustic feature information is The preset translation condition is satisfied; the translation unit 603 is configured to perform a translation operation on the first acoustic feature information to obtain a corresponding translation result in response to the determination result being yes.

In this embodiment, the specific processing of the acquisition unit 601, the determination unit 602, and the translation unit 603 of the speech information processing device and the technical effects brought about by them can refer to step 101, step 102, and step 103 in the corresponding embodiment of FIG. 1, respectively. Relevant descriptions will not be repeated here.

In some optional implementation manners, the obtaining unit 601 is further configured to: input at least one frame of speech information to be processed into a pre-trained acoustic model to obtain the first acoustic feature information.

In some optional implementations, the acoustic model includes a shielded acoustic model.

In some optional implementation manners, the determining unit 602 is further configured to: input the first acoustic feature information into a pre-trained preset semantic recognition model, and use the preset semantic recognition model to determine the first acoustic feature information. Whether the feature information corresponds to the complete semantics.

In some optional implementation manners, the preset semantic recognition model includes a continuous integration and distribution module.

In some optional implementation manners, the speech information processing device further includes a non-streaming speech information processing unit (not shown in the figure), and the non-streaming speech information processing unit is used for: under non-streaming speech recognition, receiving The multiple frames of voice information to be translated until the input end command of the voice information is detected, the second acoustic feature information of the multiple frames of voice information to be translated is obtained, and the translation operation is performed on the second acoustic feature information to obtain a corresponding translation result.

In some optional implementation manners, the translating operation includes: inputting acoustic feature information into a pre-trained translation model to obtain a translation result corresponding to the acoustic feature information.

Further referring to FIG. 7 , as an implementation of the method shown in FIG. 5 above, the present disclosure provides an embodiment of a speech information processing model training device, which corresponds to the method embodiment shown in FIG. 5 , The device can be specifically applied to various electronic devices.

As shown in FIG. 7 , the apparatus for training a speech information processing model in this embodiment includes: a sample acquisition unit 701 and a training unit 702 . Wherein, the sample obtaining unit 701 is used to obtain a training sample set, the training sample set includes a plurality of training sample pairs, and the training sample pairs include the translation corresponding to the original speech information in the first language and the original speech information in the second language Result; the training unit 702 is used for inputting the original speech information in the training sample pair to the acoustic model after the initial training, using the translation result as the output of the translation model, and training the speech information processing model to obtain The trained speech information processing model.

In this embodiment, the specific processing of the sample acquisition unit 701 and the training unit 702 of the speech information processing model training device and the technical effects brought about by them can refer to the relevant descriptions of step 501 and step 502 in the embodiment corresponding to FIG. 5 , which will not be repeated here.

In some optional implementations, the above training unit 702 also includes a first training subunit (not shown in the figure), the first training subunit is used to: obtain the sample code corresponding to the translation result; Speech information is input into the acoustic model after initial training, and the semantic recognition model is trained by using the sample code of the translation result and the first loss function to obtain the trained semantic recognition model.

In some optional implementation manners, the training unit 702 is further configured to: use the second loss function and the third loss function to train the speech information processing model to obtain a trained speech information processing model.

The speech information processing method, device and electronic equipment provided by the embodiments of the present disclosure obtain the first acoustic feature information of at least one frame of speech information to be translated; under streaming speech recognition, determine whether the first acoustic feature information is complete Semantics; in response to the determination result being yes, the translation operation is performed on the first acoustic feature information to obtain the corresponding translation result, which realizes the automatic determination of the voice information to be translated with complete semantics in streaming translation, and for the voice information with complete Compared with translating the audio information intercepted at a fixed time, more accurate translation results can be obtained by translating the semantically translated voice information. Improved accuracy of translation results. In addition, because this solution can translate the voice information to be translated after it is determined to have complete semantics, it does not need to wait until the time period specified by the fixed slice time ends before translating, so the output delay of the translation result can be reduced.

Please refer to FIG. 8 , which shows an exemplary system architecture in which the voice information processing method or the voice information processing apparatus according to an embodiment of the present disclosure can be applied.

As shown in FIG. 8 , the system architecture may include

terminal devices

801 , 802 , and 803 , a network 804 , and a server 805 . The network 804 is used as a medium for providing communication links between the

terminal devices

801 , 802 , 803 and the server 805 . Network 804 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.

The

terminal devices

801, 802, 803 can interact with the server 805 through the network 804 to receive or send messages and the like. Various client applications, such as voice information collection applications, may be installed on the

terminal devices

801, 802, and 803. The client applications in the

terminal devices

801, 802, and 803 can receive user instructions and complete corresponding functions according to the user instructions, such as sending the collected voice information to the server.

Terminal devices

801, 802, and 803 may be hardware or software. When the

terminal devices

801, 802, and 803 are hardware, they may be various electronic devices that have display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, moving picture expert compression standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, moving picture expert compression standard audio layer 4) player, laptop portable computer and desktop computer, etc. When the

terminal devices

801, 802, and 803 are software, they can be installed in the electronic devices listed above. It can be implemented as a plurality of software or software modules (such as software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made here.

The server 805 may be a server that provides various services, such as analyzing voice information sent by the

terminal devices

801, 802, and 803 to obtain analysis results (translation results). And send the translation results to the

terminal devices

801, 802, 803.

It should be noted that the voice information processing method provided by the embodiment of the present disclosure may be executed by a server, and correspondingly, the voice information processing apparatus may be set in the server 805 . In addition, the voice information processing method can also be executed by a terminal device, and correspondingly, the voice information processing apparatus can be set in the

terminal devices

801 , 802 , and 803 .

It should be understood that the numbers of terminal devices, networks and servers in FIG. 8 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

Referring now to FIG. 9 , it shows a schematic structural diagram of an electronic device (such as the server or terminal device in FIG. 8 ) suitable for implementing the embodiments of the present disclosure. The terminal equipment in the embodiment of the present disclosure may include but not limited to such as mobile phone, notebook computer, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), vehicle terminal (such as mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers and the like. The electronic device shown in FIG. 9 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

As shown in FIG. 9, an electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 901, which may be loaded into a random access memory according to a program stored in a read-only memory (ROM) 902 or from a storage device 908. (RAM) 903 to execute various appropriate actions and processing. In the RAM 903, various programs and data necessary for the operation of the electronic device 900 are also stored. The processing device 901, ROM 902, and RAM 903 are connected to each other through a bus 904. An input/output (I/O) interface 905 is also connected to the bus 904 .

Typically, the following devices can be connected to the I/O interface 905: input devices 906 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 907 such as a computer; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication means 909 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. While FIG. 9 shows an electronic device having various means, it is to be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 909, or from storage means 908, or from ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

It should be noted that the above-mentioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server can communicate using any currently known or future network protocols such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and can communicate with digital data in any form or medium The communication (eg, communication network) interconnections. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: acquires the first acoustic feature information of at least one frame of speech information to be translated; In the voice recognition mode, it is determined whether the first acoustic feature information corresponds to complete semantics; in response to the determination result being yes, a translation operation is performed on the first acoustic feature information to obtain a corresponding translation result. or

Obtain a training sample set, the training sample set includes a plurality of training sample pairs, the training sample pairs include the original speech information of the first language and the translation result corresponding to the original speech information of the second language; the original training sample pairs Speech information is input into the acoustic model after initial training, and the translation result is used as an output of the translation model to train the speech information processing model to obtain a trained speech information processing model.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (such as through an Internet Service Provider). Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of a unit does not constitute a limitation of the unit itself under certain circumstances.

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principles. Those skilled in the art should understand that the disclosure scope involved in this disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but also covers the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with technical features disclosed in this disclosure (but not limited to) having similar functions.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

A voice information processing method, comprising:

Acquire first acoustic feature information of at least one frame of speech information to be translated;

Under streaming speech recognition, determine whether the first acoustic feature information corresponds to complete semantics;

In response to the determination result being yes, a translation operation is performed on the first acoustic feature information to obtain a corresponding translation result.
The method according to claim 1, wherein said obtaining the first acoustic feature information of at least one frame of speech information to be translated comprises:

Inputting at least one frame of speech information to be processed into a pre-trained acoustic model to obtain the first acoustic feature information.
The method according to claim 2, wherein the acoustic model comprises: a shielded acoustic model.
The method according to claim 1 or 2, wherein, under streaming speech recognition, determining whether the first acoustic feature information corresponds to complete semantics comprises:

The first acoustic feature information is input into a pre-trained preset semantic recognition model, and the preset semantic recognition model is used to determine whether the first acoustic feature information corresponds to complete semantics.
The method according to claim 4, wherein the preset semantic recognition model includes a continuous integration and delivery module.
The method according to claim 1, wherein the method further comprises:

Under non-streaming speech recognition, receiving multiple frames of speech information to be translated until the input end instruction of the speech information is detected, obtaining second acoustic feature information of multiple frames of speech information to be translated, and performing translation on the second acoustic feature information Operation to get the corresponding translation result.
The method according to claim 1 or 6, wherein the translating operation comprises:

The acoustic feature information is input into the pre-trained translation model to obtain a translation result corresponding to the acoustic feature information.
A speech information processing model, comprising: an acoustic model, a semantic recognition model and a translation model, wherein,

The acoustic model is used to: receive at least one frame of speech information to be translated in the streaming speech recognition mode, and extract the first acoustic feature information of the at least one frame of speech information to be translated;

The semantic recognition model is configured to: receive the at least one frame of first acoustic feature information in a streaming speech recognition mode, and determine whether the at least one frame of first acoustic feature information corresponds to complete semantics;

The translation model is used to determine a translation result of the first acoustic feature information in a streaming speech recognition mode.
The model according to claim 8, wherein the acoustic model is further used for: in the non-streaming speech recognition mode, receiving multiple frames of speech information to be translated, until the input end instruction of the speech information is detected, and extracting multiple frames of speech information to be translated Translating the second acoustic feature information of the speech information;

The semantic recognition model is further used for: compressing and aligning the second acoustic feature information in the non-streaming speech recognition mode;

The translation model is further used for: determining a translation result of the second acoustic feature information in a non-streaming speech recognition mode.
A training method of a speech information processing model, applied to the speech information processing model described in claim 8 or 9, said speech information processing model comprising an acoustic model, a semantic recognition model and a translation model, said method comprising:

Obtain a training sample set, the training sample set includes a plurality of training sample pairs, and the training sample pairs include original speech information in a first language and translation results corresponding to the original speech information in a second language;

The original speech information in the training sample pair is input into the acoustic model after initial training, the translation result is used as the output of the translation model, and the speech information processing model is trained to obtain a trained speech information processing model.
The method according to claim 10, wherein the original speech information in the training sample pair is input to the acoustic model after initial training, the translation result is used as the output of the translation model, and the speech information is processed The model is trained, and the obtained speech information processing model after training includes:

Acquiring a sample code corresponding to the translation result;

The original speech information is input into the acoustic model after initial training, and the semantic recognition model is trained by using the sample code of the translation result and the first loss function to obtain the trained semantic recognition model.
The method according to claim 10, wherein the original speech information in the training sample pair is input to the acoustic model after initial training, the translation result is used as the output of the translation model, and the speech information is processed The model is trained, and the obtained speech information processing model after training includes:

The speech information processing model is trained by using the second loss function and the third loss function to obtain a trained speech information processing model.
A voice information processing device, comprising:

An acquisition unit, configured to acquire at least one frame of first acoustic feature information of the speech information to be translated;

A determination unit, configured to determine whether the at least one frame of first acoustic feature information satisfies a preset translation condition under streaming speech recognition;

The translation unit is configured to perform a translation operation on the first acoustic feature information to obtain a corresponding translation result in response to the determination result being yes.
A training device for a speech information processing model, applied to the speech information processing model described in claim 8 or 9, said speech information processing model acoustic model, semantic recognition model and translation model, said device comprising:

A sample acquisition unit, configured to acquire a training sample set, the training sample set includes a plurality of training sample pairs, and the training sample pairs include original speech information in a first language and translation results corresponding to the original speech information in a second language;

The training unit is used to input the original speech information in the training sample pair into the acoustic model after the initial training, and use the translation result as the output of the translation model to train the speech information processing model, and obtain the trained Speech information processing model.
An electronic device, characterized in that it comprises:

one or more processors;

storage means for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-7, or the method according to claims 10-12 any one of the methods described.
A computer-readable medium on which a computer program is stored, wherein the program is executed by a processor to implement the method according to any one of claims 1-7, or any one of claims 10-12 method described in the item.