CN113689867B - A training method, device, electronic device and medium for a speech conversion model - Google Patents

Abstract

The present disclosure provides a training method, device, electronic device and medium for a speech conversion model, which relate to the technical field of artificial intelligence, and in particular, to speech and deep learning technologies. The specific implementation scheme is: input the original acoustic features of the speech into the pre-training model to obtain the latent features output by the pre-training model; input the hidden features into the speech conversion model to obtain the predicted acoustic features output by the speech conversion model; The acoustic features and the predicted acoustic features are used to train the speech conversion model to be trained. In the embodiment of the present application, the latent feature can be used as the input of the speech conversion model to predict the target acoustic feature, so that the model can learn more fully and has a wide range of application scenarios.

Description

A training method, device, electronic device and medium for a speech conversion model

technical field

The present disclosure relates to the technical field of artificial intelligence, and further relates to speech and deep learning technologies, in particular to a training method, apparatus, electronic device and medium for a speech conversion model.

Background technique

Voice conversion has become more and more concerned in the market. The purpose of voice conversion is to convert the voice of the source speaker into the timbre of the target speaker, and keep the expression content of the voice unchanged. According to the corpus required by the model, speech conversion can be divided into parallel corpus speech conversion and non-parallel corpus speech conversion; among them, parallel corpus speech conversion requires the source speaker and the target speaker to record the same text when recording the required corpus. audio. For non-parallel corpus speech conversion, several voices of the target speaker need to be recorded, and the voice of the source speaker is not required for training. The usual methods include the method based on the phoneme probability map and the method of self-reconstruction.

The method based on the phoneme probability map first extracts a ppg feature expressing the speech content from the audio of the target speaker through the speech recognition model, and then uses the model to model the connection between the ppg feature and the audio mel feature. During testing, the source speaker extracts ppg features through the speech recognition model, and inputs the trained transformation model to obtain the transformed features. Based on the self-reconstruction method, the general idea is to decouple the content information and timbre information in the feature through the encoder during training, and then restore the information through the decoder to perform self-reconstruction training.

Human pronunciation is composed of many speech frames. Because human pronunciation has inherent continuity, there should be correlations between adjacent speech frames. Since the existing ppg features or the mel feature frames input by the model are independent of each other, the information in them is independent of each other, and it is difficult for the neural network model to learn the correlation between frames. The learning ability of the model is limited.

SUMMARY OF THE INVENTION

The present disclosure provides a method, apparatus, electronic device and medium for training a speech conversion model.

In a first aspect, the present application provides a method for training a speech conversion model, the method comprising:

Input the original acoustic features of the speech to the pre-training model to obtain the hidden features output by the pre-training model;

Inputting the latent features to a speech conversion model to obtain the predicted acoustic features output by the speech conversion model;

The speech conversion model to be trained is trained based on the original acoustic features and the predicted acoustic features.

In a second aspect, the present application provides a training device for a voice conversion model, the device includes: a pre-training module, a voice conversion module and a training module; wherein,

The pre-training module is used to input the original acoustic features of the speech into the pre-training model to obtain the latent features output by the pre-training model;

The speech conversion module is used for inputting the latent features into the speech conversion model to obtain the predicted acoustic features output by the speech conversion model;

The training module is configured to train the speech conversion model to be trained based on the original acoustic features and the predicted acoustic features.

In a third aspect, an embodiment of the present application provides an electronic device, including:

one or more processors;

memory for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the method for training a speech conversion model described in any embodiment of the present application.

In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method for training a speech conversion model described in any embodiment of the present application.

In a fifth aspect, a computer program product is provided, which, when the computer program product is executed by a computer device, implements the method for training a speech conversion model described in any embodiment of the present application.

The technology according to the present application solves the problem in the prior art that because the ppg feature or the mel feature frame input by the model is independent of each other, the information in it is independent of each other, and it is difficult for the neural network model to learn from it. The technical problem of limiting the learning ability of the model, the technical solution provided in this application can use acoustic features to train a pre-trained self-supervised model, and then extract the frame-level hidden features as new acoustic features, such as The hidden features contain the interrelated information of the acoustic features, and the hidden features are used as the input of the conversion model to predict the target acoustic features, so that the model can learn more fully and has a wide range of application scenarios.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present disclosure. in:

Fig. 1 is the first schematic flow chart of the training method of the speech conversion model provided by the embodiment of the present application;

2 is a second schematic flowchart of a training method of a speech conversion model provided by an embodiment of the present application;

3 is a schematic structural diagram of a training system for a speech conversion model provided by an embodiment of the present application;

4 is a third schematic flowchart of a training method for a speech conversion model provided by an embodiment of the present application;

5 is a schematic structural diagram of a prediction system for a speech conversion model provided by an embodiment of the present application;

6 is a schematic structural diagram of a training device for a speech conversion model provided by an embodiment of the present application;

FIG. 7 is a block diagram of an electronic device used to implement the training method of the speech conversion model according to the embodiment of the present application.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

Example 1

FIG. 1 is a first schematic flowchart of a training method for a speech conversion model provided by an embodiment of the present application. The method may be performed by a training device or electronic device for a speech conversion model, and the device or electronic device may be performed by software and/or hardware. The device or electronic device can be integrated into any smart device with network communication function. As shown in Figure 1, the training method of the speech conversion model may include the following steps:

S101. Input the original acoustic features of speech into a pre-training model to obtain latent features output by the pre-training model.

In this step, the electronic device may input the original acoustic features of the speech into the pre-training model to obtain the latent features output by the pre-training model. Specifically, when the speech conversion model to be trained does not meet the preset convergence conditions, the electronic device can input the original acoustic features of the speech into the pre-training model to obtain the latent features output by the pre-training model. Further, the electronic device can input the original acoustic features into the neural network model to obtain a feature sequence output by the neural network model; and use the feature sequence output by the neural network model as the latent feature output by the pre-training model. Further, the electronic device may firstly divide the original acoustic features into N acoustic feature units; wherein, N is a natural number greater than 1; and then mask one or more acoustic features in the N acoustic feature units to obtain a masked Then input the masked acoustic feature unit into the neural network model to obtain the feature sequence output by the neural network model.

S102. Input the latent features into the speech conversion model to obtain the predicted acoustic features output by the speech conversion model.

In this step, the electronic device may input the latent features into the speech conversion model to obtain the predicted acoustic features output by the speech conversion model. Specifically, the electronic device can input latent features into any model with speech conversion function. For example, the speech conversion model may include: a content encoder, a tone encoder, and a decoder.

S103 , train the speech conversion model to be trained based on the original acoustic feature and the predicted acoustic feature.

In this step, the electronic device may train the speech conversion model to be trained based on the original acoustic features and the predicted acoustic features. Next, the electronic device may re-select a speech to be trained speech conversion model for training, until the to-be-trained speech conversion model satisfies a preset convergence condition. The voice to be reselected here may be a voice adjacent to the previous voice, or may be a voice not adjacent to the previous voice, which is not limited herein. Further, the electronic device can calculate the loss value of the speech conversion model to be trained for speech based on the original acoustic feature and the predicted acoustic characteristic; The model parameters are adjusted.

In the training method of the speech conversion model proposed in the embodiment of the present application, the original acoustic features of speech are firstly input into the pre-training model to obtain the latent features output by the pre-training model; then the hidden features are input into the speech conversion model to obtain the speech conversion model The output predicted acoustic features; the speech conversion model to be trained is then trained based on the original acoustic features and the predicted acoustic features. That is to say, the present application extracts interrelated information from the original acoustic features through the pre-training model when training the speech conversion model. In the existing training methods of speech conversion models, because the ppg features or the mel feature frames input by the model are independent of each other, the information in them is independent of each other, and it is difficult for the neural network model to learn from them. The interrelatedness between them limits the learning ability of the model. Because the present application adopts the technical means of extracting the associated information through the pre-training model, it overcomes the fact that the mel feature frames input by the ppg feature or the model are independent of each other in the prior art, thereby causing the information in them to be independent of each other and the neural network. It is difficult for the network model to learn the correlation between frames and the technical problem that the learning ability of the model is limited. The technical solution provided in this application can use acoustic features to train a pre-trained self-supervised model, and then extract the frame level. The latent features of , as new acoustic features, such hidden features contain the interrelated information of the acoustic features, and the hidden features are used as the input of the speech conversion model to predict the target acoustic features, so that the model can learn more fully and has a wide range of application scenarios; and, The technical solutions of the embodiments of the present application are simple and convenient to implement, easy to popularize, and have wider application range.

Embodiment 2

FIG. 2 is a second schematic flowchart of a training method for a speech conversion model provided by an embodiment of the present application. Based on the above technical solutions, it is further optimized and expanded, and can be combined with the above-mentioned optional embodiments. As shown in Figure 2, the training method of the speech conversion model may include the following steps:

S201. Input the original acoustic feature into the neural network model to obtain a feature sequence output by the neural network model; use the feature sequence output by the neural network model as the latent feature output by the pre-training model.

In this step, the electronic device may input the original acoustic features into the neural network model to obtain a feature sequence output by the neural network model; and use the feature sequence output by the neural network model as a latent feature output by the pre-training model. Specifically, the electronic device may first divide the original acoustic feature into N acoustic feature units; where N is a natural number greater than 1; and then mask one or more acoustic features in the N acoustic feature units to obtain a masked Then input the masked acoustic feature unit into the neural network model to obtain the feature sequence output by the neural network model. For example, the original acoustic features can be divided into: 11 acoustic feature units A, B, C, D, E, F, G, H, I, J, K; then four C, F, G, I The acoustic feature unit is masked to obtain the masked acoustic feature unit; then the masked acoustic feature unit is input into the neural network model to obtain the feature sequence output by the neural network model, the feature sequence may include A, B, C, D, 11 acoustic feature units E, F, G, H, I, J, K.

S202. Input the latent features into the speech conversion model to obtain the predicted acoustic features output by the speech conversion model.

In this step, the electronic device may input the latent features into the speech conversion model to obtain the predicted acoustic features output by the speech conversion model. Specifically, the electronic device can input the original acoustic features into the neural network model to obtain the feature sequence output by the neural network model; and use the feature sequence output by the neural network model as the latent feature output by the pre-training model.

S203. Calculate the loss value of the speech conversion model to be trained for speech based on the original acoustic feature and the predicted acoustic feature.

In this step, the electronic device may calculate the loss value of the speech conversion model to be trained for speech based on the original acoustic features and the predicted acoustic features. Specifically, the electronic device may input the original acoustic features and the predicted acoustic features into a pre-built loss function, and obtain a loss value of the speech conversion model to be trained for speech through the loss function.

S204: Adjust the model parameters in the speech conversion model to be trained according to the speech loss value of the speech conversion model to be trained.

In this step, the electronic device may adjust the model parameters in the speech conversion model to be trained according to the speech loss value of the speech conversion model to be trained. Specifically, the speech conversion system to be trained may be a neural network, and the neural network may include: a convolution layer, a pooling layer, and a fully connected layer. The model parameters in each layer are adjusted.

FIG. 3 is a schematic structural diagram of a training system for a speech conversion model provided by an embodiment of the present application. As shown in Figure 3, the training system of the speech conversion model may include: a pre-training model and a speech conversion model; first, input the original acoustic features of the speech into the pre-training model to obtain the latent features output by the pre-training model; Input to the speech conversion model to obtain the predicted acoustic features output by the speech conversion model; and then train the speech conversion model to be trained based on the original acoustic features and the predicted acoustic features.

In the training method of the speech conversion model proposed in the embodiment of the present application, the original acoustic features of speech are firstly input into the pre-training model to obtain the latent features output by the pre-training model; then the hidden features are input into the speech conversion model to obtain the speech conversion model The output predicted acoustic features; the speech conversion model to be trained is then trained based on the original acoustic features and the predicted acoustic features. That is to say, the present application extracts interrelated information from the original acoustic features through the pre-training model when training the speech conversion model. In the existing training methods of speech conversion models, because the ppg features or the mel feature frames input by the model are independent of each other, the information in them is independent of each other, and it is difficult for the neural network model to learn from them. The interrelatedness between them limits the learning ability of the model. Because the present application adopts the technical means of extracting the associated information through the pre-training model, it overcomes the fact that the mel feature frames input by the ppg feature or the model are independent of each other in the prior art, thereby causing the information in them to be independent of each other and the neural network. It is difficult for the network model to learn the correlation between frames and the technical problem that the learning ability of the model is limited. The technical solution provided in this application can use acoustic features to train a pre-trained self-supervised model, and then extract the frame level. The latent features of , as new acoustic features, such hidden features contain the interrelated information of the acoustic features, and the hidden features are used as the input of the speech conversion model to predict the target acoustic features, so that the model can learn more fully and has a wide range of application scenarios; and, The technical solutions of the embodiments of the present application are simple and convenient to implement, easy to popularize, and have wider application range.

Embodiment 3

FIG. 4 is a third schematic flowchart of a training method for a speech conversion model provided by an embodiment of the present application. Based on the above technical solutions, it is further optimized and expanded, and can be combined with the above-mentioned optional embodiments. As shown in Figure 4, the training method of the speech conversion model may include the following steps:

S401. Input the original acoustic feature into the neural network model to obtain a feature sequence output by the neural network model; use the feature sequence output by the neural network model as a latent feature output by the pre-training model.

S402. Input the latent features into the speech conversion model to obtain the predicted acoustic features output by the speech conversion model.

S403. Calculate the loss value of the speech conversion model to be trained for speech based on the original acoustic feature and the predicted acoustic feature.

S404. Adjust the model parameters in the speech conversion model to be trained according to the speech loss value of the speech conversion model to be trained.

S405: Input the original acoustic features of the first user for the first voice and the original acoustic features of the second user for the second voice into the trained voice conversion model respectively, and obtain the first voice and the second voice through the voice conversion model. The target voice after voice conversion; wherein, the target voice includes content information of the first voice and timbre information of the second voice.

In this step, the electronic device can respectively input the original acoustic features of the first user for the first voice and the original acoustic features of the second user for the second voice into the trained voice conversion model, and obtain the The target voice converted from the first voice and the second voice; wherein, the target voice includes content information of the first voice and timbre information of the second voice. Specifically, the electronic device can first input the original acoustic features of the first user for the first speech into the trained pre-training model to obtain the latent features output by the pre-training model; then the latent features output by the pre-training model and the second The user inputs the original acoustic features of the second voice into the trained voice conversion model respectively, and obtains the predicted acoustic features output by the voice conversion model; and then inputs the predicted acoustic features output by the voice conversion model into the vocoder to obtain the acoustic features. The target voice output by the encoder.

FIG. 5 is a schematic structural diagram of a prediction system for a speech conversion model provided by an embodiment of the present application. As shown in FIG. 5 , the prediction system of the speech conversion model may include: a pre-trained model model, a speech conversion model and a vocoder; Acoustic features) are input to the trained pre-training model to obtain the latent features output by the pre-training model; then the latent features output by the pre-training model and the second user (B user) are aimed at the original acoustic features of the second voice (B user). (acoustic features) are respectively input to the trained voice conversion model to obtain the predicted acoustic features output by the voice conversion model; then the predicted acoustic features output by the voice conversion model are input to the vocoder to obtain the target voice output by the vocoder .

In the training method of the speech conversion model proposed in the embodiment of the present application, the original acoustic features of speech are firstly input into the pre-training model to obtain the latent features output by the pre-training model; then the hidden features are input into the speech conversion model to obtain the speech conversion model The output predicted acoustic features; the speech conversion model to be trained is then trained based on the original acoustic features and the predicted acoustic features. That is to say, the present application extracts interrelated information from the original acoustic features through the pre-training model when training the speech conversion model. In the existing training methods of speech conversion models, because the ppg features or the mel feature frames input by the model are independent of each other, the information in them is independent of each other, and it is difficult for the neural network model to learn from them. The interrelatedness between them limits the learning ability of the model. Because the present application adopts the technical means of extracting the associated information through the pre-training model, it overcomes the fact that the mel feature frames input by the ppg feature or the model are independent of each other in the prior art, thereby causing the information in them to be independent of each other and the neural network. It is difficult for the network model to learn the correlation between frames and the technical problem that the learning ability of the model is limited. The technical solution provided in this application can use acoustic features to train a pre-trained self-supervised model, and then extract the frame level. The latent features of , as new acoustic features, such hidden features contain the interrelated information of the acoustic features, and the hidden features are used as the input of the speech conversion model to predict the target acoustic features, so that the model can learn more fully and has a wide range of application scenarios; and, The technical solutions of the embodiments of the present application are simple and convenient to implement, easy to popularize, and have wider application range.

Embodiment 4

FIG. 6 is a schematic structural diagram of an apparatus for training a speech conversion model provided by an embodiment of the present application. As shown in FIG. 6 , the apparatus 600 includes: a pre-training module 601, a speech conversion module 602 and a training module 603; wherein,

The pre-training module 601 is used to input the original acoustic features of the speech into the pre-training model to obtain the latent features output by the pre-training model;

The voice conversion module 602 is used to input the latent features into a voice conversion model to obtain the predicted acoustic features output by the voice conversion model;

The training module 603 is configured to train the speech conversion model to be trained based on the original acoustic features and the predicted acoustic features.

Further, the pre-training module 601 is specifically configured to input the original acoustic features into a neural network model to obtain a feature sequence output by the neural network model; and use the feature sequence output by the neural network model as the The latent features output by the pretrained model.

Further, the pre-training module 601 is specifically configured to divide the original acoustic feature into N acoustic feature units; wherein, N is a natural number greater than 1; one or more of the N acoustic feature units Mask the acoustic features to obtain masked acoustic feature units; input the masked acoustic feature units into the neural network model to obtain a feature sequence output by the neural network model.

Further, the training module 603 is specifically configured to calculate the loss value of the speech conversion model to be trained for the speech based on the original acoustic features and the predicted acoustic features; The conversion model adjusts the model parameters in the to-be-trained speech conversion model according to the loss value of the speech.

Further, the apparatus further includes: a prediction module 604 (not shown in the figure), configured to respectively input the original acoustic features of the first user for the first voice and the original acoustic features of the second user for the second voice To the trained voice conversion model, the target voice converted from the first voice and the second voice is obtained through the voice conversion model; wherein, the target voice includes the content information of the first voice and all Describe the timbre information of the second voice.

Further, the prediction module 604 is specifically configured to input the original acoustic features of the first user for the first speech into the trained pre-training model to obtain the latent features output by the pre-training model; The latent features output by the pre-training model and the original acoustic features of the second user for the second voice are respectively input into the trained voice conversion model to obtain the predicted acoustic features output by the voice conversion model; The predicted acoustic features output by the speech conversion model are input to a vocoder to obtain the target speech output by the vocoder.

The above-mentioned training device for the speech conversion model can execute the method provided by any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, reference may be made to the training method of the speech conversion model provided by any embodiment of this application.

In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

Embodiment 5

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

FIG. 7 shows a schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 7 , the device 700 includes a computing unit 701 that can be executed according to a computer program stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage unit 708 Various appropriate actions and handling. In the RAM 703, various programs and data necessary for the operation of the device 700 can also be stored. The computing unit 701 , the ROM 702 , and the RAM 703 are connected to each other through a bus 704 . An input/output (I/O) interface 705 is also connected to bus 704 .

Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc. ; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 701 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the training method of the speech conversion model. For example, in some embodiments, the training method of the speech conversion model may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 700 via ROM 702 and/or communication unit 709 . When the computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the training method of the speech conversion model described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured by any other suitable means (eg, by means of firmware) to perform the training method of the speech conversion model.

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in different orders. As long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, there is no limitation herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.

Claims (10)

1. A training method for a speech conversion model, the method comprising: Divide the original acoustic feature into N acoustic feature units; wherein, N is a natural number greater than 1; mask one or more acoustic features in the N acoustic feature units to obtain a masked acoustic feature unit; The masked acoustic feature unit is input to the neural network model, and the feature sequence output by the neural network model is obtained; the feature sequence output by the neural network model is used as the hidden feature output by the pre-training model; Inputting the latent features to a speech conversion model to obtain the predicted acoustic features output by the speech conversion model; The speech conversion model to be trained is trained based on the original acoustic features and the predicted acoustic features. 2. The method according to claim 1, wherein the training of the speech conversion model to be trained based on the original acoustic features and the predicted acoustic features comprises: Calculate the loss value of the speech conversion model to be trained for the speech based on the original acoustic feature and the predicted acoustic feature; The model parameters in the to-be-trained speech-conversion model are adjusted according to the speech loss value of the to-be-trained speech-conversion model. 3. The method of claim 1, further comprising: The original acoustic features of the first user for the first voice and the original acoustic features of the second user for the second voice are respectively input into the trained voice conversion model, and the first voice and the second voice are obtained through the voice conversion model. The target voice after the second voice conversion; wherein, the target voice includes content information of the first voice and timbre information of the second voice. 4. The method according to claim 3, wherein the original acoustic features of the first user for the first voice and the original acoustic features of the second user for the second voice are respectively input into the trained voice conversion model , obtain the target voice converted by the first voice and the second voice through the voice conversion model, including: Inputting the original acoustic features of the first voice to the trained pre-training model by the first user to obtain the latent features output by the pre-training model; The latent features output by the pre-training model and the original acoustic features of the second user for the second voice are respectively input into the trained voice conversion model to obtain the predicted acoustic features output by the voice conversion model; The predicted acoustic features output by the speech conversion model are input to a vocoder to obtain the target speech output by the vocoder. 5. A training device for a voice conversion model, the device comprising: a pre-training module, a voice conversion module and a training module; wherein, The pre-training module is used to divide the original acoustic feature into N acoustic feature units; wherein, N is a natural number greater than 1; mask one or more acoustic features in the N acoustic feature units to obtain The masked acoustic feature unit; the masked acoustic feature unit is input into the neural network model, and the feature sequence output by the neural network model is obtained; the feature sequence output by the neural network model is used as the hidden output of the pre-training model. feature; The speech conversion module is used for inputting the latent features into the speech conversion model to obtain the predicted acoustic features output by the speech conversion model; The training module is configured to train the speech conversion model to be trained based on the original acoustic features and the predicted acoustic features. 6. The apparatus according to claim 5, wherein the training module is specifically configured to calculate the loss value of the speech conversion model to be trained for the speech based on the original acoustic feature and the predicted acoustic feature; The model parameters in the to-be-trained speech-conversion model are adjusted according to the speech loss value of the to-be-trained speech-conversion model. 7. The apparatus according to claim 5, further comprising: a prediction module for inputting the original acoustic features of the first user for the first voice and the original acoustic features of the second user for the second voice respectively. To a trained voice conversion model, the target voice converted from the first voice and the second voice is obtained through the voice conversion model; wherein, the target voice includes the content information of the first voice and all Describe the timbre information of the second voice. 8 . The apparatus according to claim 7 , wherein the prediction module is specifically configured to input the original acoustic features of the first voice by the first user into a trained pre-training model to obtain the pre-training. 9 . The hidden features output by the model; the hidden features output by the pre-training model and the original acoustic features of the second user for the second voice are respectively input into the trained voice conversion model to obtain the output of the voice conversion model and input the predicted acoustic features output by the speech conversion model to a vocoder to obtain the target speech output by the vocoder. 9. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-4 Methods. 10. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any of claims 1-4.

Images