CN114187892A - Style migration synthesis method and device and electronic equipment - Google Patents

Abstract

The disclosure provides a style migration and synthesis method and device and electronic equipment. The invention relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, speech synthesis and style migration, and specifically relates to a speech style migration and synthesis method, a speech style migration and synthesis device and electronic equipment. The specific implementation scheme is as follows: inputting the target text and the target audio clip into a speech synthesis model obtained by training the sample text and the sample audio clip in advance; superposing the coarse-grained audio features and the fine-grained audio features on each audio unit in the target audio clip to obtain superposed audio features of the audio units; extracting pronunciation characteristics of each pronunciation unit in the target text; fusing the pronunciation characteristics of the pronunciation units and the target superposition audio characteristics aiming at each pronunciation unit in the target text to obtain the fusion characteristics of the pronunciation units; the audio segments are synthesized according to the fusion features. An audio piece can be synthesized that has a target style in its entirety and in detail.

Description

A style transfer synthesis method, device and electronic device

technical field

The present disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, speech synthesis, and style transfer, and in particular, to a voice style transfer synthesis method, device, and electronic device.

Background technique

For various practical needs, for example, in order to realize the voice changing function provided in the voice chat software, hide the real identity of the speaker, etc., it is necessary to synthesize a given audio clip and text to obtain a voice style with the same voice style as the audio clip. The speech content is the audio segment of the text. Since this process can be regarded as transferring the speech style of the audio segment to the text, this process is called style transfer synthesis.

SUMMARY OF THE INVENTION

The present disclosure provides a style transfer synthesis method, device and electronic device.

According to a first aspect of the present disclosure, a style transfer synthesis method is provided, comprising:

Input the target text and the target audio clip with the target speech style into the speech synthesis model trained in advance through the sample text and the sample audio clip;

Through the style extraction sub-model of the speech synthesis model, for each audio unit in the target audio segment, the coarse-grained audio feature used to characterize the target audio segment and the fine-grained audio feature used to characterize the audio unit are superimposed feature, to obtain the superimposed audio feature of the audio unit;

Extract the pronunciation feature of each pronunciation unit in the target text by the content coding sub-model of the speech synthesis model;

Through the content style cross-attention sub-model of the speech synthesis model, for each pronunciation unit in the target text, the pronunciation feature of the pronunciation unit and the target superimposed audio feature are fused to obtain the fusion feature of the pronunciation unit, Wherein, the target superimposed audio feature is the superimposed audio feature matched with the pronunciation feature;

Through the sound spectrum decoding sub-model of the speech synthesis model, according to the fusion feature of each pronunciation unit in the target text, an audio segment having the target speech style and the speech content of the target text is synthesized.

According to a second aspect of the present disclosure, a method for training a speech synthesis model is provided, including:

Input the sample audio clip and sample text into the original model, wherein the sample text is the voice content of the sample audio clip;

Using the original model, for each audio unit in the sample audio segment, superimposing the coarse-grained audio feature used to characterize the sample audio segment and the fine-grained audio feature used to characterize the audio unit to obtain the audio Overlay audio characteristics of the unit;

Through the original model, extract the pronunciation feature of each pronunciation unit in the sample text;

Through the original model, for each pronunciation unit in the sample text, the pronunciation feature of the pronunciation unit and the target superimposed audio feature are fused to obtain the fusion feature of the pronunciation unit, wherein the target superimposed audio feature is an overlay audio feature that matches the pronunciation feature;

Through the original model, the fusion feature of each pronunciation unit in the sample text is converted into a predicted spectral feature;

adjusting the model parameters of the original model according to the difference between the predicted acoustic spectrum feature and the real acoustic spectrum feature of the sample audio clip;

Obtain new sample audio clips and new sample texts, return to performing the step of inputting the sample audio clips and sample texts into the original model, until the first convergence condition is reached, and use the adjusted original model as a speech synthesis model.

According to a third aspect of the present disclosure, there is provided a style transfer synthesis device, comprising:

The first input module is used to input the target text and the target audio clip with the target voice style into the speech synthesis model trained in advance through the sample text and the sample audio clip;

A style extraction module for extracting a sub-model through the style of the speech synthesis model, for each audio unit in the target audio segment, superimposing the coarse-grained audio features used to characterize the target audio Fine-grained audio features of the audio unit, to obtain the superimposed audio features of the audio unit;

A content coding module for extracting the pronunciation feature of each pronunciation unit in the target text by the content coding sub-model of the speech synthesis model;

The content style cross attention module is used for the content style cross attention sub-model of the speech synthesis model, for each pronunciation unit in the target text, fuse the pronunciation feature of the pronunciation unit and the target superimposed audio feature, Obtain the fusion feature of the pronunciation unit, wherein, the target overlay audio feature is the overlay audio feature matched with the pronunciation feature;

The sound spectrum decoding module is used for the sound spectrum decoding sub-model of the speech synthesis model, according to the fusion feature of each pronunciation unit in the target text, the synthesis has the target voice style and the voice content is the target An audio clip of the text.

According to a fourth aspect of the present disclosure, there is provided a training device for a speech synthesis model, comprising:

a second input module for inputting sample audio clips and sample texts into the original model, wherein the sample texts are the voice content of the sample audio clips;

The first original module is configured to, through the original model, superimpose, for each audio unit in the sample audio segment, a coarse-grained audio feature used to characterize the sample audio segment and a fine-grained audio feature used to characterize the audio unit Audio features, to obtain the superimposed audio features of the audio unit;

The second original module is used to extract the pronunciation feature of each pronunciation unit in the sample text through the original model;

The third original module is used for, through the original model, for each pronunciation unit in the sample text, fuse the pronunciation feature of the pronunciation unit and the target superimposed audio feature to obtain the fusion feature of the pronunciation unit, wherein, The target superimposed audio feature is the superimposed audio feature matched with the pronunciation feature;

The fourth original module is used for converting into predicted spectral features according to the fusion feature of each pronunciation unit in the sample text through the original model;

A parameter adjustment module, configured to adjust the model parameters of the original model according to the difference between the predicted sound spectrum feature and the real sound spectrum feature of the sample audio clip;

The acquisition module is used to acquire new sample audio clips and new sample texts, and returns to execute the step of inputting the sample audio clips and sample texts into the original model until the first convergence condition is reached, and the adjusted original model is used as the voice synthetic model.

According to a fifth aspect of the present disclosure, there is provided an electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any one of the first or second aspects above method described in item.

According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform any one of the first or second aspects above method described in item.

According to a seventh aspect provided by the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of the above first or second aspects.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present disclosure. in:

1 is a schematic flowchart of a style transfer synthesis method provided according to the present disclosure;

2 is a schematic structural diagram of a speech synthesis model used in the style transfer synthesis method provided according to the present disclosure;

3a is a schematic structural diagram of a style extraction sub-model in a speech synthesis model used in the style transfer synthesis method provided according to the present disclosure;

3b is a schematic structural diagram of a content coding sub-model in a speech synthesis model used in the style transfer synthesis method provided according to the present disclosure;

3c is a schematic structural diagram of a content-style cross-attention sub-model in a speech synthesis model used in the style transfer synthesis method provided according to the present disclosure;

3d is a schematic structural diagram of a spectral decoding sub-model in a speech synthesis model used in the style transfer synthesis method provided according to the present disclosure;

4 is a schematic flowchart of a training method for a speech synthesis model provided according to the present disclosure;

5 is a schematic structural diagram of a style transfer synthesis device provided according to the present disclosure;

6 is a schematic structural diagram of a training device for a speech synthesis model provided according to the present disclosure;

FIG. 7 is a block diagram of an electronic device used to implement the style transfer synthesis method or the training method of the speech synthesis model according to the embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In order to explain the style transfer synthesis method provided by the present disclosure more clearly, a possible application scenario of the style transfer synthesis method provided by the present disclosure will be exemplified below. It can be understood that the following example is only a style provided by the disclosure. A possible application scenario of the migration synthesis method, in other possible embodiments, the style migration synthesis method provided by the present disclosure can also be applied to other possible application scenarios, and the following examples do not make any limitation to this.

For the purpose of hiding the true identity of the target person, such as players of online games, interviewees in news interviews, etc., the words spoken by the target person can be converted into text, and then synthesized through style transfer, so that the voice style different from the target person can be changed. Another voice style (hereinafter referred to as the target voice style) is transferred to the text. Since the voice style of the synthesized audio clip is different from that of the target person, when other people hear the synthesized audio clip, they will not associate the target person. So as to achieve the purpose of hiding the real identity.

In the related art, in order to realize the style transfer synthesis, the encoding network is often used to encode the target audio segment with the target speech style to obtain the style features, and then the words spoken by the target person are converted into text to encode the content features, and the style is converted into text. The features and content features are input to the pre-trained decoding network to obtain the spectral features output by the decoding network, and then the vocoder converts the spectral features into audio clips, so as to obtain the text with the target voice style and the voice content. audio clip.

However, this solution can only make the synthesized audio segment sound similar to the target audio frequency band as a whole, but some details, such as speech rate, emotion, pitch, cadence, short pause, accent, etc., are different from the target audio frequency. Fragments are quite different. In other words, the synthesized audio segment does not have the target style in detail.

Based on this, the present disclosure provides a style transfer synthesis method, which can be applied to any electronic device with a style transfer synthesis function, including but not limited to mobile phones, tablet computers, personal computers, servers, etc. The style transfer synthesis method provided by the present disclosure It can be shown in Figure 1, including:

S101, input the target text and the target audio segment with the target speech style into a speech synthesis model trained in advance through the sample text and the sample audio segment.

S102, extracting the sub-model by the style of the speech synthesis model, for each audio unit in the target audio segment, superimpose the coarse-grained audio feature used to characterize the target audio segment and the fine-grained audio feature used to characterize the audio unit to obtain the audio unit of the audio unit. Overlay audio features.

S103, extract the pronunciation feature of each pronunciation unit in the target text through the content coding sub-model of the speech synthesis model.

S104, through the content style cross-attention sub-model of the speech synthesis model, for each pronunciation unit in the target text, the pronunciation feature of the fusion pronunciation unit and the target superimposed audio feature are obtained, and the fusion feature of the pronunciation unit is obtained, wherein, the target must add audio Features are superimposed audio features matched with pronunciation features.

S105 , synthesizing an audio segment having the target speech style and the speech content as the target text according to the fusion feature of each pronunciation unit in the target text by using the sound spectrum decoding sub-model of the speech synthesis model.

In this embodiment, the superimposed audio features are obtained by superimposing audio features with different granularities, so that the superimposed audio features can not only reflect the overall characteristics of the target style, but also can reflect the detailed characteristics of the target audio segment. Then, the cross-attention mechanism is used to fuse the audio features of each pronunciation unit with the matching superimposed audio features. Since the audio features are fused with the matching superimposed audio features, the detailed features reflected by the fusion features of each pronunciation unit can reflect the detailed features of the pronunciation unit in the target style. Therefore, in the audio clip synthesized according to the fusion feature, not only the overall acoustic features are similar to the target style, but also the pronunciation of each pronunciation unit is similar to the target style, that is, the audio clip with the target style in the whole and details can be synthesized.

In order to more clearly describe the steps of S101-S105, the speech synthesis model provided by the present disclosure will first be described below. Referring to FIG. 2, FIG. 2 shows a schematic structural diagram of the speech synthesis model provided by the present disclosure, including :

Style extraction sub-model, content encoding sub-model, content-style attention sub-model, and spectral decoding sub-model.

The input of the style extraction sub-model is the target audio segment, and the output is the superimposed audio feature of each audio unit in the target audio segment. Each audio unit is composed of M consecutive audio frames in the audio segment, and each audio frame belongs to only one audio unit. M may be set by the user according to actual experience or requirements, such as M=2, 4, 5, 9, etc., which is not limited in the present disclosure. Each audio frame is continuous N ms audio data in the audio segment, the interval between two adjacent audio frames is Q ms, and Q is not greater than N, for example, N=25, Q=10, N=20, Q= 20, N=28, Q=21, etc., which is not limited in the present disclosure.

The input of the content coding sub-model is the target text, and the output is the pronunciation features of each pronunciation unit in the target text. The content coding sub-model is used to implement the aforementioned step of S103.

Wherein, each pronunciation unit in the target text is composed of k consecutive phonemes in the pronunciation of the target text, and k can be set according to the actual needs or experience of the user. For example, when k=1, each pronunciation The unit is a phoneme in the pronunciation of the target text. Taking the target text as "hundred" and the pronunciation of the target text as Chinese pronunciation as an example, it includes the pronunciation unit "b" and the pronunciation unit "ai".

The input of the content-style attention sub-model is the superimposed audio features of each audio unit in the target audio segment, and the audio features of each pronunciation unit in the target text, that is, the input of the content-style attention sub-model is the style extraction sub-model and the output of the content encoding submodel. The output of the content-style attention sub-model is each pronunciation unit in the target text. The content-style cross-attention sub-model is used to implement the aforementioned step of S104.

The input of the audio spectrum decoding sub-model is each pronunciation unit in the target text, that is, the input of the audio spectrum decoding sub-model is the output of the content style and attention sub-model. The output of the spectral decoding submodel is a synthesized audio segment. The spectral decoding sub-model is used to implement the aforementioned step of S105.

The implementation of the aforementioned S102-S105 will be described below in conjunction with the structure of each sub-model in the speech synthesis model, referring to Fig. 3a-Fig. 3d, and Fig. 3a-Fig. 3d is a schematic structural diagram of each sub-model in the speech synthesis model:

As shown in Figure 3a, the style extraction sub-model includes a wave pair feature vector (Wav2Vec) sub-network, which consists of a Linear (Linear) sub-network, a Long Short-Term Memory (LSTM) sub-network, a pooling (pooling) The first branch composed of sub-networks, and the second branch composed of long short-term memory sub-networks and pooling sub-networks.

The input of the wave pair feature vector sub-network is the target audio segment, and the output is the audio features of each audio frame in the target audio segment. The number of target audio segments input to the wave pair feature vector sub-network may be one or multiple, which is not limited in the present disclosure. Exemplarily, in a possible embodiment, the number of target audio clips is two, wherein one target audio clip is used to reflect the overall feature of the target style, and the other target audio clip is used to reflect the detailed feature of the target style, In another possible embodiment, the number of target audio segments is four, wherein one target audio segment is used to reflect the overall feature of the target style, and the remaining three target audio segments are used to reflect the detailed feature of the target style.

The input of the first branch and the second branch is the audio feature of each audio frame in the target audio segment, that is, the input of the first branch and the second branch is the output of the wave pair feature vector sub-network. If the number of target audio clips is one, the inputs of the first branch and the second branch are the same, and both are feature vectors of each audio frame in the one target audio clip. If the number of target audio clips is multiple, the input of the first branch and the second branch are different, and the input of the first branch is the audio feature of each audio frame in the target audio clip used to reflect the detailed features of the target style , and the input of the second branch is the audio feature of each audio frame in the target audio segment that is used to reflect the overall feature of the target style.

Exemplarily, assuming that the target audio clip includes a first target audio clip and a second target audio clip, wherein the first target audio clip is used to reflect the detailed features of the target style, and the second target audio clip reflects the overall characteristics of the target style, then The input of the first branch is the audio feature of each audio frame in the first target audio segment, and the input of the second branch is the audio feature of each audio frame in the second target audio segment.

The output of the first branch is the fine-grained audio features of each audio unit in the target audio segment. For each audio unit, the first branch averages the audio features of all audio frames included in the audio unit to obtain fine-grained audio features for characterizing the audio unit. The first branch obtains fine-grained audio features in this way. Since the fine-grained audio features only include the audio features of each audio frame in the audio unit, the obtained fine-grained audio features can retain more detailed features of the target style. .

The output of the second branch is the coarse-grained audio features of the target audio segment. The second branch averages the audio features of all audio frames in the target audio segment to obtain coarse-grained audio features for characterizing the target audio segment. The second branch obtains coarse-grained audio features in this way. Since the coarse-grained audio features average the audio features of all audio frames, the obtained coarse-grained features retain more overall features of the target style.

The style extraction sub-model adds each fine-grained audio feature output from the first branch to the coarse-grained audio feature output from the second branch to obtain superimposed audio features, which are output to the content-style cross-attention sub-model.

Exemplarily, it is assumed that there are three audio units in total, which are denoted as audio units 1-3 respectively. The fine-grained audio feature of audio unit 1 is x1, the fine-grained audio feature of audio unit 2 is x2, and the fine-grained audio feature of audio unit 3 is x2. The feature is x3, and assuming that the coarse-grained audio feature is x4, the superimposed audio feature of audio unit 1 is x1+x4, the fine-grained audio feature of audio unit 2 is x2+x4, and the fine-grained audio feature of audio unit 3 is x3+ x4. Among them, x1+x4 refers to adding the values of x1 and x4 on each feature dimension. For example, suppose there are three feature dimensions in total, and x1 is (a1, b1, c1), and x4 is (a4, b4, c4 ), the superimposed audio feature is (a1+a4, b1+b4, c1+c4). The same is true for x2+x4 and x3+x4, which will not be repeated here.

As shown in Figure 3b, the content coding sub-model consists of a phoneme embedding (Phoneme Embedding) sub-network, a layer norm (Layer Norm) sub-network and a position embedding (Position embedding) sub-network.

The input of the phoneme embedding sub-network is the target text, and the output is the feature vector of each pronunciation unit in the target text. The phoneme embedding sub-network is used to encode the pronunciation of each phonetic unit in the target text as a feature vector.

The input of the layer norm sub-network is the feature vector of each pronunciation unit in the target text, that is, the input of the layer norm sub-network is the output of the phoneme embedding sub-network. A layer norm digit network is used to further extract pronunciation features from the feature vector.

The input of the position embedding sub-network is the target text, and the output is the position code of each pronunciation unit in the target text. The position embedding sub-network is used to encode the position of each pronunciation unit in the target text, and obtain the position code of each pronunciation unit in the target text.

The content encoder combines the pronunciation features of each pronunciation unit in the target text with the position encoding to indicate the position of the pronunciation unit to which each pronunciation feature belongs in the target text through the position encoding, and outputs the pronunciation features to the content style cross-attention sub-model.

As shown in Figure 3c, the content-style cross-attention sub-model includes the Text Self-attention sub-network (hereinafter referred to as the self-attention sub-network), the first add, norm and mapping (Add&Norm&Projection) sub-network, Multi-head cross-attention sub-network (hereinafter referred to as cross-attention sub-network) and second addition, norm and mapping sub-network. The first addition, norm and mapping sub-network and the second addition, norm and mapping sub-network consist of an addition, norm and mapping sub-network and a Feedforward Neural sub-network.

The input of the self-attention network is the audio features of each pronunciation unit in the target text, that is, the input of the self-attention sub-network is the output of the content coding sub-model. The output of the self-attention sub-network is the adjusted audio features. The self-attention sub-network is used to strengthen the audio features of the relatively important pronunciation units in each pronunciation unit and weaken the audio features of the relatively unimportant pronunciation units through the self-attention mechanism, so that the pronunciation characteristics of each pronunciation unit can be better reflect the characteristics of the target text.

The input of the cross-attention sub-network is the superimposed audio features of each audio unit in the target audio segment and the adjusted pronunciation features of each pronunciation unit in the target text. The output of the cross-attention sub-network is the fusion feature of each pronunciation unit in the target text.

The cross-attention sub-network is used to use the superimposed audio features of each audio unit as the key (key) and value (value), and the adjusted pronunciation feature of each pronunciation unit as the query (query). Query, find the key matching the query in the key, and fuse the value corresponding to the key (ie the target superimposed audio feature) with the query to obtain the fusion feature. That is, for each pronunciation unit, the cross-attention sub-network fuses the adjusted pronunciation feature of the pronunciation unit with the target superimposed audio feature to obtain the fusion feature of the pronunciation unit.

As shown in Figure 3d, the sound spectrum decoding sub-model includes a plurality of transformation sub-networks, a pre-net sub-network, a post-processing (Post-net) sub-network, and a vocoder (WaveGlow Vocoder).

The input of the preprocessing sub-network is the original Mel-spectrograms (Mel-spectrograms), which are used as the original spectral features, and the output of the preprocessing sub-network is the preprocessed original spectral features. Features are preprocessed. The preprocessing network consists of multiple linear sub-networks and linear rectification function (Relu) sub-networks.

In a possible embodiment, the input of the conversion sub-network is the preprocessed original spectral feature and the fusion feature of each pronunciation unit. The output of the transformation sub-network is transformed into a fused feature into a spectral feature. In this embodiment, the conversion sub-network is used to convert the input fusion feature into a sound spectrum feature according to the input original sound spectrum feature.

In another possible embodiment, the input of the conversion sub-network is the preprocessed original sound spectrum feature, the coarse-grained feature, and the fusion feature of each pronunciation unit, and the coarse-grained feature is the output of the aforementioned lower branch. The output of the transformation sub-network is transformed into a fused feature into a spectral feature. In this embodiment, the conversion sub-network is used to convert the input fusion feature into a sound spectrum feature according to the input original sound spectrum feature and coarse-grained feature.

It can be understood that the fusion feature is obtained by fusing superimposed audio features and pronunciation features, and superimposed audio features are obtained by superimposing coarse-grained audio features and fine-grained audio features, so fusion features can reflect coarse-grained features to a certain extent. audio features. However, as explained above, the fusion features are obtained from coarse-grained audio features through a series of calculations, so it is difficult for the fusion features to accurately reflect the coarse-grained audio features. Therefore, the fusion spectral features converted by the conversion sub-network cannot accurately reflect the target. The overall character of the style.

Therefore, inputting coarse-grained audio features into the conversion sub-network enables the conversion sub-network to accurately refer to the overall features of the target style when converting the fusion features into spectral features, so that the converted spectral features can accurately reflect The overall characteristics of the target style are obtained, so that the subsequent synthesized audio clips have the target style as a whole.

The input of the post-processing sub-network is the converted sound spectrum feature, and the output of the post-processing sub-network is the post-processed sound spectrum feature. The post-processing sub-network is used to post-process the transformed spectral features. The post-processing sub-network consists of multiple Convolutional Neural Networks.

The input of the vocoder is the post-processed spectral features, that is, the input of the vocoder is the output of the post-processing sub-network, and the output of the vocoder is the synthesized audio segment with the target style and the speech content as the target text. Vocoders are used to convert input audio features into audio clips.

Corresponding to the foregoing style transfer synthesis method, the present disclosure also provides a speech synthesis model training method for training the speech synthesis model used in the foregoing style transfer synthesis method.

The speech synthesis model training method provided in the present disclosure can be applied to any electronic device capable of speech synthesis model training, including but not limited to servers, personal computers, and the like. In addition, the speech synthesis model provided by the present disclosure and the style transfer synthesis method provided by the present disclosure may be applied to the same device or to different devices, which is not limited in the present disclosure.

The speech synthesis model training method provided by the present disclosure can refer to FIG. 4, including:

S401 , input the sample audio segment and the sample text into the original model, wherein the sample text is the speech content of the sample audio segment.

The sample text is the speech content of the sample audio segment means: the text obtained by the speech recognition of the sample audio segment is the sample text. Exemplarily, assuming that a sample audio clip is an audio clip recorded when Zhang San said "AAAABBBB", the sample text is "AAAABBBB".

S402, through the original model, for each audio unit in the sample audio clip, superimpose the coarse-grained audio feature used to characterize the sample audio clip and the fine-grained audio feature used to characterize the audio unit to obtain the superimposed audio feature of the audio unit.

The principle of the original model is exactly the same as the structure and principle of the aforementioned speech synthesis model, the only difference is that the model parameters of the original model are different from those of the speech synthesis model. Therefore, reference can be made to the foregoing related descriptions on the style extraction sub-model, which will not be repeated here.

S403, extract the pronunciation feature of each pronunciation unit in the sample text through the original model.

The principle of this step is the same as the principle of the foregoing content coding sub-model, and reference may be made to the foregoing relevant description of the content coding sub-model, and details are not repeated here.

S404, through the original model, for each pronunciation unit in the sample text, the pronunciation feature of the fusion pronunciation unit and the target overlay audio feature are obtained, and the fusion feature of the pronunciation unit is obtained, wherein, the target overlay audio feature is the overlay audio feature matched with the pronunciation feature .

The principle of this step is the same as the principle of the aforementioned content-style cross-attention sub-model, and reference may be made to the aforementioned relevant description of the content-style cross-attention sub-model, which will not be repeated here.

S405, through the original model, convert the fusion features of each pronunciation unit in the sample text into predicted spectral features.

The principle of this step is the same as the principle of the aforementioned sound spectrum decoding sub-model, and reference may be made to the aforementioned related description of the sound spectrum decoding sub-model and sub-model, which will not be repeated here.

S406, according to the difference between the predicted sound spectrum feature and the real sound spectrum feature of the sample audio feature, adjust the model parameters of the original model.

It can be understood that since the sample text input to the original network is the content of the sample audio clip, the audio clip obtained by the original network migrating the semantic style of the sample audio clip to the sample text should be the sample audio clip. In other words, if the original network can For accurate style transfer synthesis, the predicted spectral features should be the same as the real spectral features of the sample audio segment. The reason for the difference between the predicted spectral features and the real spectral features is that the original model cannot accurately perform style transfer synthesis.

Therefore, the difference between the predicted sound spectrum feature and the real sound spectrum feature can be used to guide the adjustment of the model parameters of the original model, so that the model parameters of the original model are adjusted in the direction of reducing the difference, so that the training can accurately perform style transfer. Synthesized speech synthesis model.

In S407, a new sample audio segment and a new sample text are acquired, and the process returns to S401 until the first convergence condition is reached, and the adjusted original model is used as the speech synthesis model.

The newly acquired sample text should be the content of the newly acquired sample audio segment, and the newly acquired sample audio segment is different from the previous sample audio segment. The first convergence condition may be set by the user according to actual requirements or requirements. Exemplarily, the first convergence condition may be that the convergence of the model parameters of the original model reaches a preset convergence threshold, and the first convergence condition may also be that the first convergence condition has been used. The number of sample audio clips reaches a preset number threshold.

This embodiment is selected to supervise the training of the original model by using the real spectral features of the sample audio, and since the sample text is the speech content of the sample audio clips, the superimposed audio features extracted by the original model can better match the various features in the sample text. The audio features of the pronunciation units are matched, so that the speech synthesis model obtained by training can better learn the matching relationship between the audio features and the pronunciation features, thereby synthesizing audio clips with more target style in the process of style transfer synthesis.

It can be understood that, since the sample text is the speech content of the sample audio clip, the number of pronunciation units in the sample text should be similar to or even the same as the number of audio units in the sample audio clip. When using the speech synthesis model for style transfer synthesis, the target text is not the speech content of the target audio segment, so the number of pronunciation units of the target text may be quite different from the number of audio units in the target audio segment.

In order to enable the speech synthesis model to be able to accurately implement style transfer synthesis even when the number of pronunciation units and the number of audio units is quite different. In a possible embodiment, the target superimposed audio feature in the aforementioned S404 is a filtered audio feature matching the pronunciation feature, and the filtered audio feature is a partial superimposed audio feature extracted from all superimposed audio features.

The extraction method is random extraction, and the number of superimposed audio features to be extracted can be set according to the actual needs or experience of the user, such as extracting 80% of the superimposed audio features as the filtered audio features, or extracting 90% of the superimposed audio features as the filter. Post audio features.

By selecting this embodiment, in the training process, by extracting part of the superimposed audio features, the audio features are only fused with the selected part of the superimposed audio features through the cross-attention mechanism, so that the speech synthesis model obtained by training can learn how to Implementing style transfer synthesis when the number of pronunciation units and audio units differs greatly, that is, the speech synthesis model can accurately implement style transfer synthesis even when the numbers of pronunciation units and audio units are greatly different.

In the foregoing S407, the acquired new sample audio segment and the previous sample audio segment may have the same style (that is, the audio segment of the same person), or may have a different voice style from the previous sample audio segment (that is, the audio segment of the same person). audio clips for different people).

In a possible embodiment, the acquisition of the new sample segment in the aforementioned S407 is implemented in the following manner:

If the second convergence condition is not met, a new sample audio segment is obtained from the first sample data set, and if the second convergence condition is met, a new sample audio segment is obtained from the second sample data set.

The first sample data set includes audio clips of the first sample person, and the second sample data set includes audio clips of multiple sample people. And in this embodiment, the sample audio segment when S401 is executed for the first time is the audio segment of the first sample person.

The difficulty of achieving the second convergence condition is lower than that of the first convergence condition, that is, when the second convergence condition is met, the first convergence condition has not been met, and when the first convergence condition is met, the second convergence condition has been met.

By choosing this embodiment, the original model can be trained first by using the audio features of the first sample person, so that the original model can learn how to transfer the voice style of the first sample person to a specific text, and then use the respective The audio clips train the original model so that the original model learns how to transfer different speech styles to specific texts. Since the original model has been pre-trained to learn how to transfer the voice style of the first sample of people to a specific text before learning how to transfer different speech styles to a specific text, only a few audio clips of different people are needed, The training of the original model can be completed, which effectively reduces the difficulty of obtaining sample audio clips.

Referring to FIG. 5, FIG. 5 shows a schematic structural diagram of a style transfer synthesis device provided by the present disclosure, including:

The first input module 501 is used to input the target text and the target audio clip with the target voice style into the speech synthesis model trained in advance through the sample text and the sample audio clip;

The style extraction module 502 is used to extract the sub-model through the style of the speech synthesis model, and for each audio unit in the target audio segment, superimpose the coarse-grained audio feature used to characterize the target audio segment and the coarse-grained audio feature used to characterize the target audio segment. the fine-grained audio feature of the audio unit, to obtain the superimposed audio feature of the audio unit;

The content coding module 503 is used to extract the pronunciation feature of each pronunciation unit in the target text by the content coding sub-model of the speech synthesis model;

The content style cross attention module 504 is used for the content style cross attention sub-model of the speech synthesis model, for each pronunciation unit in the target text, fuse the pronunciation feature of the pronunciation unit and the target superimposed audio feature , obtain the fusion feature of described pronunciation unit, wherein, described target superimposed audio feature is the superimposed audio feature matched with described pronunciation feature;

The sound spectrum decoding module 505 is used to synthesize the sound spectrum decoding sub-model of the speech synthesis model, according to the fusion feature of each pronunciation unit in the target text, to synthesize the target speech style and the speech content as described. Audio snippet of the target text.

In this embodiment, the superimposed audio features are obtained by superimposing audio features with different granularities, so that the superimposed audio features can not only reflect the overall characteristics of the target style, but also can reflect the detailed characteristics of the target audio segment. Then, the cross-attention mechanism is used to fuse the audio features of each pronunciation unit with the matching superimposed audio features. Since the audio features are fused with the matching superimposed audio features, the detailed features reflected by the fusion features of each pronunciation unit can reflect the detailed features of the pronunciation unit in the target style. Therefore, in the audio clip synthesized according to the fusion feature, not only the overall acoustic features are similar to the target style, but also the pronunciation of each pronunciation unit is similar to the target style, that is, the audio clip with the target style in the whole and details can be synthesized.

In a possible embodiment, the style extraction module 502 uses a style extraction sub-model of the speech synthesis model to superimpose, for each audio unit in the target audio segment, a rough representation of the target audio segment Granular audio features and fine-grained audio features used to characterize the audio unit, to obtain the superimposed audio features of the audio unit, including:

Through the style extraction module of the speech synthesis model, the average audio features of all audio frames in the target audio segment are extracted as coarse-grained audio features;

By the style extraction module, for each audio unit in the target audio segment, the average audio feature of all audio frames in the audio unit is extracted as the fine-grained audio feature of the audio unit;

Through the style extraction module, for each audio unit in the target audio segment, the fine-grained audio feature of the audio unit and the coarse-grained audio feature are added to obtain the superimposed audio of the audio unit feature.

In a possible embodiment, the content-style cross-attention module 504 fuses the pronunciation of the pronunciation unit for each pronunciation unit in the target text through the content-style cross-attention sub-model of the speech synthesis model Features and target superimposed audio features to obtain the fusion features of the pronunciation unit, including:

Input the pronunciation feature of each pronunciation unit in the target text to the self-attention sub-network of the content-style cross-attention sub-model in the speech synthesis model, obtain the adjusted pronunciation characteristic of the output of the sub-attention sub-network;

Through the cross-attention sub-network of the content intersection sub-model, for each pronunciation unit in the target text, the adjusted pronunciation feature of the pronunciation unit and the target superimposed audio feature are fused to obtain the fusion of the pronunciation unit feature, wherein the target superimposed audio feature is the superimposed audio feature matched with the adjusted pronunciation feature.

In a possible embodiment, the sound spectrum decoding module 505 uses the sound spectrum decoding sub-model of the speech synthesis model to synthesize the target text according to the fusion feature of each pronunciation unit in the target text. The voice style and the voice content are audio clips of the target text, including:

Inputting the fusion feature and the coarse-grained audio feature of each pronunciation unit in the target text into the audio spectrum decoding sub-model of the speech synthesis model, to obtain the audio spectrum feature output by the audio spectrum decoding sub-network;

Converting the spectral feature into an audio segment having the target speech style and the speech content being the target text.

Referring to FIG. 6, FIG. 6 shows a schematic structural diagram of a training device for a speech synthesis model provided by an embodiment of the present invention, which may include:

The second input module 601 is configured to input sample audio clips and sample texts into the original model, wherein the sample texts are the voice content of the sample audio clips;

The first original module 602 is configured to, through the original model, superimpose, for each audio unit in the sample audio segment, a coarse-grained audio feature used to characterize the sample audio segment and a fine-grained audio feature used to characterize the audio unit. Granular audio features, to obtain the superimposed audio features of the audio unit;

The second original module 603 is used to extract the pronunciation feature of each pronunciation unit in the sample text through the original model;

The third original module 604 is configured to, through the original model, for each pronunciation unit in the sample text, fuse the pronunciation feature of the pronunciation unit and the target superimposed audio feature to obtain the fusion feature of the pronunciation unit, wherein , the target superimposed audio feature is the superimposed audio feature matched with the pronunciation feature;

The fourth original module 605 is used for converting into predicted spectral features according to the fusion feature of each pronunciation unit in the sample text through the original model;

A parameter adjustment module 606, configured to adjust the model parameters of the original model according to the difference between the predicted sound spectrum feature and the real sound spectrum feature of the sample audio clip;

The acquisition module 607 is used to acquire new sample audio clips and new sample texts, and returns to execute the step of inputting the sample audio clips and sample texts into the original model, until the first convergence condition is reached, and the adjusted original model is used as Speech synthesis model.

In a possible embodiment, it also includes:

The superimposed feature extraction module is used to extract part of the superimposed audio features from all the superimposed audio features as the filtered audio features;

The target superimposed audio feature is the filtered audio feature matched with the pronunciation feature.

In a possible embodiment, the sample audio segment is initially the audio segment of the first sample person;

The acquisition module 607 acquires new sample audio clips, including:

If the second convergence condition is not reached, acquire a new sample audio clip from the first sample data set, the first sample data set includes the audio clip of the first sample person;

If the second convergence condition is reached, a new sample audio clip and new sample text are obtained from a second sample data set, where the second sample data set includes audio clips of a plurality of sample persons.

In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

It should be noted that the sample audio clips in this embodiment come from public data sets, such as LJSpeech (a public data set) and VCTK (a public data set).

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

FIG. 7 shows a schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 7 , the device 700 includes a computing unit 701 that can be executed according to a computer program stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage unit 708 Various appropriate actions and handling. In the RAM 703, various programs and data necessary for the operation of the device 700 can also be stored. The computing unit 701 , the ROM 702 , and the RAM 703 are connected to each other through a bus 704 . An input/output (I/O) interface 705 is also connected to bus 704 .

Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc. ; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 701 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as a style transfer synthesis method or a training method of a speech synthesis model. For example, in some embodiments, a style transfer synthesis method or a training method of a speech synthesis model may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 700 via ROM 702 and/or communication unit 709 . When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the style transfer synthesis method or the training method of the speech synthesis model described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a style transfer synthesis method or a training method of a speech synthesis model by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), complex programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.

Claims (17)

1. A style migration synthesis method, comprising:

inputting a target text and a target audio clip with a target voice style into a voice synthesis model obtained by training a sample text and a sample audio clip in advance;

extracting a sub-model through the style of the voice synthesis model, and superposing a coarse-grained audio feature for representing the target audio segment and a fine-grained audio feature for representing the audio unit aiming at each audio unit in the target audio segment to obtain a superposed audio feature of the audio unit;

extracting the pronunciation characteristics of each pronunciation unit in the target text through the content coding sub-model of the voice synthesis model;

fusing the pronunciation characteristics and target superposition audio characteristics of the pronunciation units aiming at each pronunciation unit in the target text through the content style cross attention submodel of the voice synthesis model to obtain the fusion characteristics of the pronunciation units, wherein the target superposition audio characteristics are superposition audio characteristics matched with the pronunciation characteristics;

and synthesizing an audio segment which has the target voice style and has voice content as the target text according to the fusion characteristics of each pronunciation unit in the target text through the sound spectrum decoding submodel of the voice synthesis model.

2. The method of claim 1, wherein the obtaining, by the style extraction submodel of the speech synthesis model, for each audio unit in the target audio segment, a superimposed audio feature of the audio unit by superimposing a coarse-grained audio feature used for characterizing the target audio segment and a fine-grained audio feature used for characterizing the audio unit comprises:

extracting the average audio features of all audio frames in the target audio clip as coarse-grained audio features through a style extraction module of the speech synthesis model;

extracting, by the style extraction module, average audio features of all audio frames in the audio unit as fine-grained audio features of the audio unit for each audio unit in the target audio clip;

and adding the fine-grained audio features and the coarse-grained audio features of the audio units to each audio unit in the target audio segment through the style extraction module to obtain the superimposed audio features of the audio units.

3. The method according to claim 1, wherein the fusing the pronunciation features of the pronunciation units and the target superimposed audio features for each pronunciation unit in the target text by the content style cross attention submodel of the speech synthesis model to obtain the fused features of the pronunciation units comprises:

inputting the pronunciation characteristics of each pronunciation unit in the target text into a self-attention sub-network of a content style cross-attention sub-model in the speech synthesis model to obtain adjusted pronunciation characteristics output by the sub-attention sub-network;

and fusing the adjusted pronunciation characteristics and the target superposition audio characteristics of the pronunciation units aiming at each pronunciation unit in the target text through the cross attention sub-network of the content cross sub-model to obtain the fusion characteristics of the pronunciation units, wherein the target superposition audio characteristics are the superposition audio characteristics matched with the adjusted pronunciation characteristics.

4. The method according to claim 1, wherein the synthesizing of the audio segment having the target speech style and speech content of the target text according to the fusion feature of each pronunciation unit in the target text by the sonographic decoding submodel of the speech synthesis model comprises:

inputting the fusion features and the coarse-grained audio features of each pronunciation unit in the target text into a sound spectrum decoding sub-model of the speech synthesis model to obtain sound spectrum features output by the sound spectrum decoding sub-network;

and converting the sound spectrum characteristics into an audio fragment which has the target voice style and the voice content of which is the target text.

5. A method of training a speech synthesis model, comprising:

inputting a sample audio clip and a sample text into an original model, wherein the sample text is the voice content of the sample audio clip;

superposing, by the original model, for each audio unit in the sample audio clip, a coarse-grained audio feature used for characterizing the sample audio clip and a fine-grained audio feature used for characterizing the audio unit to obtain a superposed audio feature of the audio unit;

extracting pronunciation characteristics of each pronunciation unit in the sample text through the original model;

fusing the pronunciation characteristics of the pronunciation units and target superposition audio characteristics aiming at each pronunciation unit in the sample text through the original model to obtain the fusion characteristics of the pronunciation units, wherein the target superposition audio characteristics are superposition audio characteristics matched with the pronunciation characteristics;

converting the fusion features of each pronunciation unit in the sample text into prediction sound spectrum features through the original model;

adjusting model parameters of the original model according to the difference between the predicted audio spectrum characteristics and the real audio spectrum characteristics of the sample audio frequency fragment;

and acquiring a new sample audio clip and a new sample text, returning to execute the step of inputting the sample audio clip and the sample text into the original model until a first convergence condition is reached, and taking the adjusted original model as a speech synthesis model.

6. The method of claim 5, further comprising:

extracting part of the superposed audio features from all the superposed audio features to serve as the screened audio features;

the target superimposed audio features are screened audio features matched with the pronunciation features.

7. The method of claim 6, wherein the sample audio clip is initially an audio clip of a first sample person;

the obtaining of the new sample audio piece comprises:

if the second convergence condition is not met, acquiring a new sample audio clip from a first sample data set, wherein the first sample data set comprises the audio clip of the first sample person;

and if the second convergence condition is reached, acquiring a new sample audio clip and a new sample text from a second sample data set, wherein the second sample data set comprises audio clips of a plurality of sample persons.

8. A style migration synthesis apparatus comprising:

the first input module is used for inputting the target text and the target audio clip with the target voice style into a voice synthesis model which is obtained by training a sample text and a sample audio clip in advance;

the style extraction module is used for extracting a sub-model through the style of the voice synthesis model, and for each audio unit in the target audio segment, overlapping a coarse-grained audio feature used for representing the target audio segment and a fine-grained audio feature used for representing the audio unit to obtain an overlapped audio feature of the audio unit;

the content coding module is used for extracting the pronunciation characteristics of each pronunciation unit in the target text through a content coding sub-model of the voice synthesis model;

the content style cross attention module is used for fusing pronunciation characteristics of the pronunciation units and target superposed audio characteristics aiming at each pronunciation unit in the target text through a content style cross attention submodel of the voice synthesis model to obtain the fused characteristics of the pronunciation units, wherein the target superposed audio characteristics are superposed audio characteristics matched with the pronunciation characteristics;

and the sound spectrum decoding module is used for synthesizing an audio segment which has the target voice style and has the voice content of the target text according to the fusion characteristics of each pronunciation unit in the target text through a sound spectrum decoding sub-model of the voice synthesis model.

9. The apparatus of claim 8, wherein the style extraction module, via a style extraction submodel of the speech synthesis model, for each audio unit in the target audio segment, superimposes a coarse-grained audio feature for characterizing the target audio segment and a fine-grained audio feature for characterizing the audio unit to obtain a superimposed audio feature of the audio unit, includes:

extracting the average audio features of all audio frames in the target audio clip as coarse-grained audio features through a style extraction module of the speech synthesis model;

extracting, by the style extraction module, average audio features of all audio frames in the audio unit as fine-grained audio features of the audio unit for each audio unit in the target audio clip;

and adding the fine-grained audio features and the coarse-grained audio features of the audio units to each audio unit in the target audio segment through the style extraction module to obtain the superimposed audio features of the audio units.

10. The apparatus of claim 8, wherein the content style cross attention module fuses, for each pronunciation unit in a target text, a pronunciation feature of the pronunciation unit and a target overlay audio feature to obtain a fused feature of the pronunciation unit through a content style cross attention submodel of the speech synthesis model, comprising:

inputting the pronunciation characteristics of each pronunciation unit in the target text into a self-attention sub-network of a content style cross-attention sub-model in the speech synthesis model to obtain adjusted pronunciation characteristics output by the sub-attention sub-network;

and fusing the adjusted pronunciation characteristics and the target superposition audio characteristics of the pronunciation units aiming at each pronunciation unit in the target text through the cross attention sub-network of the content cross sub-model to obtain the fusion characteristics of the pronunciation units, wherein the target superposition audio characteristics are the superposition audio characteristics matched with the adjusted pronunciation characteristics.

11. The apparatus according to claim 8, wherein the voice spectrum decoding module synthesizes an audio segment having the target speech style and speech content of the target text according to the fusion feature of each pronunciation unit in the target text by a voice spectrum decoding submodel of the speech synthesis model, including:

inputting the fusion features and the coarse-grained audio features of each pronunciation unit in the target text into a sound spectrum decoding sub-model of the speech synthesis model to obtain sound spectrum features output by the sound spectrum decoding sub-network;

and converting the sound spectrum characteristics into an audio fragment which has the target voice style and the voice content of which is the target text.

12. An apparatus for training a speech synthesis model, comprising:

the second input module is used for inputting a sample audio clip and a sample text into the original model, wherein the sample text is the voice content of the sample audio clip;

a first original module, configured to superimpose, by using the original model, for each audio unit in the sample audio clip, a coarse-grained audio feature used for characterizing the sample audio clip and a fine-grained audio feature used for characterizing the audio unit, so as to obtain a superimposed audio feature of the audio unit;

the second original module is used for extracting the pronunciation characteristics of each pronunciation unit in the sample text through the original model;

a third original module, configured to fuse, by using the original model, a pronunciation feature of the pronunciation unit and a target superimposed audio feature for each pronunciation unit in the sample text to obtain a fused feature of the pronunciation unit, where the target superimposed audio feature is a superimposed audio feature matched with the pronunciation feature;

a fourth original module, configured to convert, through the original model, the fusion feature of each pronunciation unit in the sample text into a predicted sound spectrum feature;

a parameter adjusting module, configured to adjust a model parameter of the original model according to a difference between the predicted audio spectrum feature and a true audio spectrum feature of the sample audio segment;

and the obtaining module is used for obtaining a new sample audio clip and a new sample text, returning to execute the step of inputting the sample audio clip and the sample text into the original model until a first convergence condition is reached, and taking the adjusted original model as a speech synthesis model.

13. The apparatus of claim 12, further comprising:

the superposition characteristic extraction module is used for extracting partial superposition audio characteristics from all the superposition audio characteristics to serve as the screened audio characteristics;

the target superimposed audio features are screened audio features matched with the pronunciation features.

14. The apparatus of claim 12, wherein the sample audio clip is initially an audio clip of a first sample person;

the obtaining module obtains a new sample audio clip, including:

if the second convergence condition is not met, acquiring a new sample audio clip from a first sample data set, wherein the first sample data set comprises the audio clip of the first sample person;

and if the second convergence condition is reached, acquiring a new sample audio clip and a new sample text from a second sample data set, wherein the second sample data set comprises audio clips of a plurality of sample persons.

15. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4 or 5-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any of claims 1-4 or 5-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-4 or 5-7.

Images