CN117854473A - Zero sample speech synthesis method based on local association information - Google Patents

Abstract

The invention discloses a zero-sample speech synthesis method based on local reference information, and relates to the technical field of speech signal processing. The zero-sample speech synthesis method based on local reference information includes the following steps: S1, preprocessing text and speech data; S2, building a basic network framework; S3, designing a reference information introduction scheme for the target speaker, and building a zero-sample speech synthesis model according to the designed scheme; S4, training the model using the deep learning Pytorch framework; S5, inputting the target text and the reference speech of the target speaker into the model to obtain the result of speech synthesis; the invention uses the proposed feature matching and recombination method, as well as the local speaker encoder module, to improve the performance of zero-sample speech synthesis to a new level.

Description

Zero-shot speech synthesis method based on local correlation information

Technical Field

The present invention relates to the technical field of speech signal processing, and in particular to a zero-sample speech synthesis method based on local correlation information.

Background technique

Zero-shot speech synthesis aims to use the speech of a few target speakers as a reference to synthesize natural and fluent speech with the speaker's timbre from text. It can be applied to many practical scenarios such as audio novels and voice broadcasting. Zero-shot speech synthesis faces many challenges such as poor synthesis generalization and unnatural rhythm synthesis. In recent years, zero-shot speech synthesis has shifted from traditional mathematical models to deep learning methods based on neural networks.

Most of these methods adopt a global reference introduction method, that is, the reference spectrogram is extracted and converted into a one-dimensional vector so that it contains the basic timbre information of the target speaker, and then the vector is used to help the model learn the sound information of the target speaker; however, when introducing the vector, the introduced vector is consistent for each time frame of the intermediate feature, which ignores the local content information of the target sentence; recently, in the task of speech synthesis, some researchers have proposed to use the content information of the target text to find reference sound features with higher information relevance; compared with the global method, this method can introduce richer useful information for different local parts, thereby making the speech synthesis result more natural.

On the other hand, the zero-sample speech synthesis system can usually be modified into a speaker conversion system. The goal of speaker conversion is to replace the timbre of the source speaker with the timbre of the target speaker while keeping the semantic content unchanged. It is used in scenarios such as digital humans and virtual voice change. Recently, researchers have proposed converting the acoustic model of zero-sample speech synthesis into a speaker conversion model, and using the adaptability of zero-sample speech synthesis to the target speaker's voice to help complete the timbre conversion between different speakers. Based on the above content, the present invention proposes a zero-sample speech synthesis method based on local correlation information.

Summary of the invention

The purpose of the present invention is to propose a zero-sample speech synthesis method based on local correlation information to solve the problems raised in the background technology, so as to generate natural and high-quality speech from text with a small number of samples.

In order to achieve the above object, the present invention adopts the following technical solutions:

The zero-sample speech synthesis method based on local correlation information comprises the following steps:

S1. Preprocessing text and speech data: Given the original text and speech data (C _raw , S _raw ), perform word-to-phoneme processing on C _raw to obtain a phoneme sequence c; perform short-time Fourier transform (STFT) on S _raw and randomly select segments to obtain a reference short-time Fourier transform (STFT) spectrum r and a true short-time Fourier transform (STFT) spectrum x, and divide the data into a training set, a validation set, a test set, and an independent reference speech set;

S2. Build a basic network framework: Design a Transformer-based prior text encoder, a stream-based multi-layer acoustic decoder, a dilated convolution-based posterior encoder, and a generative adversarial network-based vocoder. Combine the above structures into a joint adversarial training conditional variational autoencoder network framework.

S3, design scheme, build model: Based on the information correlation between the target sentence and the reference speech, combined with the preprocessing method and basic network framework described in S1-S2, design a reference information introduction scheme, and build a zero-sample speech synthesis model based on the designed scheme. The scheme specifically includes the following contents:

① Feature matching and recombination: The real and reference short-time Fourier transform (STFT) spectrum data pairs (x, r) obtained in S1 are input into the posterior encoder to obtain the target hidden feature z and the reference hidden feature _zr respectively; then, in each affine coupling layer, the matching and recombination method is performed on z and _zr to obtain the reference feature with higher information correlation

② Local speaker encoder: The reorganized features Input it into the basic voiceprint vector generation module to obtain the basic voiceprint vector e; at the same time, Input into the gated convolution module for information filtering; finally, the filtered features and e are input into the local attention module to obtain the local speaker embedding/>

③ Training and reasoning process design: The training process is performed according to the adversarial training and variational reasoning described in S2; the difference between the reasoning process and the training process is that the affine coupling layer adopts a reversible form;

④ Loss function module design: The zero-shot speech synthesis model is jointly optimized through reconstruction loss and KL divergence loss in the variational inference process, adversarial loss and feature matching loss in the adversarial training process, and speaker identity loss in the basic voiceprint vector generation process;

⑤ Design of speaker conversion method: Based on the zero-shot speech synthesis model described in S3, modifications are made to retain the reference information introduction scheme, the posterior encoder, the K-layer affine coupling layer, and the vocoder to achieve voice conversion between different speakers;

S4, training model: Use the deep learning Pytorch framework to train the model, traverse all the preprocessed text and voice data in S1, set the initial learning rate to 2e-4, and exponentially decay at a rate of 0.999. After 500k iterations of training, save the model with the best performance on the validation set;

S5. Output result: Input the test set text data and reference set speech data preprocessed in S1 into the stable model to obtain the result of zero-sample speech synthesis.

Preferably, the scheme ① further includes the following contents:

A1. In the same affine coupling layer, the cosine similarity of the target hidden feature z and the reference hidden feature _zr is calculated at the frame level. That is, for each frame z _(t) in z, the similarity between this frame and all frames in _zr is calculated, and the frame with the largest cosine similarity is selected as the matching index of z _(t) . The function of this process is expressed as:

In the formula, <·,·> represents the inner product, ψ[·] represents the normalization of the vector;

A2. According to the index value obtained by matching To reorganize/> Generate a reconstructed reference signature aligned with z in time domain/> The recombinant reference features It is the reference frame level feature combination that has the highest local correlation with the current target intermediate feature z.

Preferably, the solution ② specifically includes the following contents:

B1. The basic voiceprint vector generation module first reconstructs the reference features through a three-layer long short-term memory network (LSTM) Mapped to the hidden vector of the last layer of LSTM, and then mapped to the speaker's basic voiceprint vector e through a linear layer;

B2. The gated convolution module first reconstructs the reference features through an input gate consisting of a convolution module and a gated activation unit. Information filtering is performed, and then the information passes through the forget gate controlled by the global basic voiceprint vector g to further control redundant information;

B3, the local attention module uses the features filtered in B2 to modulate the basic voiceprint vector e; first, the filtered features are divided into different frame groups along the time dimension, and then based on the attention mechanism, different frame groups are fused with e respectively to generate a local speaker embedding vector e _j that captures both the basic voiceprint information of the target speaker and the local association information represented by the corresponding frame group j; the attention mechanism uses e as a query, and uses e as a key and value; the frame group j is not used as a query, but only as a key and value; the function of the above process is expressed as:

Where q ₀ , k ₀ and v ₀ represent the query, key and value obtained from e respectively; k _i , _vi represent the key and value obtained from the frame group respectively, where i = {1, 2, 3}.

Preferably, the solution ③ specifically includes the following contents:

The goal of the training process is to maximize the lower bound of the evidence of the log-likelihood, and its function is expressed as:

Wherein, the log-likelihood function logP _θ (x│z,r) represents the process of reconstructing the waveform by the vocoder; logQ _θ (z│x,r)-logP _θ (x│z,r) represents the KL divergence between the prior distribution and the approximate posterior distribution, which actually corresponds to the process of establishing a reversible transformation between the text hidden features and the audio hidden features by K layers of affine coupling layers;

During the inference process, the K-layer affine coupling layer adopts a reversible form, takes the text prior as input, and outputs the acoustic hidden feature z.

Preferably, the scheme ④ further includes the following contents:

C1. Project the true short-time Fourier transform (STFT) spectrum x onto the Mel scale, denoted as x _mel ; for the waveform y predicted by the vocoder, also use the same parameters to convert it into the Mel spectrum y _mel ; then, use the L ₁ loss between x _mel and y _mel as the reconstruction loss, and its function is expressed as follows:

C2. Use variational inference to make the vocoder restore x better and make the approximate posterior distribution Q _θ (z│x,r) close to the prior distribution P _θ (x│z,r); define KL divergence as:

z～Q _φ (z|x,r)＝N(z；μ _φ (x,r),σ _φ (x，r))

Where [μ _φ (x,r), σ _φ (x,r)] represents the statistics of the posterior distribution;

C3. Use the least squares loss function as the adversarial training loss function to avoid the gradient disappearance phenomenon. The specific function is expressed as follows:

C4, the vocoder uses a multi-scale and multi-cycle discriminator and a multi-layer feature matching loss function to improve the stability of adversarial training. The specific function is expressed as follows:

Where T represents the number of layers of the discriminator, D ^l and N _l represent the feature map of the lth layer and the number of features in the feature map respectively;

The basic idea of the speaker identity loss function is to shorten the distance between the voiceprint vectors of the same speaker and increase the distance between the voiceprint vectors of different speakers. The specific function is expressed as follows:

Where i(j) represents the sample index in the batch, m represents the layer index of the affine coupling layer, and e represent the reference features of the recombinant /> and the voiceprint vector generated by the target feature z, ψ(·) represents the normalization of the vector, and <·,·> represents the inner product;/> represents the similarity between the voiceprint vectors of two samples from the same speaker, and S represents the similarity between the voiceprint vectors of two samples from different speakers; positive real numbers α and β are used as S and / > The multiplier of .

Preferably, the scheme ⑤ further includes the following contents:

The purpose of speaker switching is to convert the source timbre to the target timbre while maintaining the same semantic content. Specifically, it refers to:

D1. With the help of the source speaker reference speech s, the source speaker speech x is converted into an intermediate variable z′ that is independent of the speaker through the posterior encoder Q _φ and the forward affine coupling layer f _dec . Its function is expressed as follows:

z～Q _φ (z|x,s)

z′＝f _dec (z|s)

D2, with the help of the target speaker's reference speech t, after the inverse affine coupling layer The vocoder V converts the speaker-independent intermediate variable z′ into the waveform y of the target speaker, and its function is expressed as follows:

Where V represents the vocoder; represents the inverse affine coupling layer; z′ represents the speaker-independent intermediate variable; t represents the target speaker reference speech.

Compared with the prior art, the present invention provides an event camera image reconstruction method based on a neural network, which has the following beneficial effects:

(1) The present invention proposes a zero-sample speech synthesis method based on local correlation information. Different from the previous method of introducing global target reference information, the present invention additionally considers the information correlation between the target sentence and the reference speech at the phoneme level, and can introduce richer information for different parts, thereby making the synthesis result more natural.

(2) The present invention proposes a frame-level feature matching and reorganization method, which can obtain reference features with higher information correlation; the present invention also proposes a local speaker encoder, which can enable the reference features to better express basic voiceprint features and local correlation information.

(3) The experiments conducted in the embodiments of the present invention show that the proposed method is superior to the current mainstream zero-sample speech synthesis method; the research and exploration of the present invention can inspire more research on utilizing the local correlation information between the target sentence and the reference speech.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is an overall framework diagram of the zero-sample speech synthesis method based on local correlation information proposed by the present invention;

FIG2 is a diagram of the matching and reorganization structure proposed in Example 1 of the present invention;

FIG3 is a structural diagram of a local speaker encoder proposed in Embodiment 1 of the present invention;

FIG. 4 is a diagram of a speaker conversion framework proposed in Embodiment 1 of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

Embodiment 1:

Referring to FIG. 1 , the present invention proposes a zero-sample speech synthesis method based on local correlation information, comprising the following steps:

S1. Preprocessing text and speech data: Given the original text and speech data (C _raw , S _raw ), perform word-to-phoneme processing on C _raw to obtain a phoneme sequence c; perform short-time Fourier transform (STFT) on S _raw and randomly select segments to obtain a reference STFT spectrum r and a true STFT spectrum x. All data are divided into a training set, a validation set, a test set, and an independent reference speech set;

S2. Build a basic network framework: Design a Transformer-based prior text encoder, a stream-based multi-layer acoustic decoder, a dilated convolution-based posterior encoder, and a generative adversarial network-based vocoder. The above structures together form a joint adversarial training conditional variational autoencoder network framework;

S3, design scheme, build model: Based on the information correlation between the target sentence and the reference speech, the reference information introduction scheme is designed in combination with the preprocessing methods and basic network framework described in S1-S2. The scheme specifically includes the following contents:

S3.1. Matching and recombination design

As shown in FIG2 , for local association information, the present invention uses frame-level cosine similarity to measure and matches the target feature with the reference feature. Specifically, in the same affine coupling layer, the frame-level cosine similarity calculation is performed on the target hidden feature z and the reference hidden feature z _r , that is, for each frame z _(t) in z, the similarity of this frame with all frames in z _r is calculated, and the frame with the largest cosine similarity is selected as the matching index of z _(t) . The process can be expressed as:

In the formula, <·,·> represents the inner product, ψ[·] represents the normalization of the vector;

Then, according to the matched index value To reorganize/> Generate a reconstructed reference signature aligned with z in time domain/> This feature is the combination of reference frame-level features that has the highest local correlation with the current target intermediate feature z.

S3.2. Local Speaker Encoder Design

After matching and reorganization, reference features with higher relevance to the current target sentence information are obtained. The purpose of the local speaker encoder is to convert/> It is converted into a representation that can capture the basic acoustic features of the target speaker and effectively fuses the local correlation information with the basic voiceprint representation. As shown in Figure 3, it consists of three parts: the target speaker basic voiceprint vector generation module (right branch), the gated convolution module (left branch) and the final local attention module.

The basic voiceprint vector generation module first reconstructs the reference features through a three-layer long short-term memory network (LSTM) Mapped to the hidden vector of the last layer of LSTM, and then mapped to the basic voiceprint vector e of the speaker through the linear layer; in order to make e better capture the basic voiceprint features of the target speaker, the speaker identity verification loss is introduced to constrain it. The loss function is based on a basic idea, that is, to shorten the distance between the voiceprint vectors of the same speaker and to increase the distance between the voiceprint vectors of different speakers. It can be expressed as:

Where i(j) represents the sample index in the batch, m represents the layer index of the affine coupling layer, and e represent the reference features of the recombinant /> The voiceprint vector generated by the target feature z, ψ(·) represents the normalization of the vector, and <·,·> represents the inner product. /> represents the similarity between the voiceprint vectors of two samples from the same speaker, while S represents the similarity between the voiceprint vectors of two samples from different speakers. The positive real numbers α and β are used as S and / > The multiplier of .

Under different emotional conditions, even the pronunciation of the same phoneme will be different. This personalization will bring redundant interference to the training. In order to minimize the impact caused by the above redundant information, the present invention proposes a gated convolution module. The gated convolution module consists of an input gate and a forget gate. The input gate includes a convolution module and a gated activation unit. The numerical range of the Sigmoid function output is [0,1], which can play a gating role. By multiplying the output of Tanh and Sigmoid functions, the gated activation unit can achieve selective retention and filtering of input information. The forget gate includes a convolution module and a Sigmoid gate unit, and the global speaker embedding vector g is used to control the Sigmoid gate unit. g is obtained from the pre-trained speaker encoder and has no semantic relevance, but represents the basic acoustic features of the target speaker. After g is processed by the linear layer and the Sigmoid function, the modulation coefficient of the forget gate is generated to control the retention and forgetting of the input information. All operations are performed in the frequency domain dimension. The output of the gated convolution module is the reference feature after information filtering.

The role of the local speaker encoder is to combine the basic voiceprint vector e with the reference feature after information filtering Effective fusion. First, the filtered features are integrated along the time dimension. It is divided into different frame groups, and then based on the attention mechanism, different frame groups are fused with e respectively to generate a local speaker embedding vector e _j that captures both the basic voiceprint information of the target speaker and the local correlation information represented by the corresponding frame group j. The attention mechanism uses e as a query and e as a key and value; while frame group j is not used as a query, but only as a key and value. The process can be expressed as:

Where q ₀ , k ₀ and v ₀ represent the query, key and value obtained from e, respectively, and k _i , _vi (i = {1, 2, 3}) represent the key and value obtained from the frame group, respectively. In order to achieve the alignment of the time dimension between e _j and the frame group j, each e _j is replicated three times along the time dimension. Finally, these local embedding vectors are concatenated to form a local embedding vector combination The combination is aligned with z in the temporal dimension. In other words, each local frame group in z receives a unique local speaker embedding e _j , while the three frames in the same frame group share the same local speaker embedding.

S3.3 Design of speaker conversion method based on local correlation information

As shown in Figure 4, s and t represent the reference speech of the source speaker and the target speaker respectively. and/> represent the hidden reference features and recombined reference features of the source speaker in the kth affine coupling layer, respectively./> and/> They represent the hidden reference features and reconstructed reference features of the target speaker in the kth inverse affine coupling layer. x and y represent the source speaker’s speech and the target speaker’s speech, respectively.

The system first converts the source speaker speech x into a speaker-independent intermediate variable z′ with the help of the source speaker reference speech s through the posterior encoder Q _φ and the forward affine coupling layer f _dec , as shown below:

z～Q _φ (z|x,s)

z′＝f _dec (z|s)

Then, the speaker conversion system is assisted by the target speaker reference speech t through the inverse affine coupling layer and the vocoder V, which converts z′ into the waveform y of the target speaker, namely:

S3.4. Loss function design

A total of 5 different losses are used during network training, namely: reconstruction loss and KL divergence loss associated with the conditional variational autoencoder, generative adversarial loss and multi-layer feature matching loss associated with adversarial training, and speaker authentication loss applied in the local speaker encoder. The details are as follows:

The true STFT spectrum x is projected onto the Mel scale, recorded as x _mel ; in addition, for the waveform y predicted by the vocoder, it is also converted to the Mel spectrum y _mel using the same parameters. Then, the L ₁ loss between x _mel and y _mel is used as the reconstruction loss. As shown below:

Variational reasoning expects the vocoder to restore x better on the one hand, and on the other hand, expects the approximate posterior distribution Q _θ (z|x, r) to be as close as possible to the prior distribution P _θ (x|z, r). Therefore, the KL divergence is defined as:

z～Q _φ (z|x, r)＝N(z；μ _φ (x, r), σ _φ (x, r))

Where [μ _φ (x, r), σ _φ (x, r)] represents the statistics of the posterior distribution;

In order to better avoid the gradient vanishing phenomenon, the adversarial training loss function uses the least squares loss function as shown below:

The vocoder uses a multi-scale and multi-cycle discriminator, so a multi-layer feature matching loss function is used to improve the stability of adversarial training, as shown below:

Where T represents the number of layers of the discriminator, D ^l and N _l represent the feature map of the lth layer and the number of features in the feature map respectively;

The basic idea of the speaker identity loss function is to bring the distance between the voiceprint vectors of the same speaker closer and the distance between the voiceprint vectors of different speakers farther apart, as shown below:

Where i(j) represents the sample index in the batch, m represents the layer index of the affine coupling layer, and e represent the reference features of the recombinant /> The voiceprint vector generated by the target feature z, ψ(·) represents the normalization of the vector, and <·,·> represents the inner product. /> represents the similarity between the voiceprint vectors of two samples from the same speaker, while S represents the similarity between the voiceprint vectors of two samples from different speakers. The positive real numbers α and β are used as S and / > The multiplier of .

Based on the above content, a zero-sample speech synthesis model is built according to the above designed solution.

S4, training model: Use the deep learning Pytorch framework to train the model, traverse all the preprocessed text and voice data in S1, the initial learning rate is 2e-4, and the exponential decay rate is 0.999. After 500k iterations of training, save the model with the best performance on the validation set;

S5. Output result: Input the test set text data and reference set speech data preprocessed in S1 into the stable model to obtain the result of zero-sample speech synthesis.

Embodiment 2:

Based on Example 1 but with some differences:

This example uses three different lengths of reference speech for evaluation, namely 5 seconds, 20 seconds, and 40 seconds. At the same time, three advanced comparison methods trained on the VCTK dataset are selected, including Attentron ZS, SC-GlowTTS, and YourTTS.

Among them, Attentron ZS is an autoregressive zero-shot speech synthesis method based on attention mechanism and recurrent neural network. It uses coarse and fine-grained encoders to extract basic sound information and detailed style information respectively to improve the quality of synthesized speech. SC-GlowTTS is a non-autoregressive zero-shot speech synthesis method based on stream generation model. It introduces multi-layer dilated convolution modules into the standardized stream structure to improve the ability of stream model to extract features, thereby improving the quality of synthesized speech. YourTTS is a non-autoregressive zero-shot speech synthesis method based on conditional variational autoencoder. Based on the global reference information introduction method, YourTTS introduces multiple layers in the posterior encoder and stream decoder to help improve the quality of synthesized speech.

For a fairer comparison, this embodiment provides the results generated by the YourTTS model using 40 seconds of reference speech (YourTTS-40s). Please refer to Table 1 for specific results.

Table 1 MOS and Sim-MOS evaluation under SECS and 95% confidence interval

As shown in Table 1, the table shows the evaluation results on the SECS, MOS, and Sim-MOS indicators. SECS is an indicator for evaluating the similarity of sound timbre, while MOS and Sim-MOS are indicators for evaluating the naturalness of sound. The higher the three indicators, the better. The model with a reference duration of 40 seconds achieved the best results in both SECS and Sim-MOS evaluations, while the model with a reference duration of 20 seconds performed best in MOS scores and outperformed previous zero-sample speech synthesis methods in all aspects. However, the model with a reference duration of 5 seconds did not perform as well as the YourTTS method, which means that the performance of the proposed method depends on the duration of the reference speech. A reference duration that is too short will lead to a decline in the performance of the model due to the inability to achieve high coverage of the phoneme set.

In addition, this embodiment conducts speaker conversion experiments on the VCTK dataset, and selects 8 speakers (4 males/4 females) from the VCTK test subset. The present invention is compared with two advanced speaker conversion models transformed from zero-shot speech synthesis systems. Considering the general differences in voice characteristics between male and female speakers, the present invention provides evaluation results for cross-gender speaker conversion and same-gender speaker conversion respectively. Please refer to Table 2 for the evaluation results.

Table 2 Speaker conversion evaluation results under 95% confidence interval

As shown in Table 2, the speaker conversion method proposed in the present invention achieved the highest MOS score and Sim-MOS score in same-gender conversion, which proves that the method based on local correlation information has better adaptability to the voice characteristics of different speakers. The experimental results of cross-gender conversion are similar to those of speaker conversion based on YourTTS, and worse than those of same-gender conversion, which shows that the ability of the speaker conversion model to process two significantly different voice characteristics needs to be further improved.

The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims (6)

1. The zero sample voice synthesis method based on the local association information is characterized by comprising the following steps:

s1, preprocessing text and voice data: given the original text C _raw And speech data S _raw For C _raw Word-to-phoneme processing is carried out to obtain a phoneme sequence c; for S _raw Performing short-time Fourier transform and randomly selecting fragments to obtain a reference short-time Fourier transform spectrum r and a real short-time Fourier transform spectrum x, and dividing data into a training set, a verification set, a test set and an independent reference voice set;

s2, constructing a basic network frame: designing a priori text encoder based on a transducer, a multi-layer acoustic decoder based on a stream, a posterior encoder based on expansion convolution and a vocoder based on generation of an countermeasure network, and combining the structures to form a condition variation self-encoder network framework for combined countermeasure training;

s3, designing a scheme and building a model: based on the information relevance between the target sentence and the reference voice, a reference information introduction scheme is designed by combining the preprocessing methods and the basic network frames in S1-S2, and a zero sample voice synthesis model is built according to the designed scheme, wherein the scheme specifically comprises the following contents:

(1) feature matching and recombination: inputting the real and reference short-time Fourier transform spectrum data pair (x, r) obtained in the step S1 into a posterior encoder to respectively obtain a target hidden characteristic z and a reference hidden characteristic z _r The method comprises the steps of carrying out a first treatment on the surface of the Subsequently, in each affine coupling layer, pairs z and z _r Matching and reorganizing methods are carried out to obtain reference features with higher information association degree

(2) Local speaker encoder: features after recombinationInput to the baseThe voiceprint vector generation module obtains a basic voiceprint vector e; at the same time will->Inputting the information into a gate convolution module for information filtering; finally, the filtered characteristics and e are input into a local attention module to obtain the embedding of the local speaker>

(3) Training and reasoning process design: the training process is carried out according to the countermeasure training and the variational reasoning in the step S2; the reasoning process is different from the training process in that the affine coupling layer adopts a reversible form;

(4) and (3) designing a loss function module: the zero sample speech synthesis model is subjected to joint optimization through reconstruction loss and KL divergence loss in a variational reasoning process, antagonism loss and feature matching loss in an antagonism training process, and speaker identity loss in a basic voiceprint vector generation process;

(5) the speaker conversion method is designed: making modification based on the zero sample speech synthesis model in the step S3, and reserving a reference information introduction scheme, a posterior encoder, a K-layer affine coupling layer and a vocoder to realize sound conversion among different speakers;

s4, training a model: traversing all the preprocessed text and voice data in the S1 by using a deep learning Pytorch frame training model, setting the initial learning rate to be 2e-4, carrying out exponential decay according to the rate of 0.999, carrying out 500k iterative training, and storing the model with the best performance on a verification set;

s5, outputting a result: and (3) inputting the test set text data and the reference set voice data preprocessed in the step (S1) into a stable model to obtain a zero-sample voice synthesis result.

2. The zero-sample speech synthesis method based on local correlation information according to claim 1, wherein the scheme (1) further comprises the following:

a1 is the same asWithin an affine coupling layer, hidden from object z and reference z _r Performing a frame-level cosine similarity calculation, i.e. for each frame z in z _(t) Calculate the frame and z _r The similarity of all frames in the list, selecting one frame with the maximum cosine similarity as z _(t) Is expressed as a function of:

wherein < ·, · > represents the inner product, ψ [ · ] represents the normalization of the vector;

a2, obtaining index values according to the matchingRecombinant->Generating a reorganized reference feature aligned with z in the time domainSaid recombinant reference feature->Is the reference frame level feature combination with the highest local relevance to the current target intermediate feature z.

3. The zero-sample speech synthesis method based on local correlation information according to claim 1, wherein the scheme (2) specifically comprises the following:

b1, a basic voiceprint vector generation module firstly recombines reference features through a three-layer long short-time memory networkThe hidden vector mapped to the last layer LSTM is then hidden by the linear layerThe Tibetan vector is mapped to a basic voiceprint vector e of the speaker;

b2, the gate convolution module first recombines the reference characteristics through an input gate pair consisting of the convolution module and a gate control activation unitFiltering information, and then realizing further control of redundant information through a forgetting gate controlled by the global basic voiceprint vector g;

b3, the local attention module modulates the basic voiceprint vector e by using the filtered characteristics in the B2; firstly, segmenting the filtered characteristics into different frame groups along a time dimension, then respectively fusing the different frame groups with e based on an attention mechanism, and generating a local speaker embedded vector e which simultaneously captures basic voiceprint information of a target speaker and local associated information represented by a corresponding frame group j _j The method comprises the steps of carrying out a first treatment on the surface of the The attention mechanism uses e as a query while e is a key and a value; the frame group j is not used as a query, but is only used as a key and a value; the function of the above procedure is expressed as:

wherein q is ₀ 、k ₀ And v ₀ Representing the query, key and value obtained from e, respectively; k (k) _i ,v _i The key and value resulting from the frame group are represented, respectively, where i= {1,2,3}.

4. The zero-sample speech synthesis method based on local correlation information according to claim 1, wherein the scheme (3) specifically comprises the following:

the lower bound of evidence of maximized log-likelihood is targeted for the training process, the function of which is expressed as:

in the log likelihood function log P _θ (x-z, r) represents the flow of vocoder reconstructed waveforms; log Q _θ (z│x,r)-log P _θ (x-z, r) represents the KL divergence between the a priori distribution and the approximate posterior distribution, the actual corresponding K-layer affine coupling layer establishing a reversible transform flow between the text hidden features and the audio hidden features;

in the reasoning process, the K-layer affine coupling layer adopts a reversible form, takes text prior as input, and outputs acoustic hidden characteristics z.

5. The zero-sample speech synthesis method based on local correlation information according to claim 1, wherein the scheme (4) further comprises the following:

c1, projecting a real short-time Fourier transform spectrum x onto a Mel scale, and marking the real short-time Fourier transform spectrum as x _mel The method comprises the steps of carrying out a first treatment on the surface of the For the vocoder predicted waveform y, the same parameters are also used to convert it to mel spectrum y _mel The method comprises the steps of carrying out a first treatment on the surface of the Then, x is used _mel And y is _mel L in between ₁ The loss is expressed as a function of reconstruction loss as follows:

c2, using variational reasoning to make the vocoder better recover x and make the approximate posterior distribution Q _θ (z-x, r) and a priori distribution P _θ (x-z, r) phase approach; define the KL divergence as:

z～Q _φ (z|x,r)＝N(z；μ _φ (x,r),σ _φ (x,))

wherein [ mu ] is _φ (x,r),σ _φ (x,r)]Statistics representing posterior distribution;

c3, using a least squares loss function as an countermeasure training loss function to avoid gradient disappearance, wherein the specific function is as follows:

c4, vocoders use multi-scale and multi-cycle discriminators and use multi-layer feature matching loss functions to improve stability against training, the specific functions are shown below:

wherein T represents the number of layers of the discriminator, D ^l And N _l Respectively representing a feature map of the first layer and the number of features in the feature map;

the basic idea of pulling the distances of voiceprint vectors of the same speaker and the distances of voiceprint vectors of different speakers is used as a speaker identity loss function, and the specific function is expressed as follows:

where i or j represents a sample index in the batch process, m represents a layer index of the affine coupling layer,and e represents the reference feature by recombination, respectively->And a voiceprint vector generated by the target feature z, ψ (·) represents the normalization of the vector,<·,·>representing the inner product; />Representing the similarity between voiceprint vectors from two samples of the same speaker, while S represents the similarity between voiceprint vectors from two samples of different speakers; positive real numbers alpha and beta are S and +.>Is a multiplier of (2).

6. The zero-sample speech synthesis method based on local correlation information according to claim 1, wherein the scheme (5) further comprises the following:

the purpose of speaker conversion is to convert a source tone into a target tone on the premise of keeping the same semantic content, specifically:

d1, with the help of the reference speech s of the source speaker, pass through a posterior encoder Q _φ And a forward affine coupling layer f _dec The source speaker speech x is converted to a speaker independent intermediate variable z', the function of which is expressed as follows:

zQφ(z]x,s)＝faec(zs)

d2, with the help of the target speaker reference voice t, passing through the inverse affine coupling layerAnd a vocoder V converting the speaker independent intermediate variable z' into a waveform y of the target speaker, the function of which is expressed as follows:

wherein V represents a vocoder;representing a reverse affine coupling layer; z' represents an intermediate speaker independent variable; t represents the target speaking ginseng test voice.