WO2022227190A1

WO2022227190A1 - Speech synthesis method and apparatus, and electronic device and storage medium

Info

Publication number: WO2022227190A1
Application number: PCT/CN2021/097075
Authority: WO
Inventors: 孙奥兰; 王健宗; 程宁
Original assignee: 平安科技（深圳）有限公司
Priority date: 2021-04-25
Filing date: 2021-05-30
Publication date: 2022-11-03
Also published as: CN112951203A; CN112951203B

Abstract

The present application relates to the field of artificial intelligence. Disclosed is a speech synthesis method, comprising: acquiring training text, and performing speech synthesis on the training text by using a pre-trained autoregressive speech synthesis model, so as to obtain a first sound spectrum of the training text; extracting a linguistic feature from the first sound spectrum, so as to obtain a first linguistic feature; training a pre-constructed non-autoregressive speech synthesis model according to the training text, the first sound spectrum and the first linguistic feature, so as to obtain a non-autoregressive speech synthesis model, the training of which is completed; and by using the non-autoregressive speech synthesis model, the training of which is completed, performing, speech synthesis on speech text to be subjected to synthesis, so as to obtain a speech synthesis result. In addition, the present application further relates to blockchain technology. The speech synthesis result can be stored in a blockchain. Further disclosed in the present application are a speech synthesis apparatus, an electronic device and a storage medium By means of the present application, the speed and quality of speech synthesis can be improved.

Description

Speech synthesis method, device, electronic device and storage medium

This application claims the priority of the Chinese patent application with the application number 202110450578.7 and the invention titled "Speech Synthesis Method, Apparatus, Electronic Device and Storage Medium" filed with the China Patent Office on April 25, 2021, the entire contents of which are incorporated by reference in this application.

technical field

The present application relates to the field of artificial intelligence, and in particular, to a speech synthesis method, apparatus, electronic device, and computer-readable storage medium.

Background technique

Speech synthesis refers to the technology that converts text information generated by the computer itself or input from external sources into intelligible and fluent speech output. Nowadays, speech synthesis is one of the key technical links of modern speech artificial intelligence products. The speed and quality of synthesized speech are the two main evaluation indicators for judging the pros and cons of speech synthesis systems, and are also important factors that affect the user experience of speech products.

The inventor realized that there are usually two implementation methods for acoustic models in speech synthesis: autoregressive type and non-autoregressive type. When generating speech audio, the autoregressive type is based on the generated speech and can effectively learn acoustics. The temporal change of features can generate high-quality speech, but due to the nature of speech sequence generation, the reasoning speed of speech in the generation process is slow, so that the speed of speech synthesis is slow; non-autoregressive type can generate speech audio. Parallel generation of speech makes speech inference faster during generation, but ignores temporal changes in acoustic features, resulting in relatively poor synthesized speech quality.

SUMMARY OF THE INVENTION

A speech synthesis method provided by this application includes:

Acquiring training text, using a pre-trained autoregressive speech synthesis model to perform speech synthesis on the training text to obtain the first sound spectrum of the training text;

Extracting linguistic features from the first sound spectrum to obtain first linguistic features;

According to the training text, the first sound spectrum and the first linguistic feature, the pre-built non-autoregressive speech synthesis model is trained to obtain a trained non-autoregressive speech synthesis model;

Using the trained non-autoregressive speech synthesis model to perform speech synthesis on the speech text to be synthesized to obtain a speech synthesis result.

The present application also provides a speech synthesis device, the device comprising:

an acquisition module, used for acquiring training text, and using a pre-trained autoregressive speech synthesis model to perform speech synthesis on the training text to obtain the first sound spectrum of the training text;

an extraction module for extracting linguistic features from the first sound spectrum to obtain first linguistic features;

A training module for training a pre-built non-autoregressive speech synthesis model according to the training text, the first sound spectrum and the first linguistic feature, to obtain a trained non-autoregressive speech synthesis model;

The synthesis module is configured to perform speech synthesis on the speech text to be synthesized by using the non-autoregressive speech synthesis model completed by the training to obtain a speech synthesis result.

The present application also provides an electronic device, the electronic device comprising:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to implement a speech synthesis method as described below:

The present application also provides a computer-readable storage medium, where at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to implement the speech synthesis method described below :

Description of drawings

1 is a schematic flowchart of a speech synthesis method provided by an embodiment of the present application;

2 is a detailed schematic flowchart of one of the steps of the speech synthesis method provided in FIG. 1 in the first embodiment of the application;

3 is a schematic block diagram of a speech synthesis apparatus provided by an embodiment of the present application;

4 is a schematic diagram of an internal structure of an electronic device for implementing a speech synthesis method provided by an embodiment of the present application;

The realization, functional characteristics and advantages of the purpose of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

The embodiment of the present application provides a speech synthesis method. The executive body of the speech synthesis method includes, but is not limited to, at least one of electronic devices that can be configured to execute the method provided by the embodiments of the present application, such as a server, a terminal, and the like. In other words, the speech synthesis method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.

Referring to FIG. 1 , a schematic flowchart of a speech synthesis method provided by an embodiment of the present application is shown. In the embodiment of the present application, the speech synthesis method includes:

S1. Acquire training text, and use a pre-trained autoregressive speech synthesis model to perform speech synthesis on the training text to obtain a first sound spectrum of the training text.

In the embodiment of the present application, the training text includes different types of text, such as news text, financial text, and medical text. The pre-trained autoregressive speech synthesis includes an encoder, a decoder and a vocoder for synthesizing text into speech. The autoregressive speech synthesis model can be understood as an end-to-end speech synthesis model, that is, in the speech synthesis process, the speech data needs to be processed step by step. For example, the generation process of a sentence may include: first, generate the first word according to the probability distribution. , then the second word is generated according to the first word, the third word is generated according to the first two words, and so on, until the entire sentence is generated, because the autoregressive speech synthesis model performs speech synthesis on the training text The process is serial, so the synthesis quality of the speech synthesis result of the training text is guaranteed.

In one embodiment of the present application, referring to FIG. 2 , the pre-trained autoregressive speech synthesis model is used to perform speech synthesis on the training text to obtain the first sound spectrum of the training text, including:

S20, using the encoder in the autoregressive speech synthesis model to perform acoustic feature extraction on the training text to obtain the acoustic features of the training text;

S21, using the decoder in the autoregressive speech synthesis model to perform sequence recognition on the acoustic features, to obtain the acoustic feature sequence of the training text;

S22. Use the vocoder in the autoregressive speech synthesis model to perform spectrum conversion on the acoustic feature sequence to obtain a first sound spectrum of the training text.

Wherein, the encoder is generated by connecting a convolution layer and a gated recurrent unit (GRU) residual to extract the acoustic features in the training text, and the decoder is connected to a long short-term memory network unit ( LSTM) residual connection generation, the vocoder is built with multiple layers of causal convolutional layers and fully connected layers. The residual connection is to ensure that the problem of model gradient dispersion is suppressed during the processing of the training text. The acoustic characteristics refer to the physical quantities that represent the acoustic characteristics of speech, and are also the general term for the acoustic performance of various elements of the sound, such as the energy concentration area, formant frequency, formant intensity and bandwidth that represent the timbre, as well as the duration and fundamental frequency that represent the prosody characteristics of speech. , average speech power, etc.

Further, the S20 includes: using a convolution layer in the encoder to perform acoustic feature convolution on the training text to obtain initial acoustic features, and using a gated loop unit in the encoder to perform a convolution on the initial acoustic features The features are up-sampled to obtain the acoustic features of the training text.

In an optional embodiment, the convolution of the acoustic features of the training text is implemented through the convolution kernel of the convolution layer in the encoder, and the upsampling of the initial acoustic features is implemented through the update in the gated recurrent unit. The gate and the reset gate are implemented, the update gate is used to discard and add the information of the initial acoustic features, and the reset gate is used to selectively filter the information of the initial acoustic features processed by the update gate.

Further, the S21 includes: using the convolution layer in the decoder to perform feature extraction on the acoustic features, and using the long short-term memory network unit in the decoder to perform acoustic sequence recognition on the feature-extracted acoustic features , to obtain the acoustic feature sequence of the training text. Wherein, the acoustic sequence refers to the discrete time signal in the acoustic feature.

In an optional embodiment, the feature extraction of the acoustic features is realized by the convolution kernel of the convolution layer in the decoder, and the acoustic sequence recognition is realized by the input gate and the forget gate in the long short-term memory network unit. and output gate implementation, the input gate is used to split the acoustic sequence of the acoustic features after feature extraction, the forget gate is used to identify the acoustic sequence of the acoustic features after feature extraction, and the output gate An acoustic sequence for outputting the feature-extracted acoustic features.

Further, the S22 includes: using the causal convolution layer in the vocoder to perform sound frequency extraction on the acoustic feature sequence to obtain the sound frequency, and using the fully connected layer in the vocoder to extract the sound frequency The frequency is subjected to matrix mapping to obtain the first sound spectrum of the training text. Optionally, use the following formula to perform sound frequency extraction on the acoustic feature sequence:

Among them, Mel(f) represents the sound frequency, ln represents the filter function, and f represents the acoustic feature sequence.

S2. Extract linguistic features from the first sound spectrum to obtain first linguistic features.

In the embodiment of the present application, the linguistic features include the duration of pronunciation of words and the duration of sentence pauses. For example, there are 10 words and 2 sentences in a sound spectrum, and the pronunciation duration between the 10 words is 0.89s, 2 The pause time of each sentence is 1.2s. Based on the linguistic features, the phoneme duration of the generated sound spectrum can be identified, so that the generation quality of the sound spectrum can be judged.

In one of the embodiments of the present application, the extracting linguistic features from the first sound spectrum to obtain the first linguistic features includes: acquiring word vectors and sentence vectors in the first sound spectrum, The vector and the sentence vector are smoothed, and the smoothed word vector and the sentence vector are sampled for the pronunciation duration to obtain the word pronunciation duration and the sentence pronunciation duration. According to the word pronunciation duration and the sentence pronunciation duration, the first linguistics is generated. feature.

In an optional embodiment, the word vector is smoothed using the following formula:

Y(n)=a ₁ x ₁ (n)+a ₂ x ₂ (n)-Y(n-1)

Among them, Y(n) represents the word vector after smoothing, Y(n-1) represents the word vector, x ₁ (n) and x ₂ (n) represent the sound frequency and sound signal of the word vector, a ₁ and a ₂ represents the filter coefficient. In the embodiment of the present application, the smoothing process of the sentence vector is the same as the smoothing process of the word vector, which will not be further described here.

In an optional embodiment, the pronunciation duration sampling of the word vector and the sentence vector may be implemented by a sampler in the autoregressive speech synthesis model, such as a KONTAKT sampler.

S3. Train the pre-built non-autoregressive speech synthesis model according to the training text, the first sound spectrum and the first linguistic feature, to obtain a trained non-autoregressive speech synthesis model.

In the embodiment of the present application, the non-autoregressive speech synthesis model does not consider the context dependencies between words in the training text, so as to directly perform speech decoding on the words in the training text, and realize the parallel decoding of words, thereby greatly improving the The speech synthesis speed of the training text, therefore, the embodiment of the present application trains the pre-built non-autoregressive speech synthesis model by using the training text and the first linguistic feature to supervise the training text in the The accuracy of speech synthesis points in the non-autoregressive model, thereby improving the speech synthesis accuracy rate of the non-autoregressive speech synthesis model, so that the non-autoregressive model has a faster speech synthesis speed and also has a higher voice synthetic accuracy.

In one of the embodiments of the present application, the pre-built non-autoregressive speech synthesis model is trained according to the training text, the first sound spectrum and the first linguistic feature, and a trained non-autoregressive speech synthesis model is obtained. A regression speech synthesis model, comprising: inputting the training text into the non-autoregressive speech synthesis model to output a second sound spectrum of the training text, and extracting linguistic features from the second sound spectrum to obtain the second linguistic feature; according to the first sound spectrum, the second sound spectrum, the first linguistic feature and the second linguistic feature, calculate the training loss of the non-autoregressive speech synthesis model; if the training loss is not When a preset condition is met, adjust the parameters of the non-autoregressive model, and return to the step of inputting the training text into the non-autoregressive speech synthesis model; if the training loss meets the preset condition, get The trained non-autoregressive speech synthesis model.

In an optional embodiment, for the extraction principle of the linguistic feature of the second sound spectrum, reference may be made to the linguistic feature of the first sound spectrum, which will not be further described again.

In an optional embodiment, calculating the training loss of the non-autoregressive speech synthesis model according to the first sound spectrum, the second sound spectrum, the first linguistic feature, and the second linguistic feature includes: For the first sound spectrum and the second sound spectrum, calculate the first training loss of the non-autoregressive speech synthesis model; according to the first linguistic feature and the second linguistic feature, calculate the non-autoregressive speech synthesis The second training loss of the model, the training loss of the non-autoregressive speech synthesis model is calculated according to the first training loss and the second training loss.

In an optional embodiment of the present application, the first training loss is calculated according to the following formula:

LC=mg logm _p +(1-m _g ₎ log(1-m _p )

Among them, LC represents the first training loss, m _g represents the first sound spectrum, and _mp represents the second sound spectrum.

In an optional embodiment of the present application, the second training loss is calculated according to the following formula:

L1=|α _p -α _g |

Among them, L1 represents the second training loss, α _g represents the first linguistic feature, and α _p represents the second linguistic feature.

It should be noted that the second training loss is obtained by adding the word pronunciation duration loss and the sentence pronunciation duration loss in the linguistic feature.

In an optional embodiment of the present application, the first training loss and the second training loss are added to obtain the training loss of the non-autoregressive speech synthesis model, that is, L=L1+LC.

In an optional embodiment of the present application, the preset condition includes that the training loss is less than a loss threshold. That is, when the training loss is less than the loss threshold, it means that the training loss satisfies the preset condition, and when the training loss is greater than or equal to the loss threshold, it means that the training loss does not meet the predetermined condition. when the preset conditions are stated. Wherein, the loss threshold may be set to 0.1, or may be set according to actual scenarios. Further, the parameter adjustment of the non-autoregressive speech synthesis model may be implemented by a gradient descent algorithm, such as a stochastic descent algorithm.

S4, using the non-autoregressive speech synthesis model completed by the training to perform speech synthesis on the speech text to be synthesized to obtain a speech synthesis result.

In the embodiment of the present application, the use of the trained non-autoregressive speech synthesis model to perform speech synthesis on to-be-synthesized speech text to obtain a speech synthesis result includes: inputting the to-be-synthesized speech text into the trained non-autoregressive speech text In the regression speech synthesis model, the non-autoregressive speech synthesis model completed by the training is used to identify the sound spectrum of the speech text to be synthesized, and the speech synthesis result of the speech text to be synthesized is obtained. Wherein, the recognition of the sound spectrum of the speech text to be synthesized is realized by the encoder and the decoder in the non-autoregressive speech synthesis model completed by the training.

Further, in order to ensure the security and privacy of the speech synthesis result, the speech synthesis result can also be stored in a blockchain node.

In the embodiment of the present application, the training text is first synthesized by using a pre-trained autoregressive speech synthesis model to obtain the first sound spectrum of the training text, and linguistic features are extracted from the first sound spectrum to obtain the first sound spectrum of the training text. linguistic features to obtain the training data of the subsequent non-autoregressive speech synthesis model; secondly, in this embodiment of the present application, according to the training text, the first sound spectrum and the first linguistic feature, the pre-built non-autoregressive speech The autoregressive speech synthesis model is trained to obtain the trained non-autoregressive speech synthesis model, so that the linguistic features can be added to the model training data, which can enhance the quality of the training data, and combine the first sound spectrum generated by the autoregressive model to supervise The training results of the non-autoregressive model, so that the non-autoregressive model can better learn the effect of speech synthesis of the autoregressive model, thereby improving the speech synthesis quality of the non-autoregressive model, and combined with the non-autoregressive model in the process of speech synthesis. Parallel decoding of audio sequences is implemented, so that the non-autoregressive model has a faster speech synthesis speed and higher speech synthesis quality. Therefore, a speech synthesis method proposed in this application can improve the speed and quality of speech synthesis.

As shown in FIG. 3 , it is a functional block diagram of the speech synthesis apparatus of the present application.

The speech synthesis apparatus 100 described in this application can be installed in an electronic device. According to the realized functions, the speech synthesis apparatus may include an acquisition module 101 , an extraction module 102 , a training module 103 and a synthesis module 104 . The modules described in the present invention can also be called units, which refer to a series of computer program segments that can be executed by the electronic device processor and can perform fixed functions, and are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The acquisition module 101 is used for acquiring training text, and using a pre-trained autoregressive speech synthesis model to perform speech synthesis on the training text to obtain a first sound spectrum of the training text;

The extraction module 102 is configured to extract linguistic features from the first sound spectrum to obtain first linguistic features;

The training module 103 is configured to train a pre-built non-autoregressive speech synthesis model according to the training text, the first sound spectrum and the first linguistic feature, to obtain a trained non-autoregressive speech. synthetic model;

The synthesis module 104 is configured to use the trained non-autoregressive speech synthesis model to perform speech synthesis on the speech text to be synthesized to obtain a speech synthesis result.

In detail, the modules in the speech synthesis apparatus 100 in the embodiments of the present application use the same technical means as the speech synthesis methods described in the above-mentioned FIG. 1 and FIG. 2, and can generate the same technology The effect will not be repeated here.

As shown in FIG. 4 , it is a schematic structural diagram of an electronic device implementing the speech synthesis method of the present application.

The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program, such as a speech synthesis program 12, stored in the memory 11 and executable on the processor 10.

Wherein, the memory 11 includes at least one type of readable storage medium, and the readable storage medium may be volatile or non-volatile. Specifically, the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (eg, SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, and the like. The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1 . In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a pluggable mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital) equipped on the electronic device 1. , SD) card, flash memory card (Flash Card), etc. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software installed in the electronic device 1 and various types of data, such as the code of the speech synthesis program 12, etc., but also can be used to temporarily store data that has been output or will be output.

In some embodiments, the processor 10 may be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more integrated circuits. Central Processing Unit (CPU), microprocessor, digital processing chip, graphics processor and combination of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device, by running or executing the program or module (for example, executing the program) stored in the memory 11. speech synthesis program 12, etc.), and call data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

The bus may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (Extended industry standard architecture, EISA for short) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. The bus is configured to implement connection communication between the memory 11 and at least one processor 10 and the like.

FIG. 4 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 4 does not constitute a limitation on the electronic device 1, and may include fewer or more components than those shown in the drawings. components, or a combination of certain components, or a different arrangement of components.

For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) for powering the various components, preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so that the power management The device implements functions such as charge management, discharge management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device 1 may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

Further, the electronic device 1 may also include a network interface, optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used in the electronic device 1 Establish a communication connection with other electronic devices.

Optionally, the electronic device 1 may further include a user interface, and the user interface may be a display (Display), an input unit (eg, a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch panel, and the like. The display may also be appropriately called a display screen or a display unit, which is used for displaying information processed in the electronic device 1 and for displaying a visualized user interface.

It should be understood that the embodiments are only used for illustration, and are not limited by this structure in the scope of the patent application.

The speech synthesis 12 stored in the memory 11 in the electronic device 1 is a combination of multiple computer programs. When running in the processor 10, it can realize:

Specifically, for the specific implementation method of the above-mentioned computer program by the processor 10, reference may be made to the description of the relevant steps in the corresponding embodiment of FIG. 1, and details are not described herein.

Further, if the modules/units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a non-volatile computer-readable storage medium. The computer-readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a removable hard disk, a magnetic disk, an optical disc, a computer memory, a read-only memory (ROM, Read-Only). Memory).

The present application also provides a computer-readable storage medium, where the readable storage medium stores a computer program, and when executed by a processor of an electronic device, the computer program can realize:

In the several embodiments provided in this application, it should be understood that the disclosed apparatus, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

Accordingly, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the application is to be defined by the appended claims rather than the foregoing description, which is therefore intended to fall within the scope of the claims. All changes within the meaning and scope of the equivalents of , are included in this application. Any reference signs in the claims shall not be construed as limiting the involved claim.

The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

Furthermore, it is clear that the word "comprising" does not exclude other units or steps and the singular does not exclude the plural. Several units or means recited in the system claims can also be realized by one unit or means by means of software or hardware. Second-class terms are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than limitations. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present application.

Claims

A speech synthesis method, wherein the method comprises:

Acquiring training text, using a pre-trained autoregressive speech synthesis model to perform speech synthesis on the training text to obtain the first sound spectrum of the training text;

Extracting linguistic features from the first sound spectrum to obtain first linguistic features;

According to the training text, the first sound spectrum and the first linguistic feature, the pre-built non-autoregressive speech synthesis model is trained to obtain a trained non-autoregressive speech synthesis model;

Using the trained non-autoregressive speech synthesis model to perform speech synthesis on the speech text to be synthesized to obtain a speech synthesis result.
The speech synthesis method according to claim 1, wherein said using a pre-trained autoregressive speech synthesis model to perform speech synthesis on the training text to obtain the first sound spectrum of the training text, comprising:

Use the encoder in the autoregressive speech synthesis model to perform acoustic feature extraction on the training text to obtain the acoustic features of the training text;

Use the decoder in the autoregressive speech synthesis model to perform sequence recognition on the acoustic features to obtain the acoustic feature sequence of the training text;

The vocoder in the autoregressive speech synthesis model is used to perform spectrum conversion on the acoustic feature sequence to obtain the first sound spectrum of the training text.
The speech synthesis method according to claim 1, wherein, the use of the encoder in the autoregressive speech synthesis model to perform acoustic feature extraction on the training text to obtain the acoustic features of the training text, comprising:

Use the convolution layer in the encoder to perform acoustic feature convolution on the training text to obtain initial acoustic features;

The initial acoustic features are up-sampled by the gated loop unit in the encoder to obtain the acoustic features of the training text.
The speech synthesis method according to claim 2, wherein the sequence recognition of the acoustic features by the decoder in the autoregressive speech synthesis model to obtain the acoustic feature sequence of the training text, comprising:

Feature extraction is performed on the acoustic features by using a convolutional layer in the decoder;

The long-short-term memory network unit in the decoder is used to perform acoustic sequence recognition on the acoustic features after feature extraction, so as to obtain the acoustic feature sequence of the training text.
The speech synthesis method according to claim 1, wherein the extracting linguistic features from the first sound spectrum to obtain the first linguistic features comprises:

Acquiring word vectors and sentence vectors in the first sound spectrum, and smoothing the word vectors and sentence vectors;

Sampling the pronunciation duration of the smoothed word vector and the sentence vector to obtain the pronunciation duration of the word and the pronunciation duration of the sentence;

The first linguistic feature is generated according to the pronunciation duration of the word and the pronunciation duration of the sentence.
The speech synthesis method according to any one of claims 1 to 5, wherein the pre-built non-autoregressive speech is performed according to the training text, the first sound spectrum and the first linguistic feature. The synthetic model is trained to obtain a trained non-autoregressive speech synthesis model, including:

Inputting the training text into the non-autoregressive speech synthesis model, to output the second sound spectrum of the training text, and extracting linguistic features from the second sound spectrum to obtain second linguistic features;

Calculate the training loss of the non-autoregressive speech synthesis model according to the first sound spectrum, the second sound spectrum, the first linguistic feature, and the second linguistic feature;

If the training loss does not meet the preset condition, adjust the parameters of the non-autoregressive model, and return to the step of inputting the training text into the non-autoregressive speech synthesis model;

If the training loss satisfies the preset condition, a trained non-autoregressive speech synthesis model is obtained.
The speech synthesis method according to claim 6, wherein calculating the non-autoregressive speech synthesis model according to the first sound spectrum, the second sound spectrum, the first linguistic feature and the second linguistic feature Training losses, including:

calculating the first training loss of the non-autoregressive speech synthesis model according to the first sound spectrum and the second sound spectrum;

calculating a second training loss of the non-autoregressive speech synthesis model according to the first linguistic feature and the second linguistic feature;

According to the first training loss and the second training loss, the training loss of the non-autoregressive speech synthesis model is calculated.
A speech synthesis device, wherein the device comprises:

an acquisition module, used for acquiring training text, and using a pre-trained autoregressive speech synthesis model to perform speech synthesis on the training text to obtain the first sound spectrum of the training text;

an extraction module for extracting linguistic features from the first sound spectrum to obtain first linguistic features;

A training module for training a pre-built non-autoregressive speech synthesis model according to the training text, the first sound spectrum and the first linguistic feature, to obtain a trained non-autoregressive speech synthesis model;

The synthesis module is configured to perform speech synthesis on the speech text to be synthesized by using the non-autoregressive speech synthesis model completed by the training to obtain a speech synthesis result.
An electronic device, wherein the electronic device comprises:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a speech synthesis method as described below:

Acquiring training text, using a pre-trained autoregressive speech synthesis model to perform speech synthesis on the training text to obtain the first sound spectrum of the training text;

Extracting linguistic features from the first sound spectrum to obtain first linguistic features;

According to the training text, the first sound spectrum and the first linguistic feature, the pre-built non-autoregressive speech synthesis model is trained to obtain a trained non-autoregressive speech synthesis model;

Using the trained non-autoregressive speech synthesis model to perform speech synthesis on the speech text to be synthesized to obtain a speech synthesis result.
The electronic device as claimed in claim 9, wherein, performing speech synthesis on the training text by using a pre-trained autoregressive speech synthesis model to obtain the first sound spectrum of the training text, comprising:

Use the encoder in the autoregressive speech synthesis model to perform acoustic feature extraction on the training text to obtain the acoustic features of the training text;

Use the decoder in the autoregressive speech synthesis model to perform sequence recognition on the acoustic features to obtain the acoustic feature sequence of the training text;

The acoustic feature sequence is spectrally converted by the vocoder in the autoregressive speech synthesis model to obtain the first sound spectrum of the training text.
The electronic device according to claim 9, wherein, the use of the encoder in the autoregressive speech synthesis model to perform acoustic feature extraction on the training text to obtain the acoustic features of the training text, comprising:

Use the convolution layer in the encoder to perform acoustic feature convolution on the training text to obtain initial acoustic features;

The initial acoustic features are up-sampled by the gated loop unit in the encoder to obtain the acoustic features of the training text.
The electronic device according to claim 10, wherein the sequence recognition of the acoustic features by using the decoder in the autoregressive speech synthesis model, to obtain the acoustic feature sequence of the training text, comprises:

Feature extraction is performed on the acoustic features by using a convolutional layer in the decoder;

The long-short-term memory network unit in the decoder is used to perform acoustic sequence recognition on the acoustic features after feature extraction, so as to obtain the acoustic feature sequence of the training text.
The electronic device according to claim 9, wherein the extracting linguistic features from the first sound spectrum to obtain the first linguistic features comprises:

Acquiring word vectors and sentence vectors in the first sound spectrum, and smoothing the word vectors and sentence vectors;

Sampling the pronunciation duration of the smoothed word vector and the sentence vector to obtain the pronunciation duration of the word and the pronunciation duration of the sentence;

The first linguistic feature is generated according to the pronunciation duration of the word and the pronunciation duration of the sentence.
The electronic device according to any one of claims 9 to 13, wherein the pre-built non-autoregressive speech synthesis is performed according to the training text, the first sound spectrum and the first linguistic feature The model is trained to obtain a trained non-autoregressive speech synthesis model, including:

Inputting the training text into the non-autoregressive speech synthesis model, to output the second sound spectrum of the training text, and extracting linguistic features from the second sound spectrum to obtain second linguistic features;

Calculate the training loss of the non-autoregressive speech synthesis model according to the first sound spectrum, the second sound spectrum, the first linguistic feature, and the second linguistic feature;

If the training loss does not meet the preset condition, adjust the parameters of the non-autoregressive model, and return to the step of inputting the training text into the non-autoregressive speech synthesis model;

If the training loss satisfies the preset condition, a trained non-autoregressive speech synthesis model is obtained.
A computer-readable storage medium storing a computer program, wherein the computer program implements the following speech synthesis method when the computer program is executed by a processor:

Acquiring training text, using a pre-trained autoregressive speech synthesis model to perform speech synthesis on the training text to obtain the first sound spectrum of the training text;

Extracting linguistic features from the first sound spectrum to obtain first linguistic features;

According to the training text, the first sound spectrum and the first linguistic feature, the pre-built non-autoregressive speech synthesis model is trained to obtain a trained non-autoregressive speech synthesis model;

Using the trained non-autoregressive speech synthesis model to perform speech synthesis on the speech text to be synthesized to obtain a speech synthesis result.
The computer-readable storage medium according to claim 15 , wherein, performing speech synthesis on the training text by using a pre-trained autoregressive speech synthesis model to obtain the first sound spectrum of the training text, comprising:

Use the encoder in the autoregressive speech synthesis model to perform acoustic feature extraction on the training text to obtain the acoustic features of the training text;

Use the decoder in the autoregressive speech synthesis model to perform sequence recognition on the acoustic features to obtain the acoustic feature sequence of the training text;

The acoustic feature sequence is spectrally converted by the vocoder in the autoregressive speech synthesis model to obtain the first sound spectrum of the training text.
The computer-readable storage medium according to claim 15 , wherein, using the encoder in the autoregressive speech synthesis model to perform acoustic feature extraction on the training text to obtain the acoustic features of the training text, comprising:

Use the convolution layer in the encoder to perform acoustic feature convolution on the training text to obtain initial acoustic features;

The initial acoustic features are up-sampled by the gated loop unit in the encoder to obtain the acoustic features of the training text.
The computer-readable storage medium according to claim 16, wherein the sequence identification of the acoustic features by the decoder in the autoregressive speech synthesis model to obtain the acoustic feature sequence of the training text comprises:

Feature extraction is performed on the acoustic features by using a convolutional layer in the decoder;

The long-short-term memory network unit in the decoder is used to perform acoustic sequence recognition on the acoustic features after feature extraction, so as to obtain the acoustic feature sequence of the training text.
The computer-readable storage medium of claim 15, wherein the extracting linguistic features from the first sound spectrum to obtain the first linguistic features comprises:

Acquiring word vectors and sentence vectors in the first sound spectrum, and smoothing the word vectors and sentence vectors;

Sampling the pronunciation duration of the smoothed word vector and the sentence vector to obtain the pronunciation duration of the word and the pronunciation duration of the sentence;

The first linguistic feature is generated according to the pronunciation duration of the word and the pronunciation duration of the sentence.
The computer-readable storage medium according to any one of claims 15 to 19, wherein the pre-built non-self The regression speech synthesis model is trained, and the trained non-autoregressive speech synthesis model is obtained, including:

Inputting the training text into the non-autoregressive speech synthesis model, to output the second sound spectrum of the training text, and extracting linguistic features from the second sound spectrum to obtain second linguistic features;

Calculate the training loss of the non-autoregressive speech synthesis model according to the first sound spectrum, the second sound spectrum, the first linguistic feature, and the second linguistic feature;

If the training loss does not meet the preset condition, adjust the parameters of the non-autoregressive model, and return to the step of inputting the training text into the non-autoregressive speech synthesis model;

If the training loss satisfies the preset condition, a trained non-autoregressive speech synthesis model is obtained.