CN114093342B - Fine-grained rhythm modeling voice generation device, fine-grained rhythm modeling voice generation equipment and fine-grained rhythm modeling voice generation storage medium - Google Patents
Fine-grained rhythm modeling voice generation device, fine-grained rhythm modeling voice generation equipment and fine-grained rhythm modeling voice generation storage medium Download PDFInfo
- Publication number
- CN114093342B CN114093342B CN202210078586.8A CN202210078586A CN114093342B CN 114093342 B CN114093342 B CN 114093342B CN 202210078586 A CN202210078586 A CN 202210078586A CN 114093342 B CN114093342 B CN 114093342B
- Authority
- CN
- China
- Prior art keywords
- information
- prosody
- text
- voice
- prosodic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000033764 rhythmic process Effects 0.000 title claims description 17
- 238000001228 spectrum Methods 0.000 claims abstract description 49
- 238000000034 method Methods 0.000 claims description 20
- 238000013528 artificial neural network Methods 0.000 claims description 12
- 238000007781 pre-processing Methods 0.000 claims description 12
- 230000000306 recurrent effect Effects 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 6
- 238000012805 post-processing Methods 0.000 claims description 6
- 230000003595 spectral effect Effects 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 4
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 230000008859 change Effects 0.000 abstract description 3
- 238000004590 computer program Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 238000012545 processing Methods 0.000 description 7
- 230000015572 biosynthetic process Effects 0.000 description 6
- 238000003786 synthesis reaction Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 3
- 230000008451 emotion Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L13/10—Prosody rules derived from text; Stress or intonation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
Abstract
The invention provides a speech generation model, equipment and a storage medium for fine-grained prosody modeling, wherein the model comprises the following components: inputting text information into a text encoder module to obtain the encoding characteristics of the text; inputting the frequency spectrum information into a prosody coding module to obtain prosody characteristics of the phoneme-level voice; the prosody feature input decoupling module of the voice decouples text content information and prosody information contained in the prosody feature of the voice and only leaves the prosody information of the prosody feature of the voice; the encoding characteristics of the text and the prosodic information of the prosodic characteristics of the speech are input to a decoder to generate synthesized speech. The scheme provided by the invention can divide the time domain of the audio by using the time-length information, the prosody encoder can model the local prosody information of the voice to describe the change trend of the prosody, and meanwhile, the prosody decoupling module ensures that the prosody encoder model only learns the prosody information of the voice and does not contain text information.
Description
Technical Field
The invention belongs to the field of voice generation, and particularly relates to a voice generation device for fine-grained prosody modeling.
Background
With the continuous abundance and diversity of applications of speech synthesis, the user's standards for intelligibility, stability, naturalness, and speech expressiveness of synthesized speech are increasing. Application scenarios such as audio books, voice assistants, dialogue interaction, etc. expect that synthesized speech can have a naturalness similar to that of real human speech. Therefore, the robustness, real-time rate, and the influence of intonation, rereading, emotion, style, semantic and other information on the naturalness of the synthesized speech need to be considered. With the rapid development of speech generation technology in recent years, a non-autoregressive speech generation learning-oriented model framework has become a mainstream research trend. Compared with the prior autoregressive network, the non-autoregressive network has the advantages of a training time period, high generation speed, strong speech robustness, strong controllability and the like, but meanwhile, due to the characteristic of parallelization generation, the generation of speech completely depends on the input text characteristic information and cannot be modeled by using historical speech information. Since speech synthesis is a highly up-sampling process, the mapping between text-speech data pairs is a one-to-many timbre, and the modeling process of speech generation loses much text without information, namely speech prosody information, after modeling of historical information is lacked. A method for representing learning can be flexibly introduced based on a speech synthesis model from a sequence to a sequence framework, and more accurate prosodic representation is extracted, so that more effective and controllable speech synthesis acoustic modeling is realized, and the naturalness of speech generation is improved.
Disadvantages of the prior art
(1) Since speech synthesis is a highly up-sampling process, the mapping between text-speech data pairs is a one-to-many timbre, and the modeling process of speech generation loses much text without information, namely speech prosody information, after modeling of historical information is lacked. Therefore, the generated speech of the current speech synthesis system has the defects of unclear energy of high-frequency parts, flat tone style and the like.
(2) Most of the current speech prosody modeling is global information, that is, the whole speech is coded by a hidden vector corresponding to only one prosody. However, it is known that prosody of speech is a time-varying process, global prosodic information can well model the whole emotion of speech, but is ineffective to local variation and expression.
Disclosure of Invention
In order to solve the technical problems, the invention provides a technical scheme of a speech generating device with fine-grained prosody modeling, so as to solve the technical problems.
The invention discloses a voice generating device for fine-grained prosody modeling in a first aspect; the device comprises:
a text encoder, a prosodic encoder, and a decoder;
the prosody encoder includes: a rhythm coding module and a decoupling module;
inputting text information into the text encoder to obtain the encoding characteristics of the text;
inputting the frequency spectrum information into the rhythm coding module to obtain the rhythm characteristics of the phoneme-level voice;
inputting the prosodic features of the voice into the decoupling module, decoupling text content information and prosodic information contained in the prosodic features of the voice, and only leaving the prosodic information of the prosodic features of the voice;
and inputting the encoding characteristics of the text and prosody information of the prosody characteristics of the voice into the decoder to generate synthetic voice.
According to an aspect of the first aspect of the present invention, the text encoder includes:
a word embedding layer, a precoding layer and a recurrent neural network;
and the text information is sequentially input into the word embedding layer, the pre-coding layer and the cyclic neural network to obtain the coding characteristics of the text.
According to the technical scheme of the first aspect of the present invention, the specific method for obtaining the coding features of the text by sequentially inputting the text information into the word embedding layer, the pre-coding layer and the recurrent neural network comprises:
and inputting the text information into the word embedding layer to obtain a representation in a high-dimensional continuous space, inputting the representation in the high-dimensional continuous space into the pre-coding layer, compressing dimensions and information of the representation in the high-dimensional continuous space to obtain a representation in the compressed continuous space, and inputting the representation in the compressed continuous space into a recurrent neural network to obtain the coding characteristics of the text.
According to the technical solution of the first aspect of the present invention, the prosody encoding module includes:
the system comprises a frequency spectrum preprocessing network, a multi-head self-attention module and a full connection layer;
and the frequency spectrum information is sequentially input into the frequency spectrum preprocessing network, the multi-head self-attention module and the full-connection layer to obtain the prosodic features of the phoneme-level voice.
According to the technical scheme of the first aspect of the present invention, the specific method for obtaining the prosodic features of the phoneme-level speech by sequentially inputting the spectrum information into the spectrum preprocessing network, the multi-head self-attention module and the full-link layer includes:
the frequency spectrum preprocessing network expands the frequency spectrum information to high-dimensional frequency spectrum characteristics;
inputting the high-dimensional spectral features into the multi-head self-attention module, and calculating the weighted sum of the high-dimensional spectral features in the time dimension through a self-attention mechanism to obtain attention features;
and the full connection layer calculates the weighted sum of the attention features in feature dimensions, and finally averages the output of the full connection layer in a time period corresponding to the phoneme to obtain the final prosodic features of the phoneme-level speech.
According to an aspect of the first aspect of the invention, the decoupling module is designed to generate text content information and prosody information contained in prosody features that decouple the voice against the network.
According to the technical solution of the first aspect of the present invention, the decoupling module needs to perform iterative optimization separately from the text encoder and the decoder in turn during the network optimization process.
According to the technical solution of the first aspect of the present invention, the decoder includes: a multi-layer multi-head self-attention module and a post-processing network;
inputting the encoding characteristics of the text and the prosodic information of the prosodic characteristics of the voice into the multi-head attention module for characteristic transformation to obtain a fused characteristic frequency spectrum, inputting the fused characteristic frequency spectrum into the post-processing network, and finely adjusting the fused characteristic frequency spectrum by using local frequency spectrum information to generate synthetic voice; the local spectrum information is prosodic information of prosodic features of the voice.
A second aspect of the invention provides an electronic device comprising a memory and a processor, the memory having stored thereon a computer program which, when executed by the processor, performs a speech generating means in a fine-grained prosody modeling speech generating means as described in the first aspect of the invention.
The scheme provided by the invention can divide the time domain of the audio by using the time-length information, the prosody encoder can model the local prosody information of the voice to describe the change trend of the prosody, and meanwhile, the prosody decoupling module ensures that the prosody encoder device only learns the prosody information of the voice and does not contain text information.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a block diagram of a fine-grained prosody modeled speech generation apparatus according to an embodiment of the present invention;
fig. 2 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
Example 1:
a first aspect of the present invention discloses a fine-grained prosody modeling speech generating device, where fig. 1 is a structural diagram of a fine-grained prosody modeling speech generating device according to an embodiment of the present invention, specifically as shown in fig. 1, the device includes:
a text encoder, a prosodic encoder, and a decoder;
the prosody encoder includes: a rhythm coding module and a decoupling module;
inputting text information into the text encoder module to obtain the encoding characteristics of the text;
in some embodiments, the text encoder comprises:
a word embedding layer, a precoding layer and a recurrent neural network;
the text information is sequentially input into the word embedding layer, the pre-coding layer and the cyclic neural network to obtain the coding characteristics of the text;
in some embodiments, the text information is sequentially input into the word embedding layer, the pre-coding layer and the recurrent neural network, and the specific method for obtaining the coding features of the text includes:
inputting the text information into the word embedding layer to obtain a representation in a 256-dimensional continuous space, inputting the representation in the high-dimensional continuous space into the pre-coding layer, compressing dimensions and information of the representation in the high-dimensional continuous space to obtain a representation in a 128-dimensional continuous space, and inputting the compressed representation in the continuous space into a recurrent neural network to obtain a coding feature of the text;
in some embodiments, the pre-coding layer is a fully connected layer of two layers, and removing some minor information and dimensions in the representation in the 256-dimensional continuous space by the pre-coding layer helps to improve the stability and generalization capability of the device and reduce the training difficulty;
inputting the frequency spectrum information into the rhythm coding module to obtain the rhythm characteristics of the phoneme-level voice;
in some embodiments, the prosody encoding module includes:
the system comprises a frequency spectrum preprocessing network, a multi-head self-attention module and a full connection layer;
the frequency spectrum information is sequentially input into the frequency spectrum preprocessing network, the multi-head self-attention module and the full-connection layer to obtain the prosodic features of the phoneme-level voice;
in some embodiments, the specific method for obtaining the prosodic features of the phoneme-level speech by sequentially inputting the spectrum information into the spectrum preprocessing network, the multi-head self-attention module and the full-connection layer includes:
the spectrum information refers to implicit vector characterization of the spectrum, and in some embodiments, specifically, the spectrum information may be 80-dimensionel spectrum features, and the prosodic features at the phoneme level are obtained through an averaging pooling operation on phoneme intervals.
The frequency spectrum preprocessing network expands the 80-dimensional Meyer frequency spectrum characteristic to a high-dimensional frequency spectrum characteristic;
inputting the high-dimensional spectral features into the multi-head self-attention module, and calculating the weighted sum of the high-dimensional spectral features in the time dimension through a self-attention mechanism to obtain attention features;
the full-connection layer calculates the weighted sum of the attention features in feature dimensions, and finally averages the output of the full-connection layer in a time period corresponding to the phoneme to obtain the final prosodic features of the phoneme-level speech;
because the prosodic features of the speech simultaneously comprise prosodic information and text content information, if the two kinds of information are not decoupled, the output result of a prosodic encoder comprises text information, which conflicts with the result of a text encoder, and various serious problems such as wrong pronunciation of the generated speech, low speech intelligibility and the like are caused;
inputting the prosodic features of the voice into the decoupling module, decoupling text content information and prosodic information contained in the prosodic features of the voice, and only leaving the prosodic information of the prosodic features of the voice;
in some embodiments, the decoupling module is designed to generate textual content information and prosodic information contained in prosodic features that decouple the speech against a network;
in some embodiments, the decoupling module needs to separately perform alternate iterative optimization with the text encoder and decoder during network optimization;
inputting the coding features of the text and the prosody information of the prosody features of the voice into the decoder to generate synthetic voice;
in some embodiments, the decoder comprises: 6 layers of multi-head self-attention modules and a post-processing network;
inputting the encoding characteristics of the text and the prosodic information of the prosodic characteristics of the voice into the 6-head attention module for characteristic transformation to obtain a fused characteristic frequency spectrum, inputting the fused characteristic frequency spectrum into the post-processing network, and finely adjusting the fused characteristic frequency spectrum by using local frequency spectrum information to generate synthetic voice; the local spectrum information is prosodic information of prosodic features of the voice.
The method comprises the steps of using a local self-attention mechanism of a small window, wherein the window length is 100ms, and modeling residual errors between frequency spectrum information generated by a decoder and real frequency spectrum information by using information of adjacent frames.
And (3) a voice generation process: the input of the text encoder is the text of the forged voice to be generated, the input of the prosody encoder is the guide voice corresponding to the prosody characteristics contained in the expected forged voice, and the decoder can output the forged voice containing the tone of the guide voice. The forged voice has the similar tone color and rhythm with the guiding voice, so that the characteristic of flat and light rhythm can be avoided, and the attack to the voice identification device is realized.
In summary, the technical solutions of the aspects of the present invention have the following advantages compared with the prior art: the time domain division is carried out on the audio by using the time-length information, the prosody encoder can model the local prosody information of the voice to describe the change trend of the prosody, and meanwhile, the prosody decoupling module ensures that the prosody encoder device only learns the prosody information of the voice and does not contain text information.
Example 2:
a second aspect of the present invention discloses an electronic device, which includes a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the speech generating apparatus for fine-grained prosody modeling according to any one of embodiments 1 of the present invention is implemented.
Fig. 2 is a block diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 2, the electronic device includes a processor, a memory, a communication interface, a display screen, and an input device, which are connected by a system bus. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic equipment comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the electronic device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, an operator network, Near Field Communication (NFC) or other technologies. The display screen of the electronic equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the electronic equipment, an external keyboard, a touch pad or a mouse and the like.
It will be understood by those skilled in the art that the structure shown in fig. 2 is only a partial block diagram related to the technical solution of the present disclosure, and does not constitute a limitation of the electronic device to which the solution of the present application is applied, and a specific electronic device may include more or less components than those shown in the drawings, or combine some components, or have a different arrangement of components.
It should be noted that the technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, however, as long as there is no contradiction between the combinations of the technical features, the scope of the present description should be considered. The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not to be construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Embodiments of the subject matter and the functional operations described in this specification can be implemented in: digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode and transmit information to suitable receiver apparatus for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
Computers suitable for executing computer programs include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and/or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer does not necessarily have such a device. Moreover, a computer may be embedded in another device, e.g., a mobile telephone, a Personal Digital Assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device such as a Universal Serial Bus (USB) flash drive, to name a few.
Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., an internal hard disk or a removable disk), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. In other instances, features described in connection with one embodiment may be implemented as discrete components or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Further, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (7)
1. An apparatus for fine-grained prosody modeled speech generation, the apparatus comprising:
a text encoder, a prosodic encoder, and a decoder;
the prosody encoder includes: a rhythm coding module and a decoupling module;
inputting text information into the text encoder to obtain the encoding characteristics of the text;
inputting the frequency spectrum information into the rhythm coding module to obtain the rhythm characteristics of the phoneme-level voice;
inputting the prosodic features of the voice into the decoupling module, decoupling text content information and prosodic information contained in the prosodic features of the voice, and only leaving the prosodic information of the prosodic features of the voice;
inputting the encoding characteristics of the text and the prosody information of the prosody characteristics of the voice into the decoder to generate synthetic voice;
the prosody encoding module includes:
the system comprises a frequency spectrum preprocessing network, a multi-head self-attention module and a full connection layer;
the frequency spectrum information is sequentially input into the frequency spectrum preprocessing network, the multi-head self-attention module and the full-connection layer to obtain the prosodic features of the phoneme-level voice;
the specific method for obtaining the prosodic features of the phoneme-level speech by sequentially inputting the frequency spectrum information into the frequency spectrum preprocessing network, the multi-head self-attention module and the full-connection layer comprises the following steps:
the frequency spectrum preprocessing network expands the frequency spectrum information to high-dimensional frequency spectrum characteristics;
inputting the high-dimensional spectral features into the multi-head self-attention module, and calculating the weighted sum of the high-dimensional spectral features in the time dimension through a self-attention mechanism to obtain attention features;
and the full connection layer calculates the weighted sum of the attention features in feature dimensions, and finally averages the output of the full connection layer in a time period corresponding to the phoneme to obtain the final prosodic features of the phoneme-level speech.
2. The apparatus of claim 1, wherein the text encoder comprises:
a word embedding layer, a precoding layer and a recurrent neural network;
and the text information is sequentially input into the word embedding layer, the pre-coding layer and the cyclic neural network to obtain the coding characteristics of the text.
3. The fine-grained prosody modeling speech generating device according to claim 2, wherein the text information is sequentially input into the word embedding layer, the pre-coding layer and the recurrent neural network, and the specific method for obtaining the coding features of the text comprises:
and inputting the text information into the word embedding layer to obtain a representation in a high-dimensional continuous space, inputting the representation in the high-dimensional continuous space into the pre-coding layer, compressing dimensions and information of the representation in the high-dimensional continuous space to obtain a representation in the compressed continuous space, and inputting the representation in the compressed continuous space into a recurrent neural network to obtain the coding characteristics of the text.
4. A fine-grained prosodic modeled speech generating device according to claim 1, wherein said decoupling module is configured to generate textual content information and prosodic information contained in prosodic features that counter-network decouple said speech.
5. The apparatus of claim 4, wherein the decoupling module is configured to perform iterative optimization with the text encoder and decoder separately during a network optimization procedure.
6. The apparatus of claim 1, wherein the decoder comprises: a multi-layer multi-head self-attention module and a post-processing network;
inputting the encoding characteristics of the text and the prosodic information of the prosodic characteristics of the voice into the multi-head attention module for characteristic transformation to obtain a fused characteristic frequency spectrum, inputting the fused characteristic frequency spectrum into the post-processing network, and finely adjusting the fused characteristic frequency spectrum by using local frequency spectrum information to generate synthetic voice; the local spectrum information is prosodic information of prosodic features of the voice.
7. An electronic device comprising a fine-grained prosody modeled speech generating apparatus according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210078586.8A CN114093342B (en) | 2022-01-24 | 2022-01-24 | Fine-grained rhythm modeling voice generation device, fine-grained rhythm modeling voice generation equipment and fine-grained rhythm modeling voice generation storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210078586.8A CN114093342B (en) | 2022-01-24 | 2022-01-24 | Fine-grained rhythm modeling voice generation device, fine-grained rhythm modeling voice generation equipment and fine-grained rhythm modeling voice generation storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114093342A CN114093342A (en) | 2022-02-25 |
| CN114093342B true CN114093342B (en) | 2022-05-03 |
Family
ID=80309211
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210078586.8A Active CN114093342B (en) | 2022-01-24 | 2022-01-24 | Fine-grained rhythm modeling voice generation device, fine-grained rhythm modeling voice generation equipment and fine-grained rhythm modeling voice generation storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114093342B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115132204B (en) * | 2022-06-10 | 2024-03-22 | 腾讯科技（深圳）有限公司 | Voice processing method, equipment, storage medium and computer program product |
| CN116092479B (en) * | 2023-04-07 | 2023-07-07 | 杭州东上智能科技有限公司 | Text prosody generation method and system based on comparison text-audio pair |
| CN116364055B (en) * | 2023-05-31 | 2023-09-01 | 中国科学院自动化研究所 | Speech generation method, device, equipment and medium based on pre-training language model |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112365882A (en) * | 2020-11-30 | 2021-02-12 | 北京百度网讯科技有限公司 | Speech synthesis method, model training method, device, equipment and storage medium |
| CN113284485A (en) * | 2021-07-09 | 2021-08-20 | 中国科学院自动化研究所 | End-to-end framework for unified Chinese and English mixed text generation and speech recognition |
| CN113327627A (en) * | 2021-05-24 | 2021-08-31 | 清华大学深圳国际研究生院 | Multi-factor controllable voice conversion method and system based on feature decoupling |
| CN113889070A (en) * | 2021-09-30 | 2022-01-04 | 北京搜狗科技发展有限公司 | Voice synthesis method and device for voice synthesis |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11295721B2 (en) * | 2019-11-15 | 2022-04-05 | Electronic Arts Inc. | Generating expressive speech audio from text data |
-
2022
- 2022-01-24 CN CN202210078586.8A patent/CN114093342B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112365882A (en) * | 2020-11-30 | 2021-02-12 | 北京百度网讯科技有限公司 | Speech synthesis method, model training method, device, equipment and storage medium |
| CN113327627A (en) * | 2021-05-24 | 2021-08-31 | 清华大学深圳国际研究生院 | Multi-factor controllable voice conversion method and system based on feature decoupling |
| CN113284485A (en) * | 2021-07-09 | 2021-08-20 | 中国科学院自动化研究所 | End-to-end framework for unified Chinese and English mixed text generation and speech recognition |
| CN113889070A (en) * | 2021-09-30 | 2022-01-04 | 北京搜狗科技发展有限公司 | Voice synthesis method and device for voice synthesis |
Non-Patent Citations (2)
| Title |
|---|
| 语音合成系统中高质量的韵律生成;郭庆等;《中文信息学报》;20080315(第02期);全文 * |
| 面向航天测试的语音合成系统研究和实现技术;党建成;《中国优秀硕士学位论文全文数据库 信息科技辑》;20080615;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114093342A (en) | 2022-02-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114093342B (en) | Fine-grained rhythm modeling voice generation device, fine-grained rhythm modeling voice generation equipment and fine-grained rhythm modeling voice generation storage medium | |
| Sun et al. | Generating diverse and natural text-to-speech samples using a quantized fine-grained vae and autoregressive prosody prior | |
| Tjandra et al. | VQVAE unsupervised unit discovery and multi-scale code2spec inverter for zerospeech challenge 2019 | |
| CN107680597B (en) | Audio recognition method, device, equipment and computer readable storage medium | |
| US20220208170A1 (en) | Generating Expressive Speech Audio From Text Data | |
| JP7204989B2 (en) | Expressivity Control in End-to-End Speech Synthesis Systems | |
| Tokuda et al. | Speech synthesis based on hidden Markov models | |
| US11514888B2 (en) | Two-level speech prosody transfer | |
| Zen et al. | Statistical parametric speech synthesis using deep neural networks | |
| CN111161702B (en) | Personalized speech synthesis method and device, electronic equipment and storage medium | |
| US20210074308A1 (en) | Artificial intelligence based audio coding | |
| CN109243465A (en) | Voiceprint authentication method, device, computer equipment and storage medium | |
| CN108885870A (en) | For by combining speech to TEXT system with speech to intention system the system and method to realize voice user interface | |
| CN111179905A (en) | Rapid dubbing generation method and device | |
| JP2024510679A (en) | Unsupervised parallel tacotron non-autoregressive and controllable text reading | |
| CN112837669B (en) | Speech synthesis method, device and server | |
| CN114242033A (en) | Speech synthesis method, apparatus, device, storage medium and program product | |
| KR102198598B1 (en) | Method for generating synthesized speech signal, neural vocoder, and training method thereof | |
| CN111599339A (en) | Speech splicing synthesis method, system, device and medium with high naturalness | |
| Vasquez et al. | Hierarchical neural network structures for phoneme recognition | |
| Reddy et al. | Inverse filter based excitation model for HMM‐based speech synthesis system | |
| CN113345454B (en) | Training and application methods, devices, equipment and storage medium of voice conversion model | |
| CN113327578B (en) | Acoustic model training method and device, terminal equipment and storage medium | |
| Schmidt-Barbo et al. | Using semantic embeddings for initiating and planning articulatory speech synthesis | |
| Gao | Audio deepfake detection based on differences in human and machine generated speech |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |