CN111785247A - Voice generation method, device, equipment and computer readable medium - Google Patents
Voice generation method, device, equipment and computer readable medium Download PDFInfo
- Publication number
- CN111785247A CN111785247A CN202010667657.9A CN202010667657A CN111785247A CN 111785247 A CN111785247 A CN 111785247A CN 202010667657 A CN202010667657 A CN 202010667657A CN 111785247 A CN111785247 A CN 111785247A
- Authority
- CN
- China
- Prior art keywords
- voice
- target
- trained
- features
- acoustic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000000605 extraction Methods 0.000 claims description 39
- 239000013598 vector Substances 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 230000004044 response Effects 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 14
- 230000015572 biosynthetic process Effects 0.000 description 12
- 230000006870 function Effects 0.000 description 10
- 238000003786 synthesis reaction Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 235000009508 confectionery Nutrition 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000001755 vocal effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000001308 synthesis method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 210000001260 vocal cord Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/04—Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L13/047—Architecture of speech synthesisers
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
- G10L25/15—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being formant information
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
- G10L25/18—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being spectral information of each sub-band
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
- G10L25/24—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being the cepstrum
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
- G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Telephonic Communication Services (AREA)
Abstract
The embodiment of the disclosure discloses a voice generation method, a voice generation device, electronic equipment and a computer readable medium. One embodiment of the method comprises: determining the tone color characteristic of a target speaker based on the pre-acquired voice of the target speaker; extracting acoustic features from the target text; and generating the target voice based on the tone color characteristics and the acoustic characteristics. The embodiment ensures the universality of the vocoder by taking the acoustic characteristics and the tone characteristics as the input of the vocoder, so that the vocoder does not need to be repeatedly trained aiming at different target speakers. And, the quality of the synthesized speech is improved.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a computer-readable medium for generating speech.
Background
Driven by the internet, the voice synthesis technology is in the wake of a hot tide. However, the conventional speech synthesis method cannot ensure the universality of the vocoder, so that the vocoder needs to be repeatedly trained when a plurality of different target speakers exist. And, when the voice data of the target speaker is limited, the quality of the voice synthesis is not good.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Some embodiments of the present disclosure propose a speech generation method, apparatus, device and computer readable medium to solve the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide a method of speech generation, the method comprising: determining the tone color characteristic of a target speaker based on the pre-acquired voice of the target speaker; extracting acoustic features from the target text; and generating the target voice based on the tone color characteristics and the acoustic characteristics.
In a second aspect, some embodiments of the present disclosure provide an apparatus for generating speech, the apparatus comprising: the voice recognition method comprises a first extraction unit, a second extraction unit and a third extraction unit, wherein the first extraction unit is configured to determine the tone characteristic of a target speaker based on the voice of the target speaker acquired in advance; a second extraction unit configured to extract acoustic features from the target text; a generating unit configured to generate a target voice based on the tone color feature and the acoustic feature.
In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first aspect.
In a fourth aspect, some embodiments of the disclosure provide a computer readable medium having a computer program stored thereon, where the program when executed by a processor implements a method as described in any of the implementations of the first aspect.
One of the above-described various embodiments of the present disclosure has the following advantageous effects: the embodiment ensures the universality of the vocoder by taking the acoustic characteristics and the tone characteristics as the input of the vocoder, so that the vocoder does not need to be repeatedly trained aiming at different target speakers. And, the quality of the synthesized speech is improved.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.
FIG. 1 is a schematic diagram of one application scenario of a speech generation method of some embodiments of the present disclosure;
FIG. 2 is a flow diagram of some embodiments of a speech generation method according to the present disclosure;
FIG. 3 is a schematic diagram of another application scenario of a speech generation method according to some embodiments of the present disclosure;
FIG. 4 is a flow diagram of further embodiments of speech generation methods according to the present disclosure;
FIG. 5 is a schematic block diagram of some embodiments of a speech generating apparatus according to the present disclosure;
FIG. 6 is a schematic structural diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 shows a schematic diagram of one application scenario in which the speech generation method of some embodiments of the present disclosure may be applied.
In the application scenario illustrated in FIG. 1, first, the computing device 101 may extract the timbre characteristics 103 of the targeted speaker from the speech 102 of the targeted speaker. The computing device 101 may then extract acoustic features 105 from the target text 104. Wherein, the content of the text is 'you are called Zhang III'. Finally, the computing device 101 may generate the target speech 106 based on the timbre features 103 and the acoustic features 105.
The computing device 101 may be hardware or software. When the computing device is hardware, it may be implemented as a distributed cluster composed of a plurality of servers or electronic devices, or may be implemented as a single server or a single electronic device. When the computing device is embodied as software, it may be implemented as multiple pieces of software or software modules, for example, to provide distributed services, or as a single piece of software or software module. And is not particularly limited herein.
It should be understood that the number of computing devices 101 in FIG. 1 is merely illustrative. There may be any number of computing devices 101, as desired for implementation.
With continued reference to fig. 2, a flow 200 of some embodiments of a speech generation method according to the present disclosure is shown. The voice generation method comprises the following steps:
In some embodiments, the timbre characteristics of the target speaker may be determined by receiving manual input based on pre-acquired speech of the target speaker.
With further reference to fig. 3, fig. 3 illustrates a schematic diagram 300 of determining a timbre feature application scenario by receiving manual input in a speech generation method according to some embodiments of the present disclosure.
As shown in fig. 3, a segment of speech 301 of a target speaker is manually listened to, and tone features 302 of "sweet, beautiful", "lovely" and "soft" are extracted. Then, the extracted timbre features 302 of "sweet", "lovely", and "soft" are input to the execution main body 303. The executive 303 can then determine the timbre characteristics of the targeted speaker.
In some alternative implementations of some embodiments, the performing agent of the speech generation method may obtain the above-described timbre features by inputting the speech of the target speaker into a pre-trained speaker recognition network.
In some embodiments, the tone color characteristics may be represented by numerical values according to actual needs.
In some optional implementations of some embodiments, the above-mentioned timbre features may also be represented using a voiceprint related vector.
In some embodiments, the execution subject may perform feature analysis and extraction on the target text. Then, the feature of the text is determined as the acoustic feature. For example, the feature analysis and extraction may be performed on the text, where the punctuation mark position information for representing the pause information in the text is analyzed and extracted.
In some optional implementation manners of some embodiments, the executing subject may further input the target text into a pre-trained acoustic feature extraction network to obtain the acoustic feature.
In some embodiments, the acoustic feature extraction network may be trained by using a plurality of texts acquired in advance and a plurality of acoustic features corresponding to the plurality of texts.
In some embodiments, the acoustic features may be represented using numerical values, according to actual needs.
In some optional implementations of some embodiments, the acoustic features may also be represented using vectors.
In some embodiments, the execution subject may generate the target speech through speech synthesis software or an online speech synthesis tool.
In some optional implementations of some embodiments, the executing body may further input the timbre characteristic and the acoustic characteristic into a vocoder to obtain the target voice.
In some embodiments, the vocoder may be a speech analysis synthesis system. Such as a speech synthesis network, speech synthesis software, online speech synthesis tools, etc.
In some optional implementations of some embodiments, the vocoder may be trained by:
step one, obtaining a sample, wherein the sample comprises sample voice, sample tone color characteristics corresponding to the sample voice and sample acoustic characteristics.
And step two, inputting the sample tone color characteristics and the sample acoustic characteristics into a model to be trained to obtain voice.
And step three, analyzing the voice and the corresponding sample voice to determine a loss value of the voice.
And step four, comparing the loss value with a target value, and determining whether the model to be trained is trained according to a comparison result.
And step five, responding to the fact that the model to be trained is not trained, and adjusting relevant parameters in the training model.
And step six, in response to the fact that the training of the model to be trained is completed, determining the model to be trained as the vocoder.
Some embodiments of the present disclosure provide methods that ensure the versatility of the vocoder by using the acoustic and the chromatic characteristics as the input of the vocoder, so that the vocoder does not need to be trained repeatedly for different target speakers. And, the quality of the synthesized speech is improved.
With further reference to FIG. 4, a flow 400 of further embodiments of a speech generation method is illustrated. The process 400 of the speech generating method comprises the following steps:
In some embodiments, the speaker recognition network can be a network with any structure according to actual needs. As an example, the speaker recognition network described above may be an existing network structure. For example: a ResNet (residual Network), a VGG (Visual Geometry Group Network), and the like. As another example, the speaker recognition network may be a combination of network layer structures having specific functions.
In some embodiments, the tone color characteristics may be represented by numerical values according to actual needs.
In some optional implementations of some embodiments, the above-mentioned timbre features may also be represented using a voiceprint related vector.
And step 402, inputting the target text into a pre-trained acoustic feature extraction network to obtain acoustic features.
In some embodiments the acoustic features may include: fundamental frequency characteristics, formant characteristics, mel frequency cepstrum coefficients and the like. Wherein the fundamental frequency characteristic may be an inverse of a vocal cord vibration frequency. The formant features may be waveforms resulting from vocal tract resonances when the frequency of the acoustic excitation signal coincides with the vocal tract frequency.
In some embodiments, the above-mentioned acoustic feature extraction network may be a network having any structure according to actual needs. As an example, the above-described acoustic feature extraction network may be an existing network structure. For example: a ResNet (residual Network), a VGG (Visual Geometry Group Network), and the like. As still another example, the above-mentioned acoustic feature extraction network may also be a combination of network layer structures having specific functions.
In some embodiments, the acoustic features may be represented using numerical values, according to actual needs.
In some optional implementations of some embodiments, the acoustic features may also be represented using vectors.
And 403, inputting the tone characteristics and the acoustic characteristics into a vocoder to obtain target voice.
In some embodiments, the vocoder may be a speech analysis synthesis system. Such as a speech synthesis network, speech synthesis software, online speech synthesis tools, etc.
In some optional implementations of some embodiments, the vocoder may be trained by:
step one, obtaining a sample, wherein the sample comprises sample voice, sample tone color characteristics corresponding to the sample voice and sample acoustic characteristics.
And step two, inputting the sample tone color characteristics and the sample acoustic characteristics into a model to be trained to obtain voice.
And step three, analyzing the voice and the corresponding sample voice to determine a loss value of the voice.
And step four, comparing the loss value with a target value, and determining whether the model to be trained is trained according to a comparison result.
And step five, responding to the fact that the model to be trained is not trained, and adjusting relevant parameters in the training model.
And step six, in response to the fact that the training of the model to be trained is completed, determining the model to be trained as the vocoder.
In some embodiments, the executing body may input the tone color feature and the acoustic feature into the vocoder respectively to obtain the target voice.
In some optional implementations of some embodiments, the executing entity may further input a vector obtained by concatenating the timbre features and the acoustic features in a predetermined dimension into the vocoder, so as to obtain the target speech.
As can be seen from fig. 4, compared to fig. 2. The contents of the speaker recognition network, acoustic feature extraction network, and vocoder are added to the flow chart of FIG. 4. Thereby, more accurate timbre characteristics and acoustic characteristics can be obtained. Further, target voice with better quality and better effect is synthesized.
With further reference to fig. 5, as an implementation of the methods illustrated in the above figures, the present disclosure provides some embodiments of a speech generating apparatus, which correspond to those illustrated in fig. 2, and which may be applied in particular to various electronic devices.
As shown in fig. 5, the speech generating apparatus 500 of some embodiments includes: a first extraction unit 501, a second extraction unit 502, and a generation unit 503. The first extraction unit 501 is configured to determine a tone characteristic of a target speaker based on a pre-acquired voice of the target speaker; a second extraction unit 502 configured to extract acoustic features from the target text; a generating unit 503 configured to generate the target speech based on the timbre features and the acoustic features.
In an optional implementation of some embodiments, the first extraction unit 501 is further configured to: and inputting the voice of the target speaker into a pre-trained speaker recognition network to obtain the tone characteristics.
In an alternative implementation of some embodiments, the above-described timbre features are represented by a voiceprint related vector.
In an optional implementation of some embodiments, the second extraction unit 502 is further configured to: and inputting the target text into a pre-trained acoustic feature extraction network to obtain the acoustic features.
In an alternative implementation of some embodiments, the acoustic features are represented by vectors.
In an optional implementation of some embodiments, the generating unit 503 is further configured to: and inputting the tone features and the acoustic features into a pre-trained vocoder to obtain target voice.
In an optional implementation of some embodiments, the generating unit 503 is further configured to: and inputting a vector obtained by splicing the tone features and the acoustic features in a preset dimension into the vocoder to obtain the target voice.
In an alternative implementation of some embodiments, the vocoder is trained by: obtaining a sample, wherein the sample comprises sample voice, sample tone color characteristics and sample acoustic characteristics corresponding to the sample voice; inputting the sample tone features and the sample acoustic features into a model to be trained to obtain voice; analyzing the voice and the corresponding sample voice to determine a loss value of the voice; comparing the loss value with a target value, and determining whether the training of the model to be trained is finished according to a comparison result; in response to determining that the model to be trained is not trained, relevant parameters in the trained model are adjusted.
In an optional implementation manner of some embodiments, the foregoing step further includes: and in response to determining that the training of the model to be trained is completed, determining the model to be trained as the vocoder.
It will be understood that the elements described in the apparatus 500 correspond to various steps in the method described with reference to fig. 2. Thus, the operations, features and resulting advantages described above with respect to the method are also applicable to the apparatus 500 and the units included therein, and are not described herein again.
Referring now to fig. 6, a schematic diagram of an electronic device (e.g., the server or terminal device of fig. 1) 600 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic device in some embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle-mounted terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 6 may represent one device or may represent multiple devices as desired.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In some such embodiments, the computer program may be downloaded and installed from a network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of some embodiments of the present disclosure.
It should be noted that the computer readable medium described in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText transfer protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: extracting the tone color feature from the voice of the target speaker; generating a first acoustic characteristic of the voice of the target speaker based on the tone characteristic and a text corresponding to the voice of the target speaker; and generating the target voice based on the first acoustic characteristic of the target speaker voice and the tone characteristic.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by software, and may also be implemented by hardware. The described units may also be provided in a processor, and may be described as: a processor includes a first extraction unit, a second extraction unit, and a generation unit. Where the names of these units do not in some cases constitute a limitation on the unit itself, for example, the first extraction unit may also be described as a "unit that extracts timbre features".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
According to one or more embodiments of the present disclosure, there is provided a speech generation method including: determining the tone color characteristic of a target speaker based on the pre-acquired voice of the target speaker; extracting acoustic features from the target text; and generating the target voice based on the tone color characteristics and the acoustic characteristics.
According to one or more embodiments of the present disclosure, extracting the above-mentioned timbre features from the voice of the target speaker comprises: and inputting the voice of the target speaker into a pre-trained speaker recognition network to obtain the tone characteristics.
According to one or more embodiments of the present disclosure, the above-mentioned timbre features are represented by a voiceprint related vector.
According to one or more embodiments of the present disclosure, extracting acoustic features from target text comprises: and inputting the target text into a pre-trained acoustic feature extraction network to obtain the acoustic features.
According to one or more embodiments of the present disclosure, the acoustic features are represented by vectors.
According to one or more embodiments of the present disclosure, based on the above-described timbre features and the above-described acoustic features, target speech is generated: and inputting the tone features and the acoustic features into a pre-trained vocoder to obtain target voice.
According to one or more embodiments of the present disclosure, the inputting the tone color feature and the acoustic feature into a pre-trained vocoder to obtain a target voice includes: and inputting a vector obtained by splicing the tone features and the acoustic features in a preset dimension into the vocoder to obtain the target voice.
According to one or more embodiments of the present disclosure, the vocoder is trained by the following steps: obtaining a sample, wherein the sample comprises sample voice, sample tone color characteristics and sample acoustic characteristics corresponding to the sample voice; inputting the sample tone features and the sample acoustic features into a model to be trained to obtain voice; analyzing the voice and the corresponding sample voice to determine a loss value of the voice; comparing the loss value with a target value, and determining whether the training of the model to be trained is finished according to a comparison result; in response to determining that the model to be trained is not trained, relevant parameters in the trained model are adjusted.
According to one or more embodiments of the present disclosure, the above steps further include: and in response to determining that the training of the model to be trained is completed, determining the model to be trained as the vocoder.
According to one or more embodiments of the present disclosure, there is provided a speech generating apparatus including: the voice recognition method comprises a first extraction unit, a second extraction unit and a third extraction unit, wherein the first extraction unit is configured to determine the tone characteristic of a target speaker based on the voice of the target speaker acquired in advance; a second extraction unit configured to extract acoustic features from the target text; a generating unit configured to generate a target voice based on the tone color feature and the acoustic feature.
According to one or more embodiments of the present disclosure, the first extraction unit is further configured to: and inputting the voice of the target speaker into a pre-trained speaker recognition network to obtain the tone characteristics.
In an alternative implementation of some embodiments, the above-described timbre features are represented by a voiceprint related vector.
According to one or more embodiments of the present disclosure, the second extraction unit is further configured to: and inputting the target text into a pre-trained acoustic feature extraction network to obtain the acoustic features.
In an alternative implementation of some embodiments, the acoustic features are represented by vectors.
According to one or more embodiments of the present disclosure, the generating unit is further configured to: and inputting the tone features and the acoustic features into a pre-trained vocoder to obtain target voice.
In an optional implementation of some embodiments, the generating unit 503 is further configured to: and inputting a vector obtained by splicing the tone features and the acoustic features in a preset dimension into the vocoder to obtain the target voice.
In an alternative implementation of some embodiments, the vocoder is trained by: obtaining a sample, wherein the sample comprises sample voice, sample tone color characteristics and sample acoustic characteristics corresponding to the sample voice; inputting the sample tone features and the sample acoustic features into a model to be trained to obtain voice; analyzing the voice and the corresponding sample voice to determine a loss value of the voice; comparing the loss value with a target value, and determining whether the training of the model to be trained is finished according to a comparison result; in response to determining that the model to be trained is not trained, relevant parameters in the trained model are adjusted.
In an optional implementation manner of some embodiments, the foregoing step further includes: and in response to determining that the training of the model to be trained is completed, determining the model to be trained as the vocoder.
According to one or more embodiments of the present disclosure, there is provided an electronic device including: one or more processors; a storage device having one or more programs stored thereon which, when executed by one or more processors, cause the one or more processors to implement a method as in any above.
According to one or more embodiments of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the method as any one of the above.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (12)
1. A method of speech generation comprising:
determining the tone characteristic of a target speaker based on the pre-acquired voice of the target speaker;
extracting acoustic features from the target text;
and generating target voice based on the tone color characteristics and the acoustic characteristics.
2. The method of claim 1, wherein the determining the timbre characteristics of the target speaker based on the pre-acquired speech of the target speaker comprises:
and inputting the voice of the target speaker into a pre-trained speaker recognition network to obtain the tone characteristic.
3. The method of claim 1, wherein the timbre features are represented by a voiceprint related vector.
4. The method of claim 1, wherein the extracting acoustic features from the target text comprises:
and inputting the target text into a pre-trained acoustic feature extraction network to obtain the acoustic features.
5. The method of claim 1, wherein the acoustic features are represented by vectors.
6. The method of claim 1, wherein the generating target speech based on the timbre features and the acoustic features comprises:
and inputting the tone features and the acoustic features into a vocoder to obtain target voice.
7. The method of claim 6, wherein the inputting the timbre features and the acoustic features into a pre-trained vocoder to obtain a target speech, comprises:
and inputting a vector obtained by splicing the tone features and the acoustic features in a preset dimension into the vocoder to obtain target voice.
8. The method of claim 6, wherein the vocoder is trained by:
obtaining a sample, wherein the sample comprises sample voice, sample tone color characteristics and sample acoustic characteristics corresponding to the sample voice;
inputting the sample tone characteristic and the sample acoustic characteristic into a model to be trained to obtain voice;
analyzing the voice and the corresponding sample voice to determine a loss value of the voice;
comparing the loss value with a target value, and determining whether the model to be trained is trained according to a comparison result;
in response to determining that the model to be trained is not trained, relevant parameters in the trained model are adjusted.
9. The method of claim 8, wherein the steps further comprise:
in response to determining that training of the model to be trained is complete, determining the model to be trained as the vocoder.
10. A speech generating apparatus comprising:
the voice recognition method comprises a first extraction unit, a second extraction unit and a third extraction unit, wherein the first extraction unit is configured to determine the tone characteristic of a target speaker based on the voice of the target speaker acquired in advance;
a second extraction unit configured to extract acoustic features from the target text;
a generating unit configured to generate a target voice based on the timbre features and the acoustic features.
11. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-9.
12. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010667657.9A CN111785247A (en) | 2020-07-13 | 2020-07-13 | Voice generation method, device, equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010667657.9A CN111785247A (en) | 2020-07-13 | 2020-07-13 | Voice generation method, device, equipment and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111785247A true CN111785247A (en) | 2020-10-16 |
Family
ID=72768078
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010667657.9A Pending CN111785247A (en) | 2020-07-13 | 2020-07-13 | Voice generation method, device, equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111785247A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112382270A (en) * | 2020-11-13 | 2021-02-19 | 北京有竹居网络技术有限公司 | Speech synthesis method, apparatus, device and storage medium |
| CN112614477A (en) * | 2020-11-16 | 2021-04-06 | 北京百度网讯科技有限公司 | Multimedia audio synthesis method and device, electronic equipment and storage medium |
| CN112927674A (en) * | 2021-01-20 | 2021-06-08 | 北京有竹居网络技术有限公司 | Voice style migration method and device, readable medium and electronic equipment |
| CN114360491A (en) * | 2021-12-29 | 2022-04-15 | 腾讯科技（深圳）有限公司 | Speech synthesis method, speech synthesis device, electronic equipment and computer-readable storage medium |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105427855A (en) * | 2015-11-09 | 2016-03-23 | 上海语知义信息技术有限公司 | Voice broadcast system and voice broadcast method of intelligent software |
| CN107452369A (en) * | 2017-09-28 | 2017-12-08 | 百度在线网络技术（北京）有限公司 | Phonetic synthesis model generating method and device |
| CN107481713A (en) * | 2017-07-17 | 2017-12-15 | 清华大学 | A kind of hybrid language phoneme synthesizing method and device |
| CN108182936A (en) * | 2018-03-14 | 2018-06-19 | 百度在线网络技术（北京）有限公司 | Voice signal generation method and device |
| CN108806665A (en) * | 2018-09-12 | 2018-11-13 | 百度在线网络技术（北京）有限公司 | Phoneme synthesizing method and device |
| CN109523989A (en) * | 2019-01-29 | 2019-03-26 | 网易有道信息技术（北京）有限公司 | Phoneme synthesizing method, speech synthetic device, storage medium and electronic equipment |
| CN109686361A (en) * | 2018-12-19 | 2019-04-26 | 深圳前海达闼云端智能科技有限公司 | A kind of method, apparatus of speech synthesis calculates equipment and computer storage medium |
| US20190221201A1 (en) * | 2017-02-21 | 2019-07-18 | Tencent Technology (Shenzhen) Company Limited | Speech conversion method, computer device, and storage medium |
| CN110136690A (en) * | 2019-05-22 | 2019-08-16 | 平安科技（深圳）有限公司 | Phoneme synthesizing method, device and computer readable storage medium |
| CN110136693A (en) * | 2018-02-09 | 2019-08-16 | 百度（美国）有限责任公司 | System and method for using a small amount of sample to carry out neural speech clone |
| CN110444191A (en) * | 2019-01-22 | 2019-11-12 | 清华大学深圳研究生院 | A kind of method, the method and device of model training of prosody hierarchy mark |
| CN110930975A (en) * | 2018-08-31 | 2020-03-27 | 百度在线网络技术（北京）有限公司 | Method and apparatus for outputting information |
| CN111048064A (en) * | 2020-03-13 | 2020-04-21 | 同盾控股有限公司 | Voice cloning method and device based on single speaker voice synthesis data set |
-
2020
- 2020-07-13 CN CN202010667657.9A patent/CN111785247A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105427855A (en) * | 2015-11-09 | 2016-03-23 | 上海语知义信息技术有限公司 | Voice broadcast system and voice broadcast method of intelligent software |
| US20190221201A1 (en) * | 2017-02-21 | 2019-07-18 | Tencent Technology (Shenzhen) Company Limited | Speech conversion method, computer device, and storage medium |
| CN107481713A (en) * | 2017-07-17 | 2017-12-15 | 清华大学 | A kind of hybrid language phoneme synthesizing method and device |
| CN107452369A (en) * | 2017-09-28 | 2017-12-08 | 百度在线网络技术（北京）有限公司 | Phonetic synthesis model generating method and device |
| CN110136693A (en) * | 2018-02-09 | 2019-08-16 | 百度（美国）有限责任公司 | System and method for using a small amount of sample to carry out neural speech clone |
| CN108182936A (en) * | 2018-03-14 | 2018-06-19 | 百度在线网络技术（北京）有限公司 | Voice signal generation method and device |
| CN110930975A (en) * | 2018-08-31 | 2020-03-27 | 百度在线网络技术（北京）有限公司 | Method and apparatus for outputting information |
| CN108806665A (en) * | 2018-09-12 | 2018-11-13 | 百度在线网络技术（北京）有限公司 | Phoneme synthesizing method and device |
| CN109686361A (en) * | 2018-12-19 | 2019-04-26 | 深圳前海达闼云端智能科技有限公司 | A kind of method, apparatus of speech synthesis calculates equipment and computer storage medium |
| CN110444191A (en) * | 2019-01-22 | 2019-11-12 | 清华大学深圳研究生院 | A kind of method, the method and device of model training of prosody hierarchy mark |
| CN109523989A (en) * | 2019-01-29 | 2019-03-26 | 网易有道信息技术（北京）有限公司 | Phoneme synthesizing method, speech synthetic device, storage medium and electronic equipment |
| CN110136690A (en) * | 2019-05-22 | 2019-08-16 | 平安科技（深圳）有限公司 | Phoneme synthesizing method, device and computer readable storage medium |
| CN111048064A (en) * | 2020-03-13 | 2020-04-21 | 同盾控股有限公司 | Voice cloning method and device based on single speaker voice synthesis data set |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112382270A (en) * | 2020-11-13 | 2021-02-19 | 北京有竹居网络技术有限公司 | Speech synthesis method, apparatus, device and storage medium |
| CN112614477A (en) * | 2020-11-16 | 2021-04-06 | 北京百度网讯科技有限公司 | Multimedia audio synthesis method and device, electronic equipment and storage medium |
| CN112614477B (en) * | 2020-11-16 | 2023-09-12 | 北京百度网讯科技有限公司 | Method and device for synthesizing multimedia audio, electronic equipment and storage medium |
| CN112927674A (en) * | 2021-01-20 | 2021-06-08 | 北京有竹居网络技术有限公司 | Voice style migration method and device, readable medium and electronic equipment |
| WO2022156413A1 (en) * | 2021-01-20 | 2022-07-28 | 北京有竹居网络技术有限公司 | Speech style migration method and apparatus, readable medium and electronic device |
| CN112927674B (en) * | 2021-01-20 | 2024-03-12 | 北京有竹居网络技术有限公司 | Voice style migration method and device, readable medium and electronic equipment |
| CN114360491A (en) * | 2021-12-29 | 2022-04-15 | 腾讯科技（深圳）有限公司 | Speech synthesis method, speech synthesis device, electronic equipment and computer-readable storage medium |
| CN114360491B (en) * | 2021-12-29 | 2024-02-09 | 腾讯科技（深圳）有限公司 | Speech synthesis method, device, electronic equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111933110B (en) | Video generation method, generation model training method, device, medium and equipment | |
| WO2022105545A1 (en) | Speech synthesis method and apparatus, and readable medium and electronic device | |
| CN111785247A (en) | Voice generation method, device, equipment and computer readable medium | |
| CN111369971B (en) | Speech synthesis method, device, storage medium and electronic equipment | |
| CN111599343B (en) | Method, apparatus, device and medium for generating audio | |
| CN112786006B (en) | Speech synthesis method, synthesis model training method, device, medium and equipment | |
| CN112489621B (en) | Speech synthesis method, device, readable medium and electronic equipment | |
| CN111798821B (en) | Sound conversion method, device, readable storage medium and electronic equipment | |
| CN111916053B (en) | Voice generation method, device, equipment and computer readable medium | |
| CN111986655B (en) | Audio content identification method, device, equipment and computer readable medium | |
| CN111597825B (en) | Voice translation method and device, readable medium and electronic equipment | |
| CN111462728A (en) | Method, apparatus, electronic device and computer readable medium for generating speech | |
| CN113257218B (en) | Speech synthesis method, device, electronic equipment and storage medium | |
| CN112364144B (en) | Interaction method, device, equipment and computer readable medium | |
| CN112509562A (en) | Method, apparatus, electronic device and medium for text post-processing | |
| CN111462727A (en) | Method, apparatus, electronic device and computer readable medium for generating speech | |
| CN111667810A (en) | Method and device for acquiring polyphone corpus, readable medium and electronic equipment | |
| CN111968657B (en) | Voice processing method and device, electronic equipment and computer readable medium | |
| CN114495901A (en) | Speech synthesis method, speech synthesis device, storage medium and electronic equipment | |
| CN115312079A (en) | Information display method and device, electronic equipment and computer readable medium | |
| CN114242035A (en) | Speech synthesis method, apparatus, medium, and electronic device | |
| CN112017685A (en) | Voice generation method, device, equipment and computer readable medium | |
| CN113221514A (en) | Text processing method and device, electronic equipment and storage medium | |
| CN113241061B (en) | Method and device for processing voice recognition result, electronic equipment and storage medium | |
| CN111292766B (en) | Method, apparatus, electronic device and medium for generating voice samples |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |