JP2023171108A - Voice conversion device, voice conversion method and program - Google Patents

Abstract

To convert only vocalization skill while preserving a speaker characteristic by changing only a dynamic feature amount of a voice feature amount.SOLUTION: A voice conversion device (1) includes: a model learning unit (11) for learning a voice conversion model which converts a voice feature amount of a conversion source speaker into a voice feature amount of a conversion object speaker; a voice conversion unit (12) for inputting the voice feature amount of the conversion source speaker to a trained voice conversion model and converting the voice feature amount into the voice feature amount of the conversion object speaker; a dynamic feature amount conversion unit (13) for converting the voice feature amount of the conversion source speaker into a converted voice feature amount by using the dynamic feature amount of the voice feature amount of the conversion source speaker and the dynamic feature amount of the voice feature amount of the conversion object speaker; and a voice waveform generation unit (14) for generating a voice waveform from the converted voice feature amount.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a voice conversion device, a voice conversion method, and a program that convert the voice pronunciation skill of an input speaker.

Traditionally, the vocal skills of vocal experts such as announcers and voice actors differ greatly from those of other amateurs. In the present disclosure, the vocalization skill refers to an index indicating the ease of audibility of the voice uttered by the speaker. For example, announcements made by amateurs on station announcements, building announcements, etc. can be difficult to hear, so technology that converts only the vocal skill without changing the identity of the speaker of the voice is needed. is necessary. In the present disclosure, speaker characteristics collectively refer to acoustic features represented by a spectrum included in speech, and prosodic features represented by pitch, speech rhythm, and the like.

FIG. 6 is a block diagram showing an example of the configuration of a conventional speech conversion device. Conventionally, speech (voice quality) conversion is a technique for converting input speech features of a source speaker into speech features of a target speaker. As shown in Figure 6, the conversion from the voice features of the source speaker to the voice features of the target speaker is performed using a voice conversion model trained using a voice conversion algorithm. This is done by entering the amount. For example, Non-Patent Document 1 describes a speech conversion algorithm that converts speech between two arbitrary speakers using vector quantization. Furthermore, Non-Patent Document 2 describes a speech conversion algorithm that converts speech between two arbitrary speakers using an artificial neural network (ANN). When using the algorithms disclosed in Non-Patent Document 1 and Non-Patent Document 2, the voices of two speakers need to be parallel data (sounds produced by two speakers uttering the same utterance). be. On the other hand, Non-Patent Document 3 describes a voice conversion algorithm using VAE (variational autoencoder) that can utilize voices that do not require the voices of two speakers to be parallel data. .

Next, as shown in FIG. 6, a speech waveform is generated from the speech features of the conversion target speaker using a speech synthesis algorithm. Non-Patent Document 4 describes a speech synthesis algorithm using a Mel-Log Spectrum Approximation (MLSA) filter.

Further, as algorithms that can be used in the present disclosure, Non-Patent Document 5 describes a parameter generation algorithm using dynamic features, and Non-Patent Document 6 describes a multiple regression mixed normal distribution model.

Abe, Masanobu, et al. “Voice conversion through vector quantization.” Journal of the Acoustical Society of Japan (E) 11.2 (1990): 71-76 Desai, Srinivas, et al. “Spectral mapping using artificial neural networks for voice conversion.” IEEE Transactions on Audio, Speech, and Language Processing 18.5 (2010): 954-964. Hsu, Chin-Cheng, et al. “Voice conversion from non-parallel corpora using variational auto-encoder.” Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2016 Asia-Pacific. IEEE, 2016. Kiyoshi Imai and two others, "Mel-logarithm spectral approximation (MLSA) filter for speech synthesis," IEICE Transactions A Vol.J66-A No.2 pp.122-129, Feb. 1983. Takashi Mashiko and three others, "Speech synthesis based on HMM using dynamic features", IEICE Journal, vol.J79-D-II, no.12, pp.2184-2190, Dec. 1996. Kumi Ota, "Voice quality conversion/control method based on multiple regression mixed normal distribution model", Master's thesis, Nara Institute of Science and Technology, 2008.

However, according to the conventional speech conversion device using the speech conversion algorithm disclosed in Non-Patent Document 1 or Non-Patent Document 2, the speaker characteristics of the conversion source speaker are not the same as the speaker characteristics of the conversion target speaker. There was a problem in that it was not possible to convert only the vocal skills of the original speaker while preserving the speaker characteristics of the original speaker.

Therefore, in the present disclosure, only the vocal skill is converted by considering the vocal skill as the quality of the smooth tongue and converting only the temporal fluctuation of the voice feature amount of the voice of an amateur speaker to that of an expert. We focused on the technology to do this.

The present disclosure was made in view of the above circumstances, and an object of the present disclosure is to provide a speech conversion device that converts only the vocal skill while preserving the speaker characteristics by converting only the temporal fluctuations (dynamic features) of the speech features; The purpose of this invention is to provide a voice conversion method and program.

In order to solve the above problems, the speech conversion device according to the present embodiment is a speech conversion device that converts the dynamic feature amount of the speech feature amount of the speaker, and converts the speech feature amount of the source speaker into the speech feature amount of the conversion target. a model learning unit that learns a voice conversion model to convert into voice features of the speaker; and a model learning unit that learns a voice conversion model that converts the voice features of the conversion source speaker into the learned voice conversion model, and inputs the voice features of the conversion source speaker to the trained voice conversion model to generate voice features of the conversion target speaker. A speech conversion unit that converts the speech of the conversion source speaker into The apparatus includes a dynamic feature converter that converts a feature into a converted audio feature, and an audio waveform generator that generates an audio waveform from the converted audio feature.

In order to solve the above problems, the voice conversion method according to the present embodiment is a voice conversion method that converts the dynamic feature amount of the voice feature amount of the speaker. a step of learning a speech conversion model that converts the voice features of the conversion target speaker into speech features of the conversion target speaker; a step of converting the speech feature into a voice feature, inputting the dynamic feature of the voice feature of the conversion source speaker and the dynamic feature of the voice feature of the conversion target speaker; The method includes the steps of converting the voice feature amount into a converted voice feature amount, and generating a voice waveform from the converted voice feature amount.

In order to solve the above problem, a program according to this embodiment causes a computer to function as the above speech conversion device.

According to the present disclosure, by converting only the temporal variation of the voice feature amount (dynamic feature amount), it is possible to convert only the vocal skill while retaining speaker characteristics.

FIG. 1 is a block diagram illustrating a configuration example of a speech conversion device according to a first embodiment. 2 is a flowchart illustrating an example of a voice conversion method executed by the voice conversion device according to the first embodiment. FIG. 2 is a block diagram showing a configuration example of a speech conversion device according to a second embodiment. It is a flow chart which shows an example of the voice conversion method performed by the voice conversion device concerning a 2nd embodiment. FIG. 1 is a block diagram showing a schematic configuration of a computer functioning as a voice conversion device. FIG. 1 is a block diagram showing a configuration example of a conventional speech conversion device.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. The present invention is not limited to the following embodiments, and can be implemented with various modifications within the scope of the gist.

(First embodiment)
FIG. 1 is a block diagram showing a configuration example of a speech conversion device 1 according to the first embodiment. As shown in FIG. 1, the speech conversion device 1 according to the first embodiment includes a model learning section 11, a speech conversion section 12, a dynamic feature amount conversion section 13, and a speech waveform generation section 14. . The speech conversion device 1 converts the dynamic feature amount of the speaker's speech feature amount. The model learning section 11, the speech conversion section 12, the dynamic feature amount conversion section 13, and the speech waveform generation section 14 constitute a control calculation circuit (controller). The control calculation circuit may be composed of dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be composed of a processor, or may be composed of both. may be done.

The model learning unit 11 learns a voice conversion model that converts voice features of a conversion source speaker stored in advance in the voice storage unit 15 into voice features of a conversion target speaker. The model learning unit 11 uses a speech conversion algorithm using vector quantization described in Non-Patent Document 1, a speech conversion algorithm using an artificial neural network (ANN) described in Non-Patent Document 2, or a speech conversion algorithm using an artificial neural network (ANN) described in Non-Patent Document 3. A speech conversion algorithm using a VAE (variational autoencoder) described in 1.2 may be used as the learning algorithm.

The voice handled by the model learning unit 11 is the voice feature amount (pitch parameter (fundamental frequency, etc.), spectral parameter (cepstrum, mel cepstrum, etc.)) obtained as a result of performing Fourier transform, signal processing, etc. on the voice signal. It is held in the audio storage unit 15 as a. In the present disclosure, audio features obtained by Fourier transform, signal processing, etc. (generally referred to as static features) are not only static features, but also from one frame (audio frame) before each time. It is assumed that the dynamic feature amount that captures the temporal change after one frame is also included. When using the voice conversion algorithm described in Non-Patent Document 1 or Non-Patent Document 2 mentioned above, the voice needs to be parallel data (sound produced by two speakers uttering the same utterance). In addition, when using the above algorithm, it is necessary to determine the correspondence of the time information of each speaker's voice in advance by DP matching (DTW; Dynamic Time Warping) or the like. On the other hand, when using the voice conversion algorithm described in Non-Patent Document 3, the voice does not need to be parallel data, and there is no need to associate time information.

The speech conversion section 12 inputs the speech feature amount 21 of the conversion source speaker into the trained speech conversion model 11a generated by the model learning section 11, and converts it into the speech feature amount 22 of the conversion target speaker.

The dynamic feature converter 13 converts the voice of the source speaker using the dynamic feature of the voice feature 21 of the source speaker and the dynamic feature of the voice feature 22 of the target speaker. The feature amount 21 is converted into a post-conversion audio feature amount 23.

The dynamic feature amount conversion unit 13 replaces the dynamic feature amount of the voice feature amount 21 of the conversion source speaker with the dynamic feature amount of the voice feature amount 22 of the conversion target speaker, and converts the voice feature amount of the conversion target speaker into a voice feature amount of the conversion target speaker. The converted dynamic feature amount may be generated by treating the dynamic feature amount 22 as the dynamic feature amount of the speech feature amount 21 of the conversion source speaker. In addition, by calculating the weighted sum of the dynamic feature quantity of the speech feature quantity 21 of the conversion source speaker and the dynamic feature quantity of the speech feature quantity 22 of the conversion target speaker for each speech frame, the post-conversion dynamic A feature quantity may also be generated. In the latter case, it is determined how much importance is given to the vocal skill of the target speaker by weighting the dynamic feature values of the voice feature value 21 of the conversion source speaker and the dynamic feature value of the voice feature value 22 of the target speaker. You can specify whether to perform the specified conversion. Thereafter, for example, using a parameter generation algorithm using dynamic features described in Non-Patent Document 5, the speech feature amount 21 of the conversion source speaker is converted to the speech feature amount 21 of the conversion target speaker using the post-conversion dynamic feature amount. The feature amount 22 is converted into a post-conversion audio feature amount 23 that reflects the dynamic feature amount.

The audio waveform generation unit 14 generates an audio waveform 24 from the converted audio feature amount 23. The speech waveform generation unit 14 may generate the speech waveform 24 using a speech synthesis algorithm using a Mel-Log Spectrum Approximation (MLSA) filter described in Non-Patent Document 4. .

The voice storage unit 15 records (holds) the voices uttered by two speakers who are the targets of conversion as voice features, and converts the voice features to the voice learning unit 11 in response to a request from the voice learning unit 11. Output to.

FIG. 2 is a flowchart illustrating an example of a voice conversion method executed by the voice conversion device 1 according to the first embodiment.

In step S101, the model learning unit 11 learns a speech conversion model 11a that converts the speech feature amount 21 of the conversion source speaker into the speech feature amount 22 of the conversion target speaker.

In step S102, the speech conversion unit 12 inputs the speech feature amount 21 of the conversion source speaker to the trained speech conversion model 11a, and converts it into the speech feature amount 22 of the conversion target speaker.

In step S103, the dynamic feature amount conversion unit 13 uses the dynamic feature amount of the voice feature amount 21 of the conversion source speaker and the dynamic feature amount of the voice feature amount 22 of the conversion target speaker. The speaker's voice feature amount 21 is converted into a converted voice feature amount 23.

In step S104, the audio waveform generation unit 14 generates the audio waveform 24 from the converted audio feature amount 23.

Unlike the conventional techniques described in Non-Patent Documents 1 to 3, which convert the entire audio feature amount, the audio conversion device 1 according to the present embodiment converts the dynamic feature amount (temporal feature amount) of the audio feature amount. (variation) is subject to conversion. Thereby, according to the voice conversion device 1, it is possible to convert only the pronunciation skill, such as the quality of the tongue, without changing the speaker characteristics of the voice. In addition, by setting the conversion source speaker to be a vocal amateur and the conversion target speaker to be a vocal expert such as an announcer or voice actor, it is possible to improve the vocal skills of the amateur (conversion source speaker) to that of an expert (conversion target speaker). It will be possible to get closer to vocal skills.

(Second embodiment)
FIG. 3 is a block diagram showing a configuration example of a speech conversion device 1' according to the second embodiment. As shown in FIG. 3, the speech conversion device 1' according to the second embodiment includes a model learning section 11', a speech conversion section 12', a dynamic feature amount conversion section 13, and a speech waveform generation section 14. , is provided. The speech conversion device 1' converts the dynamic feature amount of the speaker's speech feature amount. The voice conversion device 1' according to this embodiment differs from the voice conversion device 1 according to the first embodiment in the functions possessed by the model learning section 11' and the voice conversion section 12'. The features converter 13 and audio waveform generator 14 have the same functions. The same configurations as in the first embodiment are given the same reference numbers as in the first embodiment, and the description thereof will be omitted as appropriate. The model learning section 11', the voice conversion section 12', the dynamic feature amount conversion section 13, and the voice waveform generation section 14 constitute a control calculation circuit (controller). The control calculation circuit may be composed of dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be composed of a processor, or may be composed of both. may be done.

The model learning unit 11' inputs the voice features of a plurality of speakers and the pronunciation skills assigned to each speaker, and arbitrarily selects the voice features of one speaker determined as the conversion source speaker. A plurality of speech conversion models are learned that convert the speech features of a plurality of other speakers determined as conversion target speakers, respectively. The model learning unit 11' converts the voice feature amount 21 of one speaker determined as the conversion source speaker among the plurality of voice conversion models into one having a vocalization skill that matches an arbitrarily determined target vocalization skill 25. One speech conversion model 11a' that converts the name into the speech feature amount 22 of the speaker to be converted is held. For example, when the voice features of voices uttered by 10 speakers and the vocalization skills assigned to each of the 10 speakers are input, the model learning unit 11' Nine types of speech conversion models are learned that convert the voice features of the conversion source speaker into the voice features of nine other conversion target speakers, and then the voice features of the one conversion source speaker are learned. Only one voice conversion model 11a' is retained that converts 21 into the voice feature amount 22 of one conversion target speaker among the nine speakers whose voice skill matches the separately arbitrarily determined target voice skill 25. The learning algorithm may use the multiple regression mixed normal distribution model described in Non-Patent Document 6. The multiple regression mixed normal distribution model described in Non-Patent Document 6 proposes a technique for converting any voice quality (from a thick voice to a thin voice, etc.) as an extension of conventional voice conversion. Now, by assigning and learning vocalization skills instead of voice quality, conversion to any vocalization skill is performed.

The voice conversion unit 12' inputs the voice features 21 of the conversion source speaker and the target pronunciation skill 25 to the voice conversion model 11a' trained by the model learning unit 11', and converts the voice features of the conversion source speaker into the voice conversion model 11a'. The quantity 21 is converted into the voice feature quantity 22 of the conversion target speaker whose pronunciation skill matches the target pronunciation skill 25.

The dynamic feature amount conversion unit 13 and the audio waveform generation unit 14 included in the speech conversion device 1′ are the same as the dynamic feature amount conversion unit 13 and the audio waveform generation unit 14 included in the speech conversion device 1 according to the first embodiment. It is. The dynamic feature converter 13 converts the voice of the source speaker using the dynamic feature of the voice feature 21 of the source speaker and the dynamic feature of the voice feature 22 of the target speaker. The feature amount 21 is converted into a post-conversion audio feature amount 23. The audio waveform generation unit 14 generates an audio waveform 24 from the converted audio feature amount 23.

While the voice storage unit 15 according to the first embodiment stores voice features of voices uttered by two speakers, the voice storage unit 15' stores voice features of voices uttered by a plurality of speakers (for example, 10 people). It records the audio features of voices uttered by more speakers than others) and the vocal skills assigned to each speaker. It is desirable to use a numerical representation of the subjective score assigned to each speaker based on listening by the evaluator (for example, 1: extremely low skill... 5: extremely high skill) for the speaking skill. The voice storage unit 15' records (holds) the voice features of voices uttered by a plurality of speakers to be converted and the vocalization skills assigned to each speaker, and the voice learning unit 11' In response to the request, the voice feature amount and the vocalization skill are output to the voice learning section 11'.

FIG. 4 is a flowchart illustrating an example of a voice conversion method executed by the voice conversion device 1' according to the second embodiment.

In step S201, the model learning unit 11' learns a plurality of speech conversion models 11a' that convert the speech feature amount 21 of the conversion source speaker into the speech feature amount 22 of a plurality of other conversion target speakers. Further, the model learning unit 11' creates a voice conversion model 11a' that converts the voice feature amount 21 of the conversion source speaker into the voice feature amount 22 of the conversion target speaker whose voice skill matches the target voice skill 25. only hold.

In step S202, the voice conversion unit 12' inputs the voice feature amount 21 of the conversion source speaker and the target vocalization skill 25 to the learned voice conversion model 11a', and converts the voice feature amount 21 of the conversion source speaker into the learned voice conversion model 11a'. It is converted into a voice feature amount 22 of a conversion target speaker whose voice skill matches the target voice skill 25.

In step S203, the dynamic feature amount conversion unit 13 uses the dynamic feature amount of the voice feature amount 21 of the conversion source speaker and the dynamic feature amount of the voice feature amount 22 of the conversion target speaker. The speaker's voice feature amount 21 is converted into a converted voice feature amount 23.

In step S204, the audio waveform generation unit 14 generates the audio waveform 24 from the converted audio feature amount 23.

According to the speech conversion device 1 according to the first embodiment, it is assumed that a speaker with a high pronunciation skill is an expert in pronunciation, and the pronunciation skill of one speaker (amateur in pronunciation) is compared with that of the other speaker. Convert to the vocal skills of the speaker (pronunciation expert). However, in reality, some vocal amateurs have high vocal skills, and even vocal experts have different vocal skills. According to the speech conversion device 1' according to the present disclosure, the speech features of the conversion source speaker are arbitrarily determined by using the speech features of a plurality of speakers and the vocalization skills assigned to each speaker. can be converted into the voice feature amount of a conversion target speaker who has a vocalization skill that matches an arbitrary target vocalization skill.

It is also possible to use a computer capable of executing program instructions in order to function the above-mentioned speech conversion devices 1 and 1'. FIG. 5 is a block diagram showing a schematic configuration of a computer functioning as a voice conversion device. Here, the computers functioning as the voice conversion devices 1 and 1' may be general-purpose computers, special-purpose computers, workstations, PCs (Personal Computers), electronic notepads, or the like. Program instructions may be program code, code segments, etc. to perform necessary tasks.

As shown in FIG. 5, the computer 100 includes a processor 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, and a storage 140 as storage units, an input unit 150, an output unit 160, and communication units. An interface (I/F) 170 is provided. Each configuration is communicably connected to each other via a bus 180.

The ROM 120 stores various programs and data. The RAM 130 temporarily stores programs or data as a work area. The storage 140 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data. In the present disclosure, a program according to the present disclosure is stored in the ROM 120 or the storage 140.

Specifically, the processor 110 is a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), SoC (System on a Chip), etc., and may be of the same type or different types. It may be configured with a plurality of processors. The processor 110 reads a program from the ROM 120 or the storage 140 and executes the program using the RAM 130 as a work area, thereby controlling each of the above components and performing various calculation processes. Note that at least a part of these processing contents may be realized by hardware.

The program may be recorded on a recording medium readable by the speech conversion devices 1 and 1'. By using such a recording medium, it is possible to install it in the voice conversion devices 1 and 1'. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, a USB (Universal Serial Bus) memory, or the like. Further, this program may be downloaded from an external device via a network.

Although the embodiments described above have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of this disclosure. Therefore, the present invention should not be construed as being limited to the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, it is possible to combine a plurality of configuration blocks described in the configuration diagram of the embodiment into one, or to divide one configuration block.

1, 1' Voice conversion device
11, 11' Model learning section
11a, 11a' Voice conversion model
12, 12' Voice converter
13 Dynamic feature converter
14 Audio waveform generation section 15, 15' Audio storage section
21 Voice features of conversion source speaker
22 Voice features of conversion target speaker
23 Post-conversion voice feature amount 24 Voice waveform 25 Target vocal skill 100 Computer 110 Processor 120 ROM
130 RAM
140 Storage 150 Input section 160 Output section 170 Communication interface (I/F)
180 bus

Claims (6)

A voice conversion device that converts a dynamic feature of a speaker's voice feature,
a model learning unit that learns a voice conversion model that converts voice features of a conversion source speaker to voice features of a conversion target speaker;
a voice conversion unit that inputs voice features of the conversion source speaker into a trained voice conversion model and converts them into voice features of the conversion target speaker;
Using the dynamic feature amount of the voice feature amount of the conversion source speaker and the dynamic feature amount of the voice feature amount of the conversion target speaker, the voice feature amount of the conversion source speaker is converted into a post-conversion voice feature amount. a dynamic feature conversion unit that converts into
an audio waveform generation unit that generates an audio waveform from the converted audio feature;
A voice conversion device comprising: The dynamic feature amount conversion unit replaces the dynamic feature amount of the voice feature amount of the conversion source speaker with the dynamic feature amount of the voice feature amount of the conversion target speaker, thereby converting the voice feature amount after conversion. The speech conversion device according to claim 1, which generates. The dynamic feature converter calculates a weighted sum of a dynamic feature of the speech feature of the conversion source speaker and a dynamic feature of the speech feature of the conversion target speaker for each audio frame. 2. The method according to claim 1, wherein a post-conversion dynamic feature is generated, and the speech feature of the conversion source speaker is converted to the post-conversion speech feature using the post-conversion dynamic feature. Voice conversion device. The model learning unit inputs the voice features of a plurality of speakers and the pronunciation skills assigned to each speaker, and calculates the voice features of one speaker arbitrarily determined as the conversion source speaker. , learn multiple voice conversion models that each convert to the voice features of multiple other speakers determined as the conversion target speaker,
The voice conversion unit inputs the voice features of the conversion source speaker and the target vocalization skill into a trained voice conversion model, and converts the voice features of the conversion source speaker into a match with the target voice skill. 4. The speech conversion device according to claim 1, wherein the speech conversion device converts into a voice feature amount of a conversion target speaker who has a vocalization skill of . A voice conversion method for converting a dynamic feature of a speaker's voice feature, the method comprising:
With the voice conversion device,
learning a speech conversion model that converts the speech features of the conversion source speaker into the speech features of the conversion target speaker;
inputting the voice features of the conversion source speaker into a trained voice conversion model and converting them into the voice features of the conversion target speaker;
Using the dynamic feature amount of the voice feature amount of the conversion source speaker and the dynamic feature amount of the voice feature amount of the conversion target speaker, the voice feature amount of the conversion source speaker is converted into a post-conversion voice feature amount. and the step of converting it to
generating a speech waveform from the converted speech feature amount;
Voice conversion methods including. A program for causing a computer to function as the speech conversion device according to any one of claims 1 to 4.

Images