JP2023005191A - Voice processing learning program, voice processing learning device, voice processing learning method, voice processing program, voice processor and voice processing method - Google Patents

Abstract

To appropriately convert voice which an arbitrary speaker utters into sound quality of voice which a target speaker utters.SOLUTION: A device comprises: a voice analysis part 20 which is a sound feature amount extractor for converting voice into an input sound feature amount; a speaker encoder 22 for converting a speaker label of voice into a speaker feature amount; a voice encoder 24 configured by including a variation self-encoder having two or more sampling hierarchies which convert the sound feature amount and the speaker feature amount into latent expression; and a voice decoder 26 configured to include a variation self-encoder having two or more sampling hierarchies which generate the sound feature amount by using at least the latent expression and the speaker feature amount. The voice encoder 24, the voice decoder 26 and the speaker encoder 22 are allowed to learn so that a distance between the sound feature amount inputted to the voice encoder 24 and the sound feature amount generated in the voice decoder 26 becomes small.SELECTED DRAWING: Figure 2

Description

The present invention relates to a speech processing learning program, a speech processing learning device, a speech processing learning method, a speech processing program, a speech processing device and a speech processing method.

A speech processing device has been developed that converts speech uttered by an arbitrary speaker into speech having the voice quality of another speaker. For example, a technique is disclosed in which CycleGAN, which is an image conversion technique, is applied to audio conversion (Non-Patent Document 1).

Takuhiro Kaneko and Hirokazu Kameoka, Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks arXiv:1711.11293, Nov. 2017 (EUSIPCO 2018) http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/ projects/cyclegan-vc/

2. Description of the Related Art In a speech processing device that synthesizes and outputs the speech of another speaker from the original speaker, it is required to make the voice quality and phrasing of the synthesized speech as natural as possible. However, in some cases, the conventional learning method of the speech processing apparatus cannot make the synthesized speech sufficiently natural.

According to one aspect of the present invention, a computer comprises an acoustic feature extractor that converts speech into an input acoustic feature, a speaker encoder that converts a speaker label of the speech into a speaker feature, and an input acoustic feature. a speech encoder including a variational autoencoder having two or more sampling hierarchies for converting speaker features into latent representations; and generating acoustic features using at least the latent representations and speaker features. and a speech decoder comprising a variational autoencoder having two or more sampling hierarchies, wherein the speech encoder, the speech decoder, and the speaker encoder are configured to: A speech processing learning program characterized by learning so as to reduce the distance between an input acoustic feature quantity input to a speech encoder and an output acoustic feature quantity generated in the speech decoder.

Here, it is preferable that the speech decoder has a limited number of layers for inputting the speaker feature amount in the two or more sampling layers.

Further, the speech decoder does not input the speaker feature amount to a layer preceding a predetermined layer in the two or more sampling layers, and inputs the speaker feature amount to a layer following the predetermined layer. is preferred.

In addition, it is preferable that the audio decoder performs sampling from the posterior distribution in the hierarchy preceding the predetermined hierarchy in the two or more sampling hierarchies, and performs sampling from the prior distribution in the hierarchy following the predetermined hierarchy. .

Moreover, it is preferable that the speech decoder inputs the speaker feature amount to a conditional instance normalization layer.

According to another aspect of the present invention, a computer comprises an acoustic feature extractor that converts speech into an acoustic feature, a speaker encoder that converts a speaker label of speech into a speaker feature, and the acoustic feature extractor. and the source speaker feature obtained by converting the speaker label of the source speaker in the speaker encoder into a latent representation. a latent representation and a target speaker obtained by transforming the speaker label of the target speaker in the speaker encoder. a speech decoder including a variational autoencoder having two or more sampling hierarchies that generates target acoustic features using at least features; A voice processing program characterized by functioning as a voice processing device comprising a vocoder that converts into

Here, the speech encoder, the speech decoder, and the speaker encoder are learned so as to reduce the distance between the acoustic features input to the speech encoder and the acoustic features generated by the speech decoder. A voice processing program characterized by:

Further, it is preferable that the speech decoder has a limited number of layers for inputting the target speaker feature amount in the two or more sampling layers.

Further, the speech decoder does not input the target speaker feature quantity to a hierarchy preceding a predetermined hierarchy among the two or more sampling hierarchies, and inputs the target speaker feature quantity to a hierarchy succeeding the predetermined hierarchy. is preferably entered.

In addition, it is preferable that the audio decoder performs sampling from the posterior distribution in the hierarchy preceding the predetermined hierarchy in the two or more sampling hierarchies, and performs sampling from the prior distribution in the hierarchy following the predetermined hierarchy. .

Also, it is preferable that the speech decoder inputs the target speaker features to a conditional instance normalization layer.

Another aspect of the present invention is an acoustic feature extractor that converts speech into input acoustic features, a speaker encoder that converts speaker labels of speech into speaker features, and acoustic features and speaker features. a speech encoder comprising a variational autoencoder having two or more sampling hierarchies that transforms into latent representations, and two or more speech encoders that generate acoustic features using at least the latent representations and speaker features a speech decoder comprising a variational autoencoder having a sampling hierarchy, wherein the speech encoder, the speech decoder and the speaker encoder are adapted to input acoustic features input to the speech encoder and the A speech processing learning device characterized by learning so as to reduce the distance from an output acoustic feature quantity generated in a speech decoder.

Another aspect of the present invention is an acoustic feature extractor that converts speech into acoustic features, a speaker encoder that converts speaker labels of speech into speaker features, and a source speech in the acoustic feature extractor. converting a source acoustic feature obtained by converting a speaker's speech and a source speaker feature obtained by converting a speaker label of the source speaker in the speaker encoder into a latent expression. a speech encoder comprising a variational autoencoder having a sampling hierarchy of , a latent representation, and a target speaker feature obtained by transforming the speaker label of the target speaker in the speaker encoder, a speech decoder comprising: a variational autoencoder having two or more sampling hierarchies for generating target acoustic features using at least; and converting the target acoustic features generated by the speech decoder into speech. and a vocoder.

Another aspect of the present invention is an acoustic feature extractor that converts speech into input acoustic features, a speaker encoder that converts speaker labels of speech into speaker features, and acoustic features and speaker features. a speech encoder comprising a variational autoencoder having two or more sampling hierarchies that transforms into latent representations, and two or more speech encoders that generate acoustic features using at least the latent representations and speaker features and a speech decoder comprising a variational autoencoder having a sampling hierarchy, wherein the speech encoder, the speech decoder and the speaker encoder are inputs to the speech encoder. A speech processing learning method characterized by learning so as to reduce the distance between an acoustic feature quantity and an output acoustic feature quantity generated in the speech decoder.

Another aspect of the present invention is an acoustic feature extractor that converts speech into acoustic features, a speaker encoder that converts speaker labels of speech into speaker features, and a source speech in the acoustic feature extractor. converting a source acoustic feature obtained by converting a speaker's speech and a source speaker feature obtained by converting a speaker label of the source speaker in the speaker encoder into a latent expression. a speech encoder comprising a variational autoencoder having a sampling hierarchy of , a latent representation, and a target speaker feature obtained by transforming the speaker label of the target speaker in the speaker encoder, a speech decoder comprising: a variational autoencoder having two or more sampling hierarchies for generating target acoustic features using at least; and converting the target acoustic features generated by the speech decoder into speech. and a vocoder, converting the speech of the source speaker into the speech of the target speaker.

According to the present invention, a speech processing learning program, a speech processing learning device, a speech processing learning method, a speech processing learning program, and speech for appropriately converting speech uttered by an arbitrary speaker into speech uttered by a target speaker. A processing learning device and a speech processing learning method can be provided. Other objects of embodiments of the present invention will become apparent by reference to the specification as a whole.

It is a figure which shows the structure of the speech processing apparatus in embodiment of this invention. 1 is a functional block diagram showing the configuration of a speech processing learning device according to an embodiment of the present invention; FIG. Fig. 2 shows the configuration of a variational auto-encoder; Fig. 3 shows the structure of the Nouveau Variational Auto-Encoder; FIG. 4 is a diagram showing the configuration of a neural network of each layer of a variational auto-encoder in an embodiment of the present invention; FIG. 4 is a diagram for explaining speech learning processing according to the embodiment of the present invention; FIG. 1 is a functional block diagram showing the configuration of a speech learning device according to an embodiment of the present invention; FIG. FIG. 4 is a diagram for explaining audio processing in the embodiment of the present invention; FIG.

A speech processing apparatus 100 according to the embodiment of the present invention includes a processing unit 10, a storage unit 12, an input unit 14, an output unit 16, and a communication unit 18, as shown in FIG. The processing unit 10 includes means for performing arithmetic processing such as a CPU. The processing unit 10 learns speech processing according to the present embodiment by executing the speech processing learning program stored in the storage unit 12 . Further, the processing unit 10 implements the functions related to the sound processing in this embodiment by executing the sound processing program stored in the storage unit 12 . The storage unit 12 includes storage means such as a semiconductor memory and a memory card. The storage unit 12 is connected to the processing unit 10 so as to be accessible, and stores a speech processing learning program, a speech processing program, and information necessary for the processing. The input unit 14 includes means for inputting information. The input unit 14 includes, for example, a keyboard, a touch panel, buttons, etc. for receiving information input from the user. The input unit 14 also includes voice input means for receiving input of voices of an arbitrary speaker and a predetermined target speaker. The audio input means may be configured to include, for example, a microphone, an amplifier circuit, and the like. The output unit 16 includes means for outputting a user interface screen (UI) for accepting input information from an administrator and processing results. The output unit 16 includes, for example, a display that presents images. The output unit 16 also includes a voice output means for outputting the synthesized voice generated by the voice processing device 100 . The audio output means may have a configuration including, for example, a speaker, an amplifier, and the like. The communication unit 18 includes an interface for communicating information with an external terminal (not shown) via the network 102 . Communication by the communication unit 18 may be wired or wireless. It should be noted that the audio information used for audio processing may be obtained from an external terminal via the communication unit 18 .

The speech processing device 100 performs speech processing for converting speech uttered by an arbitrary speaker into the sound quality of the speech of a predetermined speaker (target speaker). The speech processing device 100 also functions as a speech processing learning device that performs learning for the speech processing.

[Voice learning process]
FIG. 2 is a functional block diagram showing the configuration of the speech processing device 100 during speech processing learning. The speech processing device 100 functions as a speech analysis unit 20 , a speaker encoder 22 , a speech encoder 24 , a speech decoder 26 and a learner 28 . Specifically, the speech processing device 100 functions as a speech processing learning device that realizes the following speech learning method by executing a speech processing learning program.

The speech analysis unit 20 acquires speech data and functions as an acoustic feature extractor that extracts acoustic features from the speech data. That is, the processing unit 10 of the speech processing device 100 functions as the speech analysis unit 20 . The voice data may be obtained by converting the voice of the speaker into voice data using the microphone that constitutes the input unit 14 . Also, voice data prerecorded in an external computer or the like may be received via the communication unit 18 . The acquired voice data is stored in the storage unit 12 .

Acquisition processing of speech data is performed for speech uttered by an arbitrary speaker. In the speech learning process, the speech encoder 24 and the speech decoder 26 are trained using speech from many speakers. The speech obtained from each speaker need not have identical content.

In addition, the speech analysis unit 20 further performs speech analysis necessary for speech processing. For example, the speech analysis unit 20 performs cepstrum analysis of the speech based on the frequency characteristics of the input speech, and the mel-frequency cepstrum coefficient including the spectrum envelope (information indicating the thickness of the voice, etc.) and the fine structure information. (MFCC), fundamental frequency and resonance frequency of voice (information indicating pitch of voice, hoarseness of voice, etc.) and other acoustic feature quantities are obtained. The acoustic features can be, for example, an (80×T) dimensional Euclidean space for the length T of the speech segment. Specifically, the speech analysis unit 20 generates and outputs the acoustic feature quantity _xi from the speech uttered by the speaker whose speaker ID (speaker label) is i. The acoustic features extracted by the speech analysis unit 20 are input to the speech encoder 24 and the learning device 28 .

The speaker encoder 22 converts the ID of the speaker of the speech input to the speech analysis unit 20 into a speaker feature quantity that can be used for speech processing, and outputs the speaker feature quantity. The speaker encoder 22 can include an embedding module that converts the speaker's ID into a speaker feature quantity and outputs it. For example, if the speaker ID is i, the speaker encoder 22 generates and outputs the speaker feature quantity _yi . The speaker feature amount generated by the speaker encoder 22 is input to the speech encoder 24 and speech decoder 26 .

In the training of the speech processing apparatus 100, a set of combinations ( _xi , _yi ) of the acoustic feature quantity _xi and the speaker feature quantity _yi obtained from speech uttered by a plurality of speakers is used.

The speech encoder 24 receives input of the acoustic feature amount and the speaker feature amount, and performs processing for converting the acoustic feature amount and the speaker feature amount into latent expressions. The speech decoder 26 receives the input of the latent expression and the speaker's feature quantity obtained by the speech encoder 24, and converts the latent expression and the speaker's feature quantity into an acoustic feature quantity. A latent expression represents the linguistic features of the input speech data.

As shown in FIG. 2, the speech encoder 24 and the speech decoder 26 receive the acoustic feature quantity _xi from the speech analysis unit 20, convert it into a latent expression z in the speech encoder 24, and convert it into a latent expression z in the speech decoder 26. Acoustic feature quantity x _i ̂ is reconstructed from z, and the output acoustic feature quantity x _i ̂ is learned to restore the input acoustic feature quantity x _i .

In this embodiment, audio encoder 24 and audio decoder 26 are implemented by Variational Auto-Encoders (VAEs). A variational auto-encoder is a type of variational auto-encoder that generates a latent representation by sampling based on a probability distribution, as shown in FIG. The probability distribution is assumed to be a normal distribution defined by mean μ and variance σ. A variational auto-encoder consists of a combination of an encoder that produces a latent representation z by sampling an input X based on the mean μ and the variance σ, and a decoder that produces the output X̂ from the latent representation z. In the variational auto-encoder, the speaker encoder 22, the speech encoder 24, and the speech decoder 26 are trained so that the restoration error (reconstruction distance) E between the input X and the output X̂ becomes small.

As shown in FIG. 4, a general variational auto-encoder is composed of a single-layer neural network, but in this embodiment, a nouveau variational auto-encoder is composed of two or more hierarchical neural networks. • It is preferably an auto-encoder (NVAE: Nouveau Variational Auto-Encoder). That is, the Nouveau Variational Auto-encoder consists of a variational autoencoder with two or more sampling hierarchies. For example, in the speech processing device 100, it is preferable to configure the speech encoder 24 and the speech decoder 26 with n=35 hierarchical neural networks, respectively.

Each layer of the Nouveau Variational Auto-Encoder of the audio encoder 24 and the audio decoder 26 includes a Conditional-Instance-Normalization layer (CIN layer), a Convolution layer (CONV layer), a Squeeze-and-Excitation layer, as shown in FIG. It is configured by combining layers (SE layers). The CIN layer is a layer that replaces the batch normal virtual (BN layer) in the common Nouveau Variational Auto-Encoder. The CIN layer is one of the normalization layers, and is a conditional instance normalization layer that performs normalization by setting different parameters for each style. In this embodiment, the CIN layer takes a speaker feature as one input and performs normalization conditioned by the input speaker feature. Also, the Swish activation function is an activation function expressed as f(x)=x/(1+e ^−βx ). A convolution layer is a layer that applies a convolution operation to an input and outputs the operation result to the next layer. The SE layer is a layer that adaptively applies attention to the input based on the relationship between channels and outputs weighted features.

The speech learning process in speech processing device 100 will be described with reference to FIG. The speech encoder 24 and the speech decoder 26 each show an example configured by a neural network with the number of layers n. The hierarchy number n can be set to 35 hierarchies, for example. Each layer is configured by combining the Conditional-Instance-Normalization layer (CIN layer), the Convolution layer (CONV layer), and the Squeeze-and-Excitation layer (SE layer) shown in FIG. Note that the latent expression output from the layer k of the audio encoder 24 (where k represents the number of layers from 1 to n) is denoted by hk, and the latent expression output from the layer represented by the number of layers _k of the audio decoder 26 is We denote the latent representation by _zk .

In the speech encoder 24, the acoustic feature amount _xi and the speaker feature amount _yi are input for the layer _n , and the latent expression hn is output. At the next layer n-1, the latent expression h _n and the speaker feature quantity y _i output from the preceding layer n are input, and the latent expression h _n-1 is output. Similarly, at layer k, the latent expression _hk+1 and the speaker feature quantity _yi output from the preceding layer _k +1 are input, and the latent expression hk is output. At the final level ₁ , the latent expression h2 and the speaker feature quantity _yi output from the preceding level ₂ are input, and the latent expression h1 is output. A latent expression z1 of layer ₁ , which is the _first stage of the speech decoder 26, is sampled from the latent expression h1. In this way, in the speech encoder 24, it is preferable to include the speaker feature quantity _yi in the input in all layers 1 to n.

In the audio decoder 26, the latent expression z1 is input to the hierarchy ₁ , which is the first stage, and the latent expression z2 is _output . In addition, the latent expression z _k in the layer k of the speech decoder 26 is obtained from the latent expression z _k −1 in the preceding layer k−1 in the speech decoder 26, the latent expression h _k in the k-th layer in the speech encoder 24, and the speaker feature amount. It can be obtained by sampling from the prior distribution p(z _k |z _k−1 , h _k , y _i ) based on y _i . Also, the latent expression z _k is the layer k− ₁ , layer k _{− 2} , . And it is also possible to obtain by sampling from the posterior distribution p (z _k | z _k-1 , z _k-2 ... z ₁ , h _k ) based on the k-th layer latent expression h _k of the speech encoder 24 . Note that the distribution p(a|b) is a likelihood function that indicates the likelihood that a will be output with b as a precondition.

In the speech learning process, sampling is performed from the speech encoder 24 over layers close to and far from the output of the speech decoder 26 . That is, as shown in FIG. 6, it is preferable to sample from the k-th layer latent expression h _k of the speech encoder 24 in all layers 1 to n. Moreover, it is preferable not to include the speaker feature quantity y _i in the input for sampling from the posterior distribution.

That is, it is preferable that the speech decoder 26 includes the speaker feature _yi as input only in the hierarchy near the output, and does not include the speaker feature _yi as the input in the hierarchy far from the output. At this time, the speaker feature quantity y _i is included in the input in the layer in which sampling is not performed from the speech encoder 24 but from the prior distribution, and the speaker feature is It is preferred not to include the quantity y _i in the input.

Note that, in layers in which the speaker feature y _{i is not included in the sampling, the speaker feature y i is} not input to the Conditional-Instance-Normalization layer (CIN layer).

With such a configuration, in the learning device 28, the error (distance) between the acoustic feature quantity x _i input to the speech decoder 26 and the reconstructed acoustic feature quantity x _i ^ output from the speech decoder 26 becomes small. Various parameters (weight coefficients or biases of each neuron, etc.) of the neural networks in each layer included in the speaker encoder 22, speech encoder 24, and speech decoder 26 are adjusted as follows.

Here, the speech decoder 26 is configured so that the error (distance) between the acoustic feature quantity x _i input to the speech decoder 26 and the reconstructed acoustic feature quantity x _i ^ output from the speech decoder 26 is small. It suffices to appropriately set a layer that serves as a boundary between a layer in which sampling is performed from the prior distribution considering the speaker feature y _i and a layer in which sampling is performed from the posterior distribution that does not consider the speaker feature y _i .

As described above, the speech encoder is arranged so that the speech represented by the acoustic feature quantity x _i input to the speech encoder 24 and the speech represented by the acoustic feature quantity x _i ̂ reconstructed in the speech decoder 26 are closer to each other. 24 and speech decoder 26 are trained.

[Voice processing]
FIG. 7 is a functional block diagram showing the configuration of the speech processing apparatus 100 during speech processing for converting the speech uttered by the source speaker into the speech uttered by the target speaker. The speech processing device 100 functions as a speech analysis unit 20 , a speaker encoder 22 , a speech encoder 24 , a speech decoder 26 and a vocoder 30 . Specifically, the sound processing device 100 functions as a sound processing device that implements the following sound processing by executing a sound processing program.

The speech analysis unit 20 acquires speech data of speech uttered by a source speaker and performs speech analysis necessary for speech processing. The acoustic features extracted by the speech analysis unit 20 are input to the speech encoder 24 .

The speaker encoder 22 converts the IDs of the source speaker and the target speaker into speaker feature quantities that can be used for speech processing and outputs them. If the speaker has a source speaker ID of _s , the speaker encoder 22 generates a source speaker feature quantity ys and outputs it to the speech encoder 24 . Also, when the target speaker ID is the speaker whose ID is _t , the speaker encoder 22 generates the target speaker feature amount yt and outputs it to the speech decoder 26 .

The speech encoder 24 receives the input of the acoustic features obtained from the speech of the source speaker and the source speaker features, and converts the acoustic features and the source speaker features into latent expressions. The speech decoder 26 receives the input of the latent expression and the target speaker feature quantity obtained by the speech encoder 24, and performs processing for reconstructing the acoustic feature quantity from the latent expression and the target speaker feature quantity.

Audio processing in the audio processing apparatus 100 will be described with reference to FIG. The speech processing is performed using the speech encoder 24 and the speech decoder 26 trained in the speech learning process described above.

In the speech encoder 24, the acoustic feature quantity _xs and the source speaker feature quantity _ys obtained from the speech of the source speaker are input to the layer _n , and the latent expression hn is output. Thereafter, in the same manner as during learning, at layer k, the latent expression h _k+1 and the source speaker feature quantity y _s output from the preceding layer k+1 are input, and the latent expression h _k is output. In the last stage, hierarchy ₁ , the latent expression h2 and the source speaker feature quantity _ys output from the preceding stage, hierarchy ₂ , are input, and the latent expression h1 is output. A latent expression z1 of layer ₁ , which is the _first stage of the speech decoder 26, is sampled from the latent expression h1.

In the audio decoder 26, the latent expression z1 is input to the hierarchy ₁ , which is the first stage, and the latent expression z2 is _output . In the layers far from the output of the speech decoder 26, the target speaker feature quantity yt is not included in the input, and in the speech decoder 26, the latent expressions _zk- in the layers _k -1, k-2, . . . ₁ , latent expression z _k−2 . . . posterior distribution p(z _k |z _{k −1 ,} z _k−2 . ₁ , h _k ). In the layer close to the output of the speech decoder 26, sampling from the speech encoder 24 is not performed, and the _prior distribution p _{(z k |} z _{k −1} , y _t ). FIG. 8 shows an example of sampling from the prior distribution at layer n−1 and layer n of the audio decoder 26 . At this time, it is preferable to include the target speaker feature _yt , not the source speaker feature _ys , in the input for sampling from the prior distribution.

Acoustic feature x _s obtained from the source speaker's speech from layer n, which is the final stage of the speech decoder 26, by speech processing in the speech encoder 24 and the speech decoder 26 is combined with the target speaker's speech to obtain the acoustic feature x _t is constructed and output.

The vocoder 30 converts the acoustic feature quantity _xt output from the speech decoder 26 into speech data and outputs the speech data. The vocoder 30 can convert the acoustic feature quantity _xt into speech data by performing the reverse process of extracting the acoustic feature quantity from the speech data in the speech analysis unit 20 .

As described above, according to the speech processing device 100 of the present embodiment, a speech processing device and a speech processing program for appropriately converting a speech uttered by an arbitrary speaker into the sound quality of a speech uttered by a target speaker. and a speech processing method, a speech learning processing device, a speech learning processing program, and a speech learning processing method. That is, the speech processing apparatus 100 including the trained speech encoder 24 and the speech decoder 26 can implement speech processing that converts the speech uttered by the source speaker into speech that sounds like the speech uttered by the target speaker.

In particular, by applying a Nouveau Variational Auto-Encoder (NVAE) to the speech encoder 24 and the speech decoder 26, the target speaker uttered the source speaker's speech more than before. It can be converted into a natural-sounding voice.

10 processing unit, 12 storage unit, 14 input unit, 16 output unit, 18 communication unit, 20 speech analysis unit, 22 speaker encoder, 24 speech encoder, 26 speech decoder, 28 learning device, 30 vocoder, 100 speech processing device, 102 network.

Claims (15)

the computer,
an acoustic feature quantity extractor that converts speech into an input acoustic feature quantity;
a speaker encoder that converts speaker labels of speech into speaker features;
a speech encoder comprising a variational autoencoder having two or more sampling hierarchies for converting input acoustic features and speaker features into latent representations;
a speech decoder comprising a variational autoencoder having two or more sampling hierarchies that generates acoustic features using at least latent expressions and speaker features;
function as a speech processing learning device equipped with
The speech encoder, the speech decoder, and the speaker encoder are characterized by learning so as to reduce the distance between an input acoustic feature input to the speech encoder and an output acoustic feature generated in the speech decoder. A speech processing learning program. The speech processing learning program according to claim 1,
A speech processing learning program, wherein said speech decoder is characterized in that the hierarchy for inputting the speaker feature amount is limited in said two or more sampling hierarchies. The speech processing learning program according to claim 2,
The speech decoder is characterized in that, among the two or more sampling hierarchies, the speaker feature amount is not input to a hierarchy prior to a predetermined hierarchy, and the speaker feature amount is input to a hierarchy subsequent to the predetermined hierarchy. A speech processing learning program. A speech processing learning program according to claim 3,
The audio decoder performs sampling from the posterior distribution in the hierarchy preceding the predetermined hierarchy in the two or more sampling hierarchies, and performs sampling from the prior distribution in the hierarchy following the predetermined hierarchy. learning program. The speech processing learning program according to any one of claims 1 to 4,
A speech processing learning program, wherein the speech decoder inputs the speaker feature amount to a conditional instance normalization layer. the computer,
an acoustic feature extractor that converts speech into acoustic features;
a speaker encoder that converts speaker labels of speech into speaker features;
A source acoustic feature obtained by converting the speech of the source speaker in the acoustic feature extractor, and a source speaker feature obtained by converting the speaker label of the source speaker in the speaker encoder. a speech encoder comprising a variational autoencoder having two or more sampling hierarchies that transforms into a latent representation;
A variational self-code having two or more sampling hierarchies that generates a target acoustic feature using at least a latent representation and a target speaker feature obtained by transforming a speaker label of the target speaker in the speaker encoder. an audio decoder comprising a decoder;
a vocoder that converts the target acoustic features generated by the speech decoder into speech;
A speech processing program characterized by functioning as a speech processing device comprising: The speech processing program according to claim 6,
The speech encoder, the speech decoder, and the speaker encoder are trained so as to reduce the distance between the acoustic features input to the speech encoder and the acoustic features generated by the speech decoder. A speech processing program characterized by: The speech processing program according to claim 6 or 7,
A speech processing learning program, wherein said speech decoder is characterized in that the hierarchy into which said target speaker feature quantity is input is limited in said two or more sampling hierarchies. The speech processing program according to claim 8,
The speech decoder does not input the target speaker feature quantity to a hierarchy preceding a predetermined hierarchy among the two or more sampling hierarchies, and inputs the target speaker feature quantity to a hierarchy succeeding the predetermined hierarchy. A speech processing program characterized by: The speech processing program according to claim 9,
The audio decoder performs sampling from the posterior distribution in the hierarchy preceding the predetermined hierarchy in the two or more sampling hierarchies, and performs sampling from the prior distribution in the hierarchy following the predetermined hierarchy. program. The speech processing program according to any one of claims 6 to 10,
A speech processing program, wherein the speech decoder inputs the target speaker features to a conditional instance normalization layer. an acoustic feature quantity extractor that converts speech into an input acoustic feature quantity;
a speaker encoder that converts speaker labels of speech into speaker features;
a speech encoder comprising a variational autoencoder having two or more sampling hierarchies for converting acoustic features and speaker features into latent representations;
a speech decoder comprising a variational autoencoder having two or more sampling hierarchies that generates acoustic features using at least latent expressions and speaker features;
with
The speech encoder, the speech decoder, and the speaker encoder are characterized by learning so as to reduce the distance between an input acoustic feature input to the speech encoder and an output acoustic feature generated in the speech decoder. A speech processing learning device. an acoustic feature extractor that converts speech into acoustic features;
a speaker encoder that converts speaker labels of speech into speaker features;
A source acoustic feature obtained by converting the speech of the source speaker in the acoustic feature extractor, and a source speaker feature obtained by converting the speaker label of the source speaker in the speaker encoder. a speech encoder comprising a variational autoencoder having two or more sampling hierarchies that transforms into a latent representation;
A variational self-code having two or more sampling hierarchies that generates a target acoustic feature using at least a latent representation and a target speaker feature obtained by transforming a speaker label of the target speaker in the speaker encoder. an audio decoder comprising a decoder;
a vocoder that converts the target acoustic features generated by the speech decoder into speech;
A speech processing device comprising: an acoustic feature quantity extractor that converts speech into an input acoustic feature quantity;
a speaker encoder that converts speaker labels of speech into speaker features;
a speech encoder comprising a variational autoencoder having two or more sampling hierarchies for converting acoustic features and speaker features into latent representations;
a speech decoder comprising a variational autoencoder having two or more sampling hierarchies that generates acoustic features using at least latent expressions and speaker features;
In a speech processing learning device comprising
The speech encoder, the speech decoder, and the speaker encoder are characterized by learning so as to reduce the distance between an input acoustic feature input to the speech encoder and an output acoustic feature generated in the speech decoder. A speech processing learning method that an acoustic feature extractor that converts speech into acoustic features;
a speaker encoder that converts speaker labels of speech into speaker features;
A source acoustic feature obtained by converting the speech of the source speaker in the acoustic feature extractor, and a source speaker feature obtained by converting the speaker label of the source speaker in the speaker encoder. a speech encoder comprising a variational autoencoder having two or more sampling hierarchies that transforms into a latent representation;
A variational self-code having two or more sampling hierarchies that generates a target acoustic feature using at least a latent representation and a target speaker feature obtained by transforming a speaker label of the target speaker in the speaker encoder. an audio decoder comprising a decoder;
a vocoder that converts the target acoustic features generated by the speech decoder into speech;
using an audio processor comprising
A speech processing method, characterized in that the speech of the source speaker is transformed into the speech of the target speaker.

Images