JP6764851B2 - Series data converter, learning device, and program - Google Patents

Description

The present invention relates to a sequence data conversion device, a learning device, and a program.

[Chapter 1: Introduction]
The technique of converting only non-linguistic / para-language (speaker nature, utterance style, etc.) while retaining the linguistic information (speech sentence) of the input voice is called voice quality conversion. It can be applied to voice enhancement and pronunciation conversion. The voice quality conversion problem can be formulated as a regression analysis problem that estimates the mapping function from the conversion source voice features to the conversion target voice features. Among the conventional methods of voice quality conversion, the method using the Gaussian Mixture Model (GMM) is widely used because of its effectiveness and versatility. In recent years, NN-based methods such as constrained Boltzmann machines, feed-forward neural networks (Neural Networks; NN), recurrent NNs (RNNs), and convolutional NNs (CNNs) and non-negative values Examination of case-based methods using non-negative Matrix Factorization (NMF) is also underway.

Martin Arjovsky, Soumith Chintala, and L_eon Bottou. WassersteinGAN. In Proc. ICML, 2017. James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural networks. In Proc. ICLR, 2017. Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proc. ICML, pages 933-941, 2017. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. NPIS, pages 2672-2680, 2014. Takuhiro Kaneko, Hirokazu Kameoka, Kaoru Hiramatsu, and Kunio Kashino. Sequence-to-sequence voice conversion with similaritymetric learned using generative adversarial networks. In Proc. INTERSPEECH, pages 1283-1287, 2017. Kun Liu, Jianping Zhang, and Yonghong Yan. High quality voice conversion through phoneme-based linear mapping functions with STRAIGHT for Mandarin. In Proc. FSKD, pages 410-414, 2007. Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proc. ICCV, 2017. Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. WORLD: Avocoder-based high-quality speech synthesis system for real-time appsliations. IEICE Trans. Inf. Syst., 99 (7): 1877-1884, 2016. Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross domainimage generation. In Proc. ICLR, 2017. Shinnosuke Takamichi, Tomoki Toda, Graham Neubig, Sakriani Sakti, and Satoshi Nakamura. A postfiter to modify the modulation spectrum in HMM-based speech synthesis. In Proc. ICASSP, pages290-294, 2014. Tomoki Toda, Alan W Black, and Keiichi Tokuda. Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory. IEEE / ACM Trans. Audio Speech Lang. Process., 15 (8): 2222-2235, 2007. Tomoki Toda, Ling-Hui Chen, Daisuke Saito, Fernando Villavicencio, Mirjam Wester, Zhizheng Wu, and Junichi Yamagishi. The Voice Conversion Challenge 2016. In Proc. INTERSPEECH, pages 1632-1636, 2016. Mirjam Wester, Zhizheng Wu, and Junichi Yamagishi. Analysis of the Voice Conversion Challenge 2016 evaluation results. In Proc. INTERSPEECH, pages 1637-1641, 2016. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.Un-paired image-to-image translation using cycle-consistent adversarial networks. In Proc. ICCV, pages 2223-2232, 2017.

In many of these methods, the conversion function is learned using parallel data so that the features of the converted speech are as close as possible to the features of the target speech. However, there are many situations where it is difficult to prepare pair data of the conversion source voice and the target voice of the same utterance content depending on the application. Even if such pair data can be prepared, highly accurate time matching is required, and when this is performed by automatic processing, visual or manual pre-screening is required to correct the matching error.

An object of the present invention is to provide a parallel data-free voice quality conversion method that does not require parallel data.

The series data conversion device according to the present invention uses an input unit for receiving series data and a converter for series data of two domains from the forward conversion input data which is the data of one domain to the other. The forward conversion unit that converts to the forward conversion output data, which is the data of the domain, and the reverse conversion output, which is the data of the input domain of the forward conversion unit, performs the reverse conversion of the forward conversion output data using a converter. A state determination unit that determines whether or not the reverse conversion unit that converts data and the forward conversion output data are appropriate as series data of the domain that is the target of the forward conversion output data by using a state determination device. To the results of the forward / reverse conversion distance measuring unit, the state determination unit, and the forward / reverse conversion distance measuring unit, which measures the distance between the reverse conversion output data and the forward conversion input data using a distance measuring device. A learning unit that updates the parameters of the converter and the state determination unit, a conversion unit that converts data received by the input unit using the converter learned by the learning unit, and the conversion unit. It is configured to include an output unit that outputs the converted data.

The learning device according to the present invention uses an input unit for receiving series data and a converter for the series data of two domains, and uses a converter to convert the forward conversion input data which is the data of one domain to the other domain. The forward conversion unit that converts the data to the forward conversion output data and the reverse conversion output data that is the data of the input domain of the forward conversion unit are converted in reverse using a converter. An inverse conversion unit to be converted, a state determination unit that determines whether or not the forward conversion output data is appropriate as series data of the target domain of the forward conversion output data by using a state determination device. The reverse conversion output data and the forward / reverse conversion distance measuring unit for measuring the distance of the forward conversion input data using a distance measuring device are provided, and the results of the state determination unit and the forward / reverse conversion distance measuring unit. The parameters of the converter and the state determination unit are updated accordingly.

Further, the program according to the present invention is a program for operating a computer as each part of the series data conversion device according to the above invention.

According to the series data conversion device, the learning device, and the program of the present invention, it is possible to provide a parallel data-free voice quality conversion method that does not require parallel data.

It is a figure which shows the learning process of CycleGAN. It is a figure which shows the whole structure of a data conversion apparatus. It is a figure which shows the whole structure of the data conversion apparatus which concerns on Outline 1. It is a figure which shows the whole structure of the data conversion apparatus which concerns on Outline 2. FIG. It is a figure which shows the processing routine at the time of learning by a data conversion apparatus. It is a figure which shows the processing routine at the time of conversion by a data conversion apparatus. It is a figure which shows the GV comparison for each order of mer cepstrum. It is a figure which shows the comparison of MS for each modulation frequency. It is a figure which shows the comparison (S: source, T: target, P: proposed method, B: comparison method) of the similarity between a source voice and a target voice.

[Overview]
In this paper, we propose a parallel data-free series data conversion method. The proposed method enables series data conversion without using parallel data of conversion source series data and conversion target series data, and is often a problem in many conventional series data conversion methods (for example, voice quality conversion method). The feature is that over-smoothing of data (for example, acoustic parameters) is unlikely to occur. The features of the above proposed method are realized by using the Cyclic-consistent adversarial network (CycleGAN). CycleGAN was originally proposed as a method of image style conversion, and it has a forward conversion function from the conversion source data to the conversion target data and a reverse conversion function from the conversion target data to the conversion target data. It is a methodology that enables desired conversion without using paired data of conversion source and conversion target by learning at the same time. The proposed method applies CycleGAN to the series data transformation problem, and uses the sum of the adversarial loss, the Cyclic-consistency loss, and the identity-mapping loss as the learning criteria. By doing so, it is possible to learn the conversion function of the feature series from the conversion source series data to the target series data. The circular consistency criterion is a criterion that indicates how much the inverse conversion of the forward conversion of the conversion source data matches the original conversion data, and how much the forward conversion of the reverse conversion of the conversion target data is restored. It is a standard that indicates whether or not it matches the target data. The hostile learning norm is a criterion that expresses how easy it is to distinguish the converted data from the actual data of the conversion target by the discriminator. The smaller this is, the more the probability distribution of the conversion data becomes the probability distribution of the actual data of the conversion target. Means more similar. The conformal mapping error is a criterion that indicates how well the converted data and the conversion source data match. In addition, in the proposed method, the conversion function from the feature series to the feature series is considered as the conversion function in the forward and reverse directions, and both are described by the Gated Convolutional Neural Network, so that the time dependence relationship is added to the feature conversion law. I am trying to reflect it. In the experiment, the proposed method was applied to the task of voice quality conversion and evaluated. Quantitative evaluation experiments confirmed that the converted voice by the proposed method has Global Variance (GV) and Modulation Spectra (MS) that are close to the actual voice of the conversion target. In addition, it was confirmed by a subjective evaluation experiment that a degree of naturalness equal to or higher than that of the voice quality conversion method using parallel data and a degree of similarity to the target speaker could be obtained.

The proposed method (1) does not require a separate module for data such as text labels and reference speech and speech recognition, and (2) oversmoothing of acoustic parameters, which is often a problem in many conventional voice quality conversion methods. It is characterized by the fact that (3) conversion that captures the time-frequency structure of the voice of the conversion source and conversion target is possible. The features of the above proposed method are realized by using the Cyclic-consistent adversarial network (CycleGAN) (also known as Disco-GAN and DualGAN). CycleGAN was originally proposed as a method of image style conversion, and it has a forward conversion function from the conversion source data to the conversion target data and a reverse conversion function from the conversion target data to the conversion target data. It is a methodology that enables desired conversion without using paired data of conversion source and conversion target by learning at the same time. The proposed method applies CycleGAN to the voice transformation problem, and uses the sum of the adversarial loss, the Cyclic-consistency loss, and the identity-mapping loss as the learning criteria. This makes it possible to learn the conversion function of the voice feature quantity from the conversion source voice to the target voice. The circular consistency criterion is a criterion that indicates how much the inverse conversion of the forward conversion of the conversion source data matches the original conversion data, and how much the forward conversion of the reverse conversion of the conversion target data is restored. It is a standard that indicates whether or not it matches the target data. The hostile learning norm is a criterion that expresses how easy it is to distinguish the converted data from the actual data of the conversion target by the discriminator. The smaller this is, the more the probability distribution of the conversion data becomes the probability distribution of the actual data of the conversion target. Means more similar. The conformal mapping error is a criterion that indicates how well the converted data and the conversion source data match. In addition, in the proposed method, the conversion function from the feature series to the feature series is considered as the conversion function in the forward and reverse directions, and both are described by the Gated Convolutional Neural Network, so that the time dependence relationship is added to the feature conversion law. I am trying to reflect it. In the above, we have focused on voice quality conversion, which is a typical example of series data conversion, but there is a similar awareness of issues in more general series data conversion (for example, song tone conversion, text conversion, etc.). , Features of the proposed method (1) No separate data or module is required, (2) Over-smoothing of feature series is unlikely to occur, (3) Series of conversion source and conversion target series data It is possible to take advantage of the fact that conversion that captures the hierarchical structure is possible.

[Chapter 2: Related Research]
We describe related research in voice quality conversion, which is a typical example of the task of converting series data to series data. As described above, many of the conventional methods of voice quality conversion are based on the assumption that parallel data is used, but recently, some methods that do not necessarily require parallel data have been proposed. One example is a method using voice recognition. In this method, parallel data is constructed by pairing the voice features of the time frame recognized as the same phoneme in the conversion source voice and the conversion target voice. This method is expected to perform speech recognition with extremely high accuracy, but it may require a large amount of speech corpus to learn speech recognition itself, which is a problem depending on the usage situation. Eh. Another example of the method is to use speaker adaptation technology. This method does not need to prepare parallel data of the conversion source voice and the conversion target voice, but requires parallel data of the reference voice for learning the speaker space. Further, in recent years, studies have been conducted on a method that does not require any data such as text labels and reference voices, modules such as voice recognition, and parallel data. These methods assume that both the source speech and the target speech belong to a low-dimensional embedded space, making it difficult to model the details and detailed components of the speech spectrogram. On the other hand, the proposed method is a method of directly learning the mapping from the conversion source series data to the conversion target series data. This feature of the proposed method is particularly advantageous for tasks such as voice conversion, where the details of the transformed data and the realism of the detailed structure are important.

[Chapter 3: Mode for carrying out the invention]
Hereinafter, embodiments of the present invention will be described. The principle of the series data conversion device of the present invention will be described.

3. 3. Parallel data-free series data conversion using CycleGAN

The purpose of this study is to learn the conversion function from the domain X series data x ∈ X to the domain Y series data y ∈ Y without the need for parallel data. In this study, this problem is solved based on CycleGAN (Non-Patent Document 14) . This chapter first outlines CycleGAN in Section 4.1. The original paper of CycleGAN dealt with image data, but the subject of this research is series data such as audio data. Section 4.2 describes an important device for handling series data, that is, the parallel data-free series data conversion method we propose.

3.1 CycleGAN

In CycleGAN, the conversion function G _X → _Y is learned using two loss functions, Adversarial loss and Cycle-consistency loss. The learning process is shown in FIG. 1 , (a) is a criterion showing how much the reverse conversion of the forward conversion of the conversion source data matches the conversion source data as before, and (b) is the reverse conversion of the target data. Shows the criteria that show how much the forward conversion of is consistent with the conversion target data. Adversarial loss: Adversarial Loss is a loss function that measures the validity of the conversion data G _{X → Y} (x) as the data y of the domain to be converted, and the distribution of the conversion data.

And when the data distribution P _Data (y) of the domain to be converted approaches, the value of this loss function becomes smaller. When the Generative adversarial network (GAN) (Non-Patent Document 4) is used as the formulation of Adversarial loss, the objective function is as follows.

Here generator G _{X → Y} by minimizing the objective function, the discriminator D _Y to be able to generate data which can not distinguish the data y to be converted domain. On the other hand, the identifier D _Y, by maximizing the objective function, so as not be fooled by G _{X → Y.} Here, an example of using GAN for formulating Adversarial loss is shown, but this is an extension model of any GAN, for example, Least squares GAN (LSGAN) (Non-Patent Document 7) and Wasserstein GAN (WGAN) ( It is also possible to use Non-Patent Document 1) and the like. For example, when LSGAN is used, the Cross Entropy in Eq. (1) is Least square loss. In GAN, the true data distribution and the generated data distribution are brought closer to each other based on the Jensen-Shannon divergence standard, but WGAN tries to bring them closer to each other based on the Earth Mover's Distance standard.

Cycle-consistency loss: Adversarial loss alone only constrains G _{X → Y} (x) to follow the data distribution of the domain to be translated, so there is contextual information between x and G _{X → Y} (x). Not always retained. Therefore, CycleGAN addresses this problem by adding two more constraints. The first is the Adversarial loss for the inverse transformation G _{Y → X} , that is,

Is. The other is the Cycle-consistency loss, which is given below.

In the above equation, the case where L1 is used as a method for measuring the distance between two data is shown, but it is possible to use any distance scale, for example, L2 distance, Kullback-Leibler divergence, or An arbitrary feature amount extractor may be prepared, and the distance may be measured with respect to the feature amount extracted by the feature amount extractor. The feature extractor can also be configured using, for example, a neural network. For example, the above-mentioned classifier can be used as a feature extractor, and the distance may be measured in the feature amount space in the classifier.

These added terms encourage G _{X → Y} and G _{Y → X} to pseudo-find (x, y) paired data with similar contextual information from a variety of destination candidates. ..

The entire objective function is represented below using the trade-off parameter λ _cyc .

3.2 CycleGAN for parallel data-free series data conversion

In order to apply CycleGAN to parallel data-free series data conversion, this study proposes two modifications. The first is modeling of series data using Gated CNN (Non-Patent Document 3) , and the second is retention of linguistic information using Identity-mapping loss (Non-Patent Document 9) . Although the present invention mainly describes voice conversion as an example of series data, it should be noted that the proposed method is generally effective for series data and is not limited to voice data.

Gated CNN: There are two characteristics of series data: it has a series structure and it has a hierarchical structure. For example, in the case of voice data, there are serial and hierarchical structures such as voiced / unvoiced sections and phonemes / morphemes. When trying to capture such a structure using a neural network, the method of constructing the network is one of the keys. Therefore, in this research, we propose to introduce a model that can express series relations and hierarchical relations in CycleGAN. Specifically, Gated CNN is used. In addition, RNN (LSTM, etc.) can also be used, but since RNN has a recursive structure, parallelization is difficult, and calculation cost is high, Gated CNN is used here. The important thing here is to use a model that can capture the series structure and hierarchical structure, and even if you use Quasi-RNN (Non-Patent Document 2), which is a hybrid of CNN and RNN proposed in recent years. Good.

In the original paper (Non-Patent Document 3) , Gated CNN shows the latest performance in language modeling, and in recent years, it has also shown its effectiveness in speech modeling (Non-Patent Document 5) . In Gated CNN, Gated linear units (GLUs) are used as the activation function, and the output of the (l + 1) layer.

Is the output of layer l

And model parameters

Can be calculated by the following formula using.

here,

Is the element product and σ is the sigmoid function. With this gate mechanism, when information is propagated between networks, it is possible to selectively propagate information according to the information in the previous layer.

Identity-mapping loss: Retaining semantic information is also an important requirement when trying to convert series data. For example, in the case of voice conversion, it is the speaker who wants to convert, and it is required that the utterance content (linguistic information) is retained. As mentioned above, in CycleGAN, Cycle-consistency loss contributes to the retention of context information, but this constraint is limited to the loose constraint of forward conversion and reverse conversion, and then returns, which is sufficient for retaining linguistic information. Doesn't work. In order to solve this problem without the need for an external module such as a voice recognizer, this study proposes the use of Identity-mapping loss (Non-Patent Document 9) . Identity-mapping loss is expressed by the following formula.

This loss function constrains the structure of the data to be preserved between the input and the output. In practice, we introduced the trade-off parameter λ _id and weighted the loss function.

Is used together with equation (3).

In the above equation, the case where L1 is used as a method for measuring the distance between two data is shown, but it is possible to use any distance scale, for example, L2 distance, Kullback-Leibler divergence, or An arbitrary feature amount extractor may be prepared, and the distance may be measured with respect to the feature amount extracted by the feature amount extractor. The feature extractor can also be configured using, for example, a neural network. For example, the above-mentioned classifier can be used as a feature extractor, and the distance may be measured in the feature amount space in the classifier. It should be noted that this Identity-mapping loss is a constraint that guides the direction of learning, and may be used only in the initial stage of learning instead of being used for the entire learning period.

4. Overall configuration and each flow

4.1
The overall configuration diagram is shown in FIG . 2 , and each part will be described as follows.

The data conversion device functionally includes an input unit 100, a control unit 200, and an output unit 300.

The input unit 100 receives the data included in the data group X and the data included in the data group Y.

Specifically, it accepts the data x ∈ X contained in the data group X and the data y ∈ Y contained in the data group Y.

Control unit 200 includes a rectifier unit 210, a state determination unit 220, an inverse transform unit 230, a forward and reverse converting distance measurement unit 240, a self-conversion unit 250, a self converting the distance measurement unit 260, the neural network memory section It is composed of 270, a learning unit 280, and a conversion unit 290.

The forward conversion unit 210 converts the input data of the data group X into the data of the conversion data group XY by the converter G _{X → Y.} Further, the forward conversion unit 210 converts the input data of the data group Y into the data of the conversion data group YX by the converter G _{Y → X.}

Specifically, the forward conversion unit 210 converts the data sample x of the data group X into the data G _{X → Y} (x) of the conversion data group XY by the converter G _{X → Y} stored in the neural network storage unit 270. Convert. Further, the forward conversion unit 210 converts the data sample y of the data group Y into the data G _{Y → X} (y) of the conversion data group _{Y X} by the converter G _{Y → X} stored in the neural network storage unit 270.

State determining unit 220, and converts data group XY data obtained by the forward transform unit 210, for each of the input data y, performs state determination using the state determiner D _Y. Further, the state determination unit 220 determines the state of each of the conversion data group YX data obtained by the forward conversion unit 210 and the input data x by using the state determination device D _X.

Specifically, the state determination unit 220 uses the state determination device DY of the data group Y stored in the neural network storage unit 270 to determine the state of the data G _{X → Y} (x) of the conversion data group XY and the input data y. Judgment of the state of, and each judgment result D _Y (G _{X → Y} (x)) and D _Y (y) are passed to the learning unit 280. Further, the state determination unit 220 determines the state of the data G _{Y → X} (y) of the conversion data group YX and the state of the input data x by the state determination device D _X of the data group X stored in the neural network storage unit 270. Judgment is made, and each judgment result D _X (G _{Y → X} (y)) and D _X (x) are passed to the learning unit 280.

The inverse conversion unit 230 converts the data of the conversion data group XY obtained by the forward conversion unit 210 into the data of the conversion data group XYX by the converter G _{Y → X.} Further, the inverse conversion unit 230 converts the data of the conversion data group YX obtained by the forward conversion unit 210 into the data of the conversion data group YXY by the converter G _{X → Y.}

Specifically, the inverse conversion unit 230 converts the data G _{X → Y} (x) of the conversion data group XY by the converter G _{Y → X} stored in the neural network storage unit 270, and the data G _{Y of the} conversion data group _{XY X. →} Convert to _X (G _{X → Y} (x)). The inverse transform unit 230, a data G _{Y → X} conversion data group YX (y), the transducer stored in the neural network memory section 270 G _{X → Y} data conversion data group YXY by G _{X → Y} ( Convert from G _{Y to X} (y)).

The forward / reverse conversion distance measuring unit 240 measures the distance between the input data of the data group X and the data of the converted data group XYX obtained by the reverse conversion unit 230 with the distance measuring device M ₁ . Further, the forward / reverse conversion distance measuring unit 240 measures the distance between the input data of the data group Y and the data of the converted data group YXY obtained by the reverse conversion unit 230 with the distance measuring device M ₁ .

Specifically, the forward / reverse conversion distance measuring unit 240 uses the input data x of the data group X and the data of the converted data group XYX obtained by the reverse conversion unit 230 G _{Y → X} (G _{X → Y} (x)). ) Is measured by the distance measuring device M ₁ , and the distance measurement result M ₁ (x, G _{Y → X} (G _{X → Y} (x))) is passed to the learning unit 280. Further, the forward / reverse conversion distance measuring unit 240 combines the input data y of the data group Y and the data G _{X → Y} (G _{Y → X} (y)) of the converted data group YXY obtained by the reverse conversion unit 230. The distance is measured by the distance measuring device M ₁ , and the distance measurement result M ₁ (y, G _{X → Y} (G _{Y → X} (y))) is passed to the learning unit 280.

As the distance reference of the distance measuring device M ₁ , for example, the L1 distance, the L2 distance, or the distance in the feature space of the neural network is used. When a neural network is used, a feature amount is extracted using a neural network as a feature extractor stored in the neural network storage unit 270, and the distance is measured.

The self-conversion unit 250 converts the input data of the data group Y into the data of the conversion data group YY by the converter G _{X → Y.} Further, the self-conversion unit 250 converts the input data of the data group X into the data of the conversion data group XX by the converter G _{Y → X.}

Specifically, the self-conversion unit 250 converts the data y of the input data group Y by the converter G _{X → Y} stored in the neural network storage unit 270, and the data G _{X → Y} (y) of the data group YY. ). Further, the self-conversion unit 250 converts the input data x of the data group X into the data G _{Y → X} (x) of the conversion data group XX by the converter G _{Y → X} stored in the neural network storage unit 270. To do.

The self-converted distance measuring unit 260 measures the distance between the input data of the data group Y and the data of the converted data group YY obtained by the self-converted unit 250 with the distance measuring device M ₂ . Further, the self-conversion distance measuring unit 260 measures the distance between the input data of the data group X and the data of the conversion data group XX obtained by the self-conversion unit 250 by the distance measuring device M ₂ .

Specifically, the self-conversion distance measuring unit 260 measures the distance between the input data y of the data group Y and the data (G _{X → Y} (y)) of the converted data group YY obtained by the self-conversion unit 250. Is measured by the distance measuring device M ₂ , and the distance measurement result M ₂ (y, G _{X → Y} (y)) is passed to the learning unit 280. Further, the self-conversion distance measurement unit 260 measures the distance between the input data x of the data group X and the data (G _{Y → X} (x)) of the conversion data group _{X X} obtained by the self-conversion unit 250. It is measured by the device M ₂ , and the distance measurement result M ₂ (x, G _{Y → X} (x)) is passed to the learning unit 280.

As the distance reference of the distance measuring device M ₂ , for example, the L1 distance, the L2 distance, or the distance in the feature space of the neural network is used. When a neural network is used, a feature amount is extracted using a neural network as a feature extractor stored in the neural network storage unit 270, and the distance is measured.

Neural network Symbol憶部270 stores the neural network as a neural network and a state determiner as transducers. Forward and reverse converting distance measurement unit 240, or a self-converting the distance measurement unit 2 6 0, in the case of using the distance in the feature space of the neural network, and stores the neural network as a feature extractor.

As a neural network as a converter and a neural network as a state judgment device, a neural network capable of expressing a time-series structure or a hierarchical structure is used. For example, use Gated CNN or LSTM.

Forward and reverse converting distance measurement unit 240, or a self-converting the distance measurement unit 2 6 0, in the case of using the distance in the feature space of the neural network, series structure and hierarchical time as a neural network as a feature extractor Use a structure that can express various structures. For example, use Gated CNN or LSTM.

The learning unit 280 sets the result determined by the state determination unit 220 to the state determination result of the data of the conversion data group YX so that the state determination result of the data of the conversion data group XY and the state determination result of the input data y are close to each other. Converted so that the state determination result of the input data x is close, the distance measured by the forward / reverse conversion distance measuring unit 240 is minimized, and the distance measured by the self-converted distance measuring unit 260 is minimized. Learn the neural network as a vessel. Further, regarding the result judged by the state judgment unit 220, the difference between the state judgment result of the data of the conversion data group XY and the state judgment result of the input data y is clarified, and the state judgment of the data of the conversion data group YX is made. A neural network as a state judge is trained so that the difference between the result and the state judgment result of the input data x becomes clear.

Specifically, the learning unit 280 determines by the state determination unit 220 so that the values of _DY (G _{X → Y} (x)) and _DY (y) are close to each other as a result of the determination by the state determination unit 220. Result The distance M ₁ (x, G _{Y → X} (G _{X →)} measured by the forward / reverse conversion distance measuring unit 240 so that the values of D _X (G _{Y → X} (y)) and D _X (x) are close to each other. _The distance M ₂ (y, y,) measured by the self-converting distance measuring unit 260 so as to minimize _Y (x))) and M ₁ (y, G _{X → Y} (G _{Y → X} (y))). Learn the neural networks G _{X → Y} and G _{Y → X} as converters so as to minimize G _{X → Y} (y)) and M ₂ (x, G _{Y → X} (x)).

More specifically, as an objective function that makes the values of D _Y (G _{X → Y} (x)) and D _Y (y) the same as the result of judgment by the state judgment unit 220 in the learning unit 280, for example. , When the input data y is given as the state judge D _Y , the probability p is output, and when the conversion data G _{X → Y} (x) is given, the probability 1-p is output. _{If, L adv (G X → Y} , D Y) may be minimized (formula (1) of the present invention) a. Similarly, as an objective function for making the values of D _X (G _{Y → X} (y)) and D _X (x) the same as the result of judgment by the state judgment unit 220, for example, input as the state judgment device D _X. Considering something that outputs the probability p when the data x is given and outputs the probability 1-p when the conversion data G _{Y → X} (y) is given, L _adv (G _{Y → X} , DX) should be minimized.

In the learning unit 280, the constraint of minimizing the distance measured by the self-conversion distance measuring unit 260 is used only for stabilizing the learning in the initial stage of learning, and is not used after the learning is stabilized. May be good.

Further, in the learning unit 280, the constraint of minimizing the distance measured by the self-conversion distance measuring unit 260 plays an auxiliary role in learning, and if learning is stable even if it is not used. , You do not have to use it.

Then, the learning unit 280 determines the result D by the state determination unit 220 so that the difference between the result D _Y (G _{X → Y} (x)) and D _Y (y) determined by the state determination unit 220 becomes clear. Learn the neural networks D _Y and D _X as state judges so that the difference between _X (G _{Y → X} (y)) and D _X (x) becomes clear.

Specifically, as an objective function that makes the difference between D _Y (G _{X → Y} (x)) and D _Y (y) clear as a result of judgment by the state judgment unit 220 in the learning unit 280, for example. , When the input data y is given as the state judge D _Y , the probability p is output, and when the conversion data G _{X → Y} (x) is given, the probability 1-p is output. _{If, L adv (G X → Y} , D Y) may be maximized (formula (1) of the present invention) a. Similarly, as an objective function for clarifying the difference between D _X (G _{Y → X} (y)) and D _X (x) as a result of judgment by the state judgment unit 220, for example, as a state judgment device D _X. Considering something that outputs the probability p when the input data y is given and outputs the probability 1-p when the conversion data G _{Y → X} (y) is given, L _adv (G _{Y) → X} , D _X ) should be maximized.

In [Equation 1] of the present invention , Cross Entropy is used in the objective function, but instead, the Euclidean distance, the Earth Mover distance, or the distance based on the energy function may be used.

Then, the learning unit 280 passes the learning result to the neural network storage unit 270.

The conversion unit 290 converts the input data to be converted by using the converter learned by the learning unit 280.

Acquisition Specifically, the conversion unit 290, when the input unit 100 receives the data x data group X as input data, the neural network as a transducer G _{X → Y,} from the neural network's rating憶部270 To do. The conversion unit 290 uses the neural network of the transducer G _{X → Y,} converts the data x is converted into conversion data G _{X → Y} (x). Likewise, the conversion unit 290, when the input unit 100 receives the data y data group Y as input data, the neural network as a transducer G _{Y → X,} obtained from the neural network's rating憶部270. The conversion unit 290 uses the neural network of the transducer G _{Y → X,} converts the data y to be converted in the conversion data G _{Y → X} (y).

The output unit 300 outputs the conversion data which is the conversion result converted by the conversion unit 290.

Specifically, when the input unit 100 receives the data x of the data group X as the input data, the output unit 300 outputs the conversion data G _{X → Y} (x) which is the conversion result converted by the conversion unit 290. .. Similarly, when the input unit 100 receives the data y of the data group Y as the input data, the output unit 300 outputs the conversion data G _{Y → X} (y) which is the conversion result converted by the conversion unit 290.

An embodiment is shown below.

[Summary 1]
For the series data of two domains, the input part that receives the series data and
A forward conversion unit that converts data from one domain (forward conversion input data) to data from the other domain (forward conversion output data) using a converter,
An inverse conversion unit that performs reverse conversion on the forward conversion output data using a converter and converts it into data in the input domain of the forward conversion unit (inverse conversion output data).
A state determination unit that determines whether or not the forward conversion output data is appropriate as the series data of the domain targeted by the forward conversion output data by using a state determination device.
A forward / reverse conversion distance measuring unit that measures a distance between the reverse conversion output data and the forward conversion input data using a distance measuring device.
A learning unit that updates the parameters of the converter and the state determination unit according to the results of the state determination unit and the forward / reverse conversion distance measurement unit.
A conversion unit that converts data received by the input unit using the converter learned by the learning unit, and a conversion unit.
A series data conversion device including an output unit that outputs data converted by the conversion unit.

[Summary 2]
In the series data conversion device, the converter of the forward conversion unit converts the data (self-conversion input data) of the domain to be converted by the converter to obtain the data (self-conversion output data). Department and
A series data conversion device including a self-conversion distance measuring unit that measures the distance between the self-conversion input data and the self-conversion output data.

4.2
The processing routine at the time of learning is shown in FIG. 5 , and each step will be described as follows.

1. When the data of the data group X and the data of the data group Y are input to the input unit 100, the learning processing flow is executed in the data conversion device.

2. First, in step S100, the forward conversion unit 210 and the self-conversion unit 250 acquire the data of the data group X and the data of the data group Y from the input unit 100.

3. Specifically, the input unit 100 self-converts the data x ∈ X randomly selected from the data group X and the data y ∈ Y randomly selected from the data group Y with the forward conversion unit 210. Hand over to part 250. When selecting data at random, it is not necessary that the two data x and y have a correspondence relationship. For example, in the case of voice data, x and y do not have to be the same utterance content data.

4. In step S110, the forward converter 210 converts x to G _{X → Y} (x) using the transducer G _{X → Y.} Further, the rectifier unit 210 converts the y to G _{Y → X} (y) using a transducer G _{Y → X.}

In 5. Step S120, the state determination unit 220 uses the state determining unit D _Y, G _{X → Y} a state determination result of _{(x) D Y (G X} → Y (x)), the state determination result of y Get D _{Y (y)} . The state determining unit 220 uses the state determining unit D _X, G _{Y → X} and (y) the state determination result of _{D X (G Y → X (} y)), the state determination result of x D _X (x ) Is obtained.

In 6. Step S130, the inverse transform unit 230, G _{X → Y} a (x) is converted into _{G Y → X (G X →} Y (x)) using a transducer G _{Y → X.} The inverse transform unit 230, G _{Y → X} a (y) is converted into _{G X → Y (G Y →} X (y)) by using a transducer G _{X → Y.}

7. In step S140, the forward / reverse conversion distance measuring unit 240 uses the distance measuring device M ₁ to measure the distance between x and G _{Y → X} (G _{X → Y} (x)) M ₁ (x, G _{Y → X} (G). _{X → Y} (x)))) is measured. Further, the forward / reverse conversion distance measuring unit 240 uses the distance measuring device M ₁ to measure the distance between y and G _{X → Y} (G _{Y → X} (y)) M ₁ (y, G _{X → Y} (G _{Y → X} (y)). y))) is measured.

In 8. step S150, the self-conversion unit 250 converts the y to G _{X → Y} (y) using a transducer G _{X → Y.} Moreover, self-conversion unit 250 converts the x in G _{Y → X} (x) using a transducer G _{Y → X.}

9. In step S160, the self-converting distance measuring unit 260 measures the distance M ₂ (y, G _{X → Y} (y)) between y and G _{X → Y} (y) using the distance measuring device M ₂ . Further, the self-conversion distance measuring unit 260 measures the distance M ₂ (x, G _{Y → X} (x)) between x and G _{Y → X} (x) using the distance measuring device M ₂ .

10. In step S170, the learning unit 280 determines by the state determination unit 220 so that the values of _DY (G _{X → Y} (x)) and _DY (y) are close to each other as a result of the determination by the state determination unit 220. As a result, the distance M ₁ (x, G _{Y → X} (G _X) measured by the forward / reverse conversion distance measuring unit 240 so that the values of D _X (G _{Y → X} (y)) and D _X (x) are close to each other. _{→ Y} (x))) and _{M 1 (y, G X →} Y (G Y → X (y))) and so as to minimize the distance M ₂ measured by self converting the distance measurement unit 2 6 0 Learn the neural networks G _{X → Y} and G _{Y → X} as converters to minimize (y, G _{X → Y} (y)) and M ₂ (x, G _{Y → X} (x)) and it is stored in the neural network's rating憶部270, updates the parameters of the neural network G _{X → Y} and G _{Y → X} as transducers.

11. Further, the learning unit 280 made a judgment by the state judgment unit 220 so that the difference between D _Y (G _{X → Y} (x)) and D _Y (y) becomes clear as a result of the judgment by the state judgment unit 220. differences as becomes clear results _{D X (G Y → X (} y)) and D _X (x), to learn the neural network D _Y and D _X as the state determining unit, the neural network's rating憶The parameters of the neural networks D _Y and D _X as the state judgment device stored in the part 270 are updated.

12. In step S180, it is determined whether or not all the data have been completed.

13. If all the data has not been completed (NO in step S180), the process returns to step S100 and the processes of steps S100 to S170 are performed again.

14. On the other hand, if all the data has been completed (YES in step S180), the process is terminated.

4.3 Processing routine at the time of conversion

The processing routine at the time of conversion is shown in FIG. 6 , and each step will be described as follows.

1. When the data x ∈ X to be converted or the data y ∈ Y to be converted is input to the input unit 100, the data conversion processing flow is executed in the data conversion device. Here, the case where the data to be converted x ∈ X is input will be described. The processing is the same when the data y ∈ Y to be converted is input.

2. In step S200, the conversion unit 290 acquires the input data x to be converted from the input unit 100.

In 3. Step S210, conversion unit 290, the neural network's rating憶部270 acquires a neural network of the transducer G _{X → Y} learned by the learning unit 280.

4. In step S220, the conversion unit 290 uses the converter G _{X → Y} to convert the input data x to be converted into G _{X → Y} (x).

5. In step S230, the output unit 300 outputs the converted data G _{X → Y} (x) obtained by converting the data x by the conversion unit 290.

5 Evaluation experiment 5.1 Experiment setting

The proposed method is generally applicable to series data conversion, but in the experiment, the proposed method was applied to parallel data-free speech conversion as an example and evaluated. As the data, VCC 2016 dataset (Non-Patent Document 12) was used. This dataset contains professional American English spoken voices, including 5 male and 5 female speakers. The data of each speaker is divided into 216 short sentences (about 13 minutes), of which 162 sentences are used for learning and 54 sentences are used for evaluation. In order to evaluate the proposed method under the condition without parallel data, when learning the proposed method, 81 sentences in the first half of the 162 sentences of the learning data were used as the source voice, and 81 sentences in the latter half were used as the target voice. .. In other words, learning was performed under the condition that there was no duplicate utterance between the source voice and the target voice. The audio data is downsampled to 16 kHz, and the 24-dimensional mel cepstrum (MCEP), log fundamental frequency (log F ₀ ), and asynchronous index (AP) are measured using the WORLD analysis system (Non-Patent Document 8) 5 Extraction was performed in ms. Of these speech features, the proposed method was applied to the mer cepstrum for conversion. Logarithm Gaussian normalized transformation (Non-Patent Document 6) was used for the fundamental frequency, and it was shown that there was no significant difference in the asynchrony index even after conversion, and the source audio was used as it was.

5.2 Objective evaluation

In this experiment, the proposed method was applied to mel cepstrum, so the quality of the converted mer cepstrum was objectively evaluated. As a comparison method, GMM-based speech conversion (Non-Patent Document 11) , which is one of the typical methods for speech conversion with parallel data, was used. Since GMM-based speech conversion requires parallel data for learning, all 162 sentences of training data were used. It should be noted that the proposed method was learned in a disadvantageous situation where there was no parallel data and the amount of data was half. As evaluation data, SF1 and SM1 were used for the source audio, and TF2 and TM3 were used for the target audio.

As evaluation indexes, Global variance (GV) (Non-Patent Document 11) and Modulation spectra (MS) (Non-Patent Document 10) , which are said to have a high correlation with the subjective evaluation of voice quality, were used. FIG. 7 shows a comparison of GVs for each order of the mer cepstrum of the proposed method (Proposed), the comparative method (Conventional), and the target voice (Target). From this result, it can be seen that the proposed method obtains a GV closer to the target voice than the comparison method. FIG. 8 shows a comparison of MS for each modulation frequency of the proposed method (Proposed), the comparison method (Conventional), and the target voice (Target). From this result, it can be seen that the proposed method obtains an MS closer to the target voice than the comparison method. Table 1 shows a comparison of the root mean square error (RMSE) of the logarithmic MS of the target speech and the converted speech. The smaller these values indicate that the converted voice is closer to the target voice, and the experimental results show that the proposed method obtains a logarithmic MS closer to the target voice than the comparison method.

5.3 Subjective evaluation

For the subjective evaluation experiment, the naturalness and speakerability were evaluated according to the protocol of VCC 2016 (Non-Patent Document 13) . As a comparison method, a conversion method with parallel data based on GMM (Non-Patent Document 11) was used. First, the Mean opinion score (MOS) test was performed to evaluate the naturalness. As the evaluation data, 20 sentences were randomly selected from the evaluation data for 2 seconds or more and 5 seconds or less. Nine people who were fully educated in English participated as subjects. The results of the MOS test show that in the case of same-sex speech conversion (SF1-TF2), the proposed method is 2.4, the comparison method is 1.3, and in the case of heterosexual speech conversion (SF1-TM3), the proposed method. Was 2.3, and the comparison method was 1.4. This score indicates that the larger the value, the higher the naturalness, and it was shown that the proposed method outperforms the comparative method in the subjective evaluation of naturalness.

The speaker quality was evaluated according to the criteria of whether or not the same person sounds as if the same person spoke for different utterance contents. As the evaluation data, 10 sets were randomly selected from the evaluation data and used. Nine people who were fully educated in English participated as subjects. FIG. 9 shows the results in the case of voice conversion between same-sex speakers (SF1-TF2). In this figure, the percentage of respondents who answered "absolutely the same as the target voice" was higher in the proposed method than in the comparison method. From this result, it can be seen that the proposed method is superior in terms of speaker characteristics.

Claims (6)

For the series data of two domains, the input part that receives the series data and
Using a transducer, the forward transform input data or al the data in one domain, and the rectifier unit for converting the forward conversion output data is the data of the other domain,
To the forward transform output data, using a converter, an inverse converter for converting the inverse transform output data is the data of the input domain of the rectifier unit performs the reverse transformation,
A state determination unit that determines whether or not the forward conversion output data is appropriate as the series data of the domain targeted by the forward conversion output data by using a state determination device.
A forward / reverse conversion distance measuring unit that measures a distance between the reverse conversion output data and the forward conversion input data using a distance measuring device.
A learning unit that updates the parameters of the converter and the state determination unit according to the results of the state determination unit and the forward / reverse conversion distance measurement unit.
A conversion unit that converts data received by the input unit using the converter learned by the learning unit, and a conversion unit.
A series data conversion device including an output unit that outputs data converted by the conversion unit. Prior SL-series data conversion apparatus, for the self-conversion input data is a data domain converter of the forward transform unit and converted, the self-converted output data is data converted by the converter With the self-conversion part to get
The series data conversion device according to claim 1 , further comprising a self-conversion distance measuring unit that measures a distance between the self-conversion input data and the self-conversion output data. The pre-Symbol converter and said state determination unit, series data conversion apparatus according to claim 1 or 2 constructed using a neural network that can capture the relationship between time series data. Before SL series data conversion device part using a model having a Gated CNN or LSTM or Attention structure according to claim 3, wherein the neural network. For the series data of two domains, the input part that receives the series data and
Using a transducer, the forward transform input data or al the data in one domain, and the rectifier unit for converting the forward conversion output data is the data of the other domain,
To the forward transform output data, using a converter, an inverse converter for converting the inverse transform output data is the data of the input domain of the rectifier unit performs the reverse transformation,
A state determination unit that determines whether or not the forward conversion output data is appropriate as the series data of the domain targeted by the forward conversion output data by using a state determination device.
A forward / reverse conversion distance measuring unit that measures a distance between the reverse conversion output data and the forward conversion input data using a distance measuring device.
With
A learning device that updates the parameters of the converter and the state determination unit according to the results of the state determination unit and the forward / reverse conversion distance measurement unit. A program for operating a computer as each part of the series data conversion device according to any one of claims 1 to 4.